Data Science

Quite a large number of the respondents combine data science responsibilities with other activities. These results indicate that there has been a democratization of the sphere and that there is potential for market growth.

Dedicated data specialist positions, such as Data Scientist, Data Engineer, and Machine Learning Engineer, are relatively new. Many of our respondents transitioned into these roles after working or training in adjacent fields, and therefore needed to upskill through independent study or online courses. While postgraduate degrees in STEM have traditionally been the most common path into data science or machine learning, trends from the last seven years show that an increasing number of people working in these areas entered with a bachelor's degree, rising from 20% in 2015 to 31% in 2021. As the number of people graduating from new undergraduate programs specializing in data skills increases, we may see these results shifting in favor of people who obtained these skills through formal education.

Jupyter notebooks won out as the preferred editor for data science and data analytics work, with around 40% of the respondents indicating they used notebooks for these activities. This result was even higher among respondents who reported doing data gathering and visualization, exploratory data analysis, or machine learning modeling, with 70% reporting that they use Jupyter notebooks.

Besides local files, SQL databases remain the most commonly used data sources among data specialists.

Core machine learning algorithms, such as regression and tree-based methods, continue to be used widely. However, the majority of respondents also use neural networks, especially transformer architectures. The increasing ease of use and growing popularity of transformer nets may also explain why over a quarter of the respondents reported doing NLP work. Interestingly, only a fifth of the respondents reported using statistical testing as part of their work, suggesting that machine and deep learning have overtaken classical statistics as a core data skill.

TensorFlow was the most popular deep learning framework among all respondents, although it and PyTorch were equally used by respondents who do data work as a main activity. Scikit-learn was the most popular machine learning library, although specialist packages and frameworks for tree-based modeling, such as XGBoost and LightGBM, were used by a notable minority of participants.

Amazon services are the most popular enterprise cloud solutions.

The majority of the respondents – 70% – work in small groups with no more than 10 people in a team. One in five works on a team with more than 15 data specialists.

Almost 50% of the teams or departments have a dedicated Data Engineer role.

Most respondents indicated that they use GPUs to train their machine or deep learning models. VRAM needs differed depending on how the respondents do data work. 40% of the respondents who do data work as a hobby or for educational purposes indicated that 8 GB was sufficient, compared to only 18% of those who do data work as a main work activity.

Most respondents indicated that they spend up to 20 hours a week training models, which may include the time that the models spend training overnight. Almost a third spend 5 hours a week or less training models. This is consistent with previous results showing that model training forms a relatively small part of data science work, with the majority of time being spent on data preparation and exploration.

Consistent with other answers in our survey showing that the main activity done in notebooks is data exploration and visualization and that the majority of respondents work with local files, the majority of the respondents also use local resources to complete their data science work. Surprisingly, this did not differ much depending on how the respondent does data work. People who do data work as their main activity were as likely to use local resources as those who do it as a hobby or for educational purposes.

The majority of the respondents said they do not use any tools to track the performance of their model training experiments. However, the use of such tools was much more likely on data teams comprising 15 or more people (58% of respondents from such teams use at least one), when the team has a dedicated Machine Learning Engineer (62%), or when the respondent was involved in machine learning modeling and ML Ops work (63%). This indicates that this sort of tooling tends to be used in environments where there is specialist knowledge pertaining to machine-learning-model development.

Simple but meaningful plots for exploring and presenting data were used by the majority of data specialists. These charts were used by the majority of respondents regardless of the type of data activities they were involved in, from data gathering and exploratory data analysis to data orchestration and ML Ops.