The field of data science and AI is booming: technology companies are regularly releasing new models, while others are increasing their spending on AI tools – and seeing their investments pay off. “AI-mania” covers almost all industries and leads to a race for talent. At the same time, the number of specialized specialists is growing. Universal data scientists are gradually giving way to experts who work in specific areas – from AI researchers to MLOps engineers.
How will the data science and AI labor market change by 2030 and what skills will be key to success in the industry? Denis Pinchuk , an expert in Data Science, ML and Computer Vision, Senior Data Science Engineer at The Walt Disney Company, talks about this and the impact of AI-automation and democratization of technologies on the industry.
From generalists to experts: what will happen to the data scientist profession
In the nineties, there was a profession of computer scientist on the world market – that was the name of a person who worked with computers. Today, there is no such position, because it has long since been divided into several specialties. And although data science is a narrower field, a similar fate may await it. Even now, the industry is beginning to distinguish three areas: research, implementation, and production.
Specialists from the former develop fundamentally new models. For example, the Applied Computer Vision Researcher creates computer vision algorithms, and the Applied NLP Researcher deals with natural language processing. Such people are few in number – they are usually PhD holders from top universities who then go to work at DeepMind and other bigtechs. The latter can afford to invest in creating products with delayed results.
A more mass direction is implementation. Almost every large company today has a Machine Learning Engineer. Organizations define the functionality of such a specialist in their own way, but usually ML-engineers optimize ready-made models and integrate them into business products. Data engineers and data analysts work in parallel with them. The former are responsible for collecting, processing and storing data, while the latter are responsible for analyzing this data and finding useful insights, but without the use of machine learning algorithms.
Production can include, for example, MLOps engineers. Their main goal is to automate model deployment processes and monitor their work. In the same category are product managers who manage the development and deployment of AI products. They connect business requirements and technical implementation.
In the future, even narrower specialties may emerge within these areas. These are, for example, ML engineers with a focus on specific vendors like AWS or Azure. They will use off-the-shelf neural networks from Amazon, Microsoft, Google, and other vendors and do fine-tuning for business purposes. I note that organizations are already posting job openings for experts in specific cloud solutions, with more open positions being for candidates with Microsoft Azure knowledge. I attribute this to the fact that AWS entered the market earlier, so more experts have had time to get to grips with it.
In addition, we will see professionals who work only with SQL, only with BI tools, and so on. To a large extent, this deepening of specializations is due to the expansion of the field itself. The data science industry used to be much smaller in terms of tools, investments, and the number of experts. As a result, one person could cover several disciplines at once. In recent years, the market has been growing with an average annual growth rate of at least 20%. The field is expanding and becoming more complex, and in order to implement data-based projects with high quality, more highly specialized specialists need to be involved.
It is likely that some AI professions will also experience industry divisions. For AI researchers, industry differences will likely not play an important role – unlike product managers. Their responsibilities include AI product strategy, monetization, and scaling. Accordingly, it’s important for them to understand the industry, consider regulations, user needs, and the competitive landscape. These professionals will find it much easier to work with AI solutions if they are focused on the same industry.
Finally, new professions may emerge that either do not exist now or are not widespread. An example is specialists in AI ethics and AI fairness. These experts will assess the level of bias in models, control their transparency and compliance with ethical standards. They will take on the development of methods to combat discrimination in algorithms and implement mechanisms that increase the objectivity of AI.
Must-have for the specialist: what you need to avoid getting lost on the market
As today, AI specialists of the future will have a universal set of skills, without which it is impossible to become an expert in the industry. Currently, working in this field implies knowledge of Python and SQL, basic understanding of cloud platforms and MLOps. In addition, there is a standard set of libraries and tools, such as Pandas and NumPy for data processing and analysis, Jupyter Notebook for testing and debugging, Docker and Kubernetes for model deployment, and others. Even managers of data science teams need to know these solutions so that they can speak the same language as technical experts.
In addition, every company has a “classic” set of tasks that a data scientist is likely to have to deal with – building price analytics, demand forecasting, customer segmentation, anomaly detection. The more common the problems a data science and AI professional has to work with, the more basic the skills to solve them will become.
Finally, any expert will need to have highly specialized skills. What kind of skills – it depends on the field. Even now in computer vision and natural language processing, despite similar neural network architectures, the requirements for competencies are very different starting from the middle-engineer level. In particular, a CV-specialist should understand methods of image augmentation and object segmentation, while an NLP-engineer should be competent in tokenization, language model training and text tone analysis.
Soft skills will be equally important, and they too will vary from position to position. Senior specialist will be able to choose one of two development paths:
- Individualized Contributor who focuses on deep expertise in a specific area.
- Manager who takes leadership of teams of data scientists.
In management positions, it is important to be able to identify the strengths and weaknesses of employees, and to understand the basics of data science well enough to assign tasks to experts. In addition, it is important to be able to objectively assess the feasibility and cost of accomplishing tasks and to be able to communicate information clearly to senior management.
Another key skill is strategic thinking. It is necessary not only to understand how a tool solves a specific problem, but also to anticipate its impact on the further development of the project and the company as a whole. In this context, we can recall Jeff Bezos’ concept of two types of doors: two-way (those decisions that can be canceled or adjusted relatively easily) and one-way (irreversible strategic steps). At the data science and AI manager level, it is often necessary to open doors only one way.
Here is an example from personal experience: on a project in a large American corporation, a manager decided to develop a product in a monorepository. In this case, all components are stored in a single repository. At the stage when there were 15 engineers in the team, this seemed to be the optimal approach. However, over time, the project scaled up to 100 specialists, and the volume of data processed reached hundreds of millions of transactions per day.
Because of the monorepository, the likelihood of conflicts when making changes increased dramatically. Teams had to adhere to strict work rules, which complicated and slowed down the delivery of new features. The optimal solution would have been to use a multirepository, but it was impossible to switch to it at a late stage. If the manager had envisioned scaling the project in advance, the problems would have been avoided.
In addition, there are standard soft-skills that are relevant for any specialist – critical thinking, adaptability, and the ability to communicate with other experts. It is impossible to reach the level of an individual contributor – principal data scientist – only through hard skills. After all, if an AI professional is immersed in one specific aspect and does not have his finger on the pulse of the bigger picture, he can harm the project.
A simple example: let’s say a team is developing an AI model to predict financial risks. One of the machine learning experts chose a complex architecture, but did not discuss it with the production engineers. As a result, the model is too resource intensive and cannot be run in real time on the available servers. The project is delayed and business goals are not met in a timely manner. If a dialog is established, these problems will not arise.
Democratization and automation: job simplification or a new wave of competition?
By 2030, advances in AI and AutoML will automate many routine tasks such as data cleaning and preprocessing. Modern LLM models already simplify SQL analytics because they allow specialists to formulate queries in natural language. And AutoML systems like Google AutoML and H2O AutoML independently select parameters for ML models, choose algorithms, evaluate the quality of solutions and even offer recommendations for improvement. The trend is directly related to the trend towards democratization of AI: neural networks are becoming more accessible to users without deep technical knowledge.
This opens up new perspectives, but also leads to changes in the requirements for AI specialists. Today, to become a junior data scientist, one must master complex math, statistics, machine learning principles, and programming. With the development of AutoML, some of the tasks can be automated, which will potentially shorten the learning curve and lower the threshold for entry into the profession. This will create a new dynamic in the labor market: there will be more specialists with basic knowledge, but without a deep understanding of algorithms.
Accordingly, when selecting candidates, employers will pay less attention to theory testing and more to the ability to formulate tasks, analyze data and interpret results. In the future, companies will invest in the most promising and employable candidates, turning them into highly qualified experts.
At the same time, I do not think that automation and democratization of AI will lead to a reduction in the number of vacancies in the industry. Even today, the same task can be solved by a different number of specialists – it all depends on the desired speed and available budget. Rather, this trend will change the content of work. Engineers will spend less time on routine and more time on optimization, adapting models to the specifics of the business and interpreting results, and will create better products with the same effort.
The secret to success is the ability to adapt (and communicate)
It is difficult to predict the future of the AI and data science labor market. I will not be able to say which of the narrow specializations will be most in demand in the future, and which ones will lose relevance. One thing is clear: specialists need to constantly monitor trends, adapt to changes and develop both universal and specialized skills for their field.
And, of course, we can’t forget about soft skills. The ability to interact effectively with colleagues from other teams, to formulate thoughts clearly, to transfer and perceive knowledge – all this is critical for working in the industry. Communication skill can be compared to riding a bicycle: once you learn it, you never lose it. And the ability to collaborate productively is half the battle.
