Navigating the Future of Data Careers: Skills for Tomorrow
Written on
The Future of Data Skills
The data competencies you're developing today might not align with the requirements of the coming decade. With the rise of new tools and technologies that simplify data tasks, the demand for "soft" skills in data roles is expected to surge.
About two years ago, I was assigned to create a pipeline for extracting marketing data from the Amazon Marketing Services API to our database. I prepared to write the necessary Python code when my boss messaged me: "Have you checked out Open Bridge?" A quick search revealed that Open Bridge was another player in the automated ETL arena. Having previously used platforms like Xplenty and Stitcher, I thought I understood what to expect.
Over the next week, I was amazed by Open Bridge's user-friendliness. The process was as simple as entering credentials and hitting submit. Within a day, we had a complete set of AMS reports automatically delivered to our Azure Blob storage every 24 hours. My only task was to set up a pipeline to integrate the CSV files into the appropriate database tables, and I was finished.
When our first invoice for Open Bridge arrived, the transformation in the data job market hit me: the service cost $150 a month. A task that would typically take several days—spanning multiple API endpoints and over a dozen reports—was reduced to about 10 minutes.
Simplicity Equals Change in Compensation
While Open Bridge won't fulfill all data needs for any organization, its ease of use is concerning for those holding titles like 'Data Engineer' or 'Analytics Engineer'. There will still be a need for database management, optimization, and intricate transformations, but the tools to execute these tasks are becoming more accessible.
Consider HighTouch, which markets itself as a 'reverse ETL' platform. Despite the marketing jargon, HighTouch enables users to write back data from a database to an API. For instance, if you need to transfer Salesforce data into Marketo—a frequent requirement—there's no need for a complex integration. You can simply set up a job in HighTouch using straightforward SQL. Just input SQL into a designated box, and HighTouch will push the data into the selected platform based on available parameters.
Traditionally, integrating data from a database into Marketo or Salesforce would require at least a skilled data engineer. Currently, data engineers earn an average salary of around $120,000, with hourly rates reaching as high as $200 based on experience. However, as tools like HighTouch and Open Bridge become more common, these salary levels may decline. With fewer coding and server management requirements, there will be less for the average Data/Analytics Engineer to do.
Data Scientists Not Immune
Data scientists are not exempt from this shift either. Every major cloud platform—like Azure, AWS, and Google Cloud—provides 'Auto ML' solutions that can effectively replicate the capabilities of a typical data scientist. Although we aren't fully there yet, once you can input a dataset and a few parameters into an 'Auto ML' system to produce a functional, generalizable model, the market value of a data scientist is likely to drop significantly.
As more user-friendly tools are introduced that simplify the most sought-after data tasks, the hierarchy within the data science and analytics job market will flatten. If an entry-level data analyst can build and deploy data pipelines successfully, the need for seasoned data engineers may diminish.
Embracing Opportunities for Problem Solving
Fortunately, this evolution presents opportunities for those engaged in data professions. Technological advancements allow intelligent individuals to redirect their focus toward addressing more significant challenges rather than becoming entangled in the technical details of data extraction, transformation, and predictive modeling.
Contrary to popular belief, the most pressing issues in analytics and data science today are not technical. The complexity of these roles can sometimes hinder progress in the field. Often, those who ascend to leadership positions in analytics are highly skilled technicians who have built intricate data environments, reports, and models using cutting-edge technologies.
The downside is that these specialized technicians may not excel in determining the best ways to deliver and utilize the data they are adept at handling. When someone becomes proficient in a specific area, they may inadvertently prioritize it above all else. This tendency results in the creation of overly complex data environments that can process vast amounts of data quickly, yet leave the intended users struggling to derive insights.
In several instances, I have worked with billion-dollar corporations to bypass large analytics teams due to the difficulties in collaborating with them. These teams would often approach us with straightforward requests—simply seeking basic reporting to inform crucial decisions.
Adapting to Change in the Data Landscape
How can data professionals effectively adapt? While the demand for roles focused on technical aspects of data and analytics may decline, the need for experts who can interpret, leverage, and organize data within companies will dramatically increase.
Here are some suggestions to consider:
- Become a Data Translator: Companies will increasingly seek individuals who can connect technical analytics and data science teams with departments like sales, marketing, and finance. If you lead an analytics team, focus on establishing trust and stronger relationships between analysts, engineers, and the teams they support.
- Enhance Your Soft Skills: Skills such as communication, storytelling, and active listening will gain importance as technology evolves. Can you effectively convey the value of a technical analytics project to non-technical stakeholders? Are you able to craft compelling narratives that illustrate how data can benefit different departments?
- Prioritize Outcomes Over Architecture: In traditional software engineering, effective Product Managers help guide teams toward customer-focused outcomes. Many analytics teams lack this kind of structure, leading to over-complexity in their architectures. Focusing on outcomes can prevent this issue, ensuring that analytics projects deliver valuable information rather than becoming bogged down in excessive technical detail.
As data technologies advance, many companies will find themselves overwhelmed with data they struggle to utilize effectively. As you consider your future in the data field, it's essential to emphasize the strategic and intelligent application of data, perhaps even more than the platforms and coding skills you possess. As the demand for data engineering and data science skills declines, the need for individuals who can leverage data effectively is set to soar.
Be prepared for these changes.
Thank you for reading. If you found this article insightful, I explore various topics related to analytics on Medium. You can also connect with me on Twitter @camwarrenm or on LinkedIn.
Chapter 1: Embracing New Tools in Data
The first video, "This is what you should do to have a long-lasting, successful data career," discusses how to navigate a successful career in the evolving data landscape.
Chapter 2: The Importance of Soft Skills in Data
The second video, "Freelancing with Data Skills," focuses on how freelancers can leverage their data skills in today's job market.