As we modernize our data architectures, we need to modernize our data teams, too. Not necessarily by replacing our old data team with new people and new technology – experience, deep knowledge, and critical thinking will never go out of style. But we do need to think about the roles our data team is filling. Could intelligent, skilled people be put to better use for the business? Are the skills they have the right ones, and are we giving them the right tools to up-skill?
We look at these questions and more in our ebook, Build a Modern Data Team. First and foremost, it’s important to understand what’s changed, so we can figure out how to adapt to new challenges and opportunities.
Then and Now: What’s Changed?
In the on-premises world, companies invest heavily in engineers with the expertise to build custom data ingestion workflows, monitor databases, identify data and report on it for business users, and ensure that the data on hand is consistent, up-to-date, and trustworthy. The more data and jobs you have? The more engineers you need in order to keep up.
In today’s hybrid and cloud-first environments, though, the highly technical coding skills and other hands-on tasks that were in high demand even a few years ago just to keep workflows going are giving way to new skills and demands. Even low-code and no-code tools and technologies that were cutting edge three years ago are outdated – designed for an earlier generation of big data, but not one in which the volume, velocity, and variety has continued to get bigger, faster, and more varied. And that means engineers who once had to take occasional classes to brush up their skill sets now need to engage in continual learning just to keep pace with an industry that seems to evolve almost daily.
More data, less structure
Technology market research firm IDC predicts the torrent of semi-structured and unstructured data streaming in real time from IoT sensors and devices will more than quadruple the amount of data in the world in the next five years. An IDG Marketpulse survey we conducted in fall of 2019 found that, on average, data volumes were growing by 60 percent per month at enterprise organizations. (Some companies reported data growth of 100 percent per month.) Companies reported that they used data from 400 different sources. There simply aren’t enough engineers in the world to hand-code all of the integrations necessary to manage that almost unimaginable amount of data. Even if universities were pumping out engineers at breakneck pace, human beings couldn’t write code fast enough. It’s not humanly or financially feasible to keep doing things the old way.
Cloud data warehouses are here to stay
While companies still need to prepare data for artificial intelligence and machine learning, cloud data warehouses like Amazon Redshift, Google BigQuery, Snowflake, and Azure Synapse Analytics remove the burden of normalizing data to make it fit for use. The line between structured and semi-structured cloud data warehousing and unstructured cloud data lakes has become so blurry that Azure Synapse Analytics has effectively erased it. Business users no longer need to query the right data type, or even know what the right data type is, in order to find the right information to support the analytics they want to perform.
Data teams are moving fast in all directions
A data team is also under pressure to perform every stage in a data journey simultaneously. Instead of driving automated decision making and predictive insight by moving smoothly through one advanced cloud analytics project at a time, the data team may be simultaneously ingesting data from social media for one initiative, modeling structured data from backend systems for another, and working with data consumers to define business rules for a third. Without an understanding of the data itself, they may end up reverse-engineering reports from the desired results and arriving at inaccurate results.
The rise of the citizen data professional
Business teams no longer have time to write up requirements for a report, share them with IT, and wait until IT has time to pull the data and perform analytics. They want to access and analyze data themselves so they can arrive at conclusions sooner and make strategic decisions faster. Given the global shortage of data scientists and the sheer amount of data available
for analysis, it makes sense to decentralize and democratize access to data so business users can take more responsibility for their own data-driven projects. At the same time, companies need more governance and greater insight into what business users are doing so they can support projects that are innovative and relevant – and cut off resources to projects that aren’t.
The modern data team is in the cloud
In short, a data team focused on hand-coding and retrofitting technologies for cloud migration is not necessarily one that is prepared for modernizing data architecture and, with it, cloud data management. The modern data team also needs to include people with the ability to organize data for cost-effective analysis and reporting that leads to intelligent conclusions – that is, data engineers with cloud training and experience as well as the ability to understand the data itself.
To learn more about the skills and vision it takes to modernize your data team, download our ebook, Build a Modern Data Team.