We all know that enterprises struggle to keep pace with the complexity and volume of data. But a new survey of 450 data professionals at large enterprise organizations indicates a critical root cause of that struggle: Outdated data migration, maintenance, and management processes. Legacy ETL applications, hand-coding, and other traditional methods of moving and maintaining data are sapping data teams of their time and their talent, and potentially costing businesses millions every year.
The survey, conducted on behalf of Matillion by independent research firm Vanson Bourne, reveals that 75 percent of data teams feel that outdated migration and maintenance processes are costing them time, productivity, and money. The survey found that data teams, on average, spend 57 percent of their time on data migration and maintenance. Factoring that time spent against the estimated total data costs per year, these outdated practices could potentially be running up those costs for enterprises.
The survey identified five critical areas were data teams struggle, and where process improvements such as low-code/no-code capabilities, automation, and cloud-native architecture could make a significant impact:
1. The effort spent on maintenance tasks
A majority of respondents cited three main tasks as requiring either a relatively high or very high level of effort:
- Bringing in data from disparate sources (63 percent)
- Preparing data for analysis (56 percent)
- And exporting and delivering data back to SaaS applications/systems, also known as sync back (55 percent)
Modernization in any of these areas would significantly reduce the amount of time spent on these tasks.
2. The negative impact across the business
The negative effects of inefficient data migration and maintenance aren’t just hard on data teams. The survey found that 66 percent of respondents believe that their organization is wasting time on data preparation. Over three-quarters of respondents (77 percent) believe data preparation and migration take longer than they should, delaying critical business decisions. This wasted time manifests itself in many ways across the business that erode the ability to be data-driven: A lag in time to value, outdated information, and a tendency of end users to come up with their own data and data processes that may or may not result in accurate data or adherence to best practices. Addressing inefficiencies in the data team can help close Information Gaps that open up across organizations when they don’t have steady, available access to quality data.
3. New and increasing “blind spots” because data teams can’t keep up
When it takes too long to prepare data, data teams find themselves having to pick and choose what tasks and types of data they can address. This leads to “blind spots” in the type of information organizations have, and a lack of visibility into how data is being used. Nearly 40 percent of data teams surveyed admit they don’t fully understand how data is being used in their organizations, and 44 percent worry about the challenge of dealing with the diversity in the types of data they work with. Cloud data (32 percent) and IoT data (31 percent) were noted as the most commonly unavailable or unsuitable sources for business intelligence and analytics. With both of these types of data becoming increasingly important in gleaning insight into organizational health and customer behavior, organizations literally can’t afford to overlook them due to lack of time and resources.
4. Losing the war for talent
There is a ruthless war for data talent in the job market right now, and most enterprises are losing. If data engineers and data scientists are unfulfilled on the job and spending their time on tedious, manual tasks for data migration and maintenance, they have the luxury of leaving and taking advantage of a booming job market. And many of them are unfulfilled:
- 87 percent of data decision-makers agreed that their organization struggles to retain talent.
- More than two-thirds of surveyed data users stated they are considering leaving their job in the next two years
- 29 percent are strongly considering leaving
In a climate where data teams are struggling to retain talent, it pays to not only be able to increase productivity and make the work more fulfilling, it’s critical to have tools and skills where you can easily and quickly ramp up new talent to start adding value right away.
data 5. Burnout is real
Not unrelated to the war for talent, survey respondents revealed that they are buckling under the constant stress and pressure of their job, which is not helped by inefficient, ineffective processes.
- 50 percent of those surveyed are burned out
- 79 percent are stressed
- 53 percent are bored
How can enterprises combat these challenges?
All of these are very real challenges that data teams face, and they are extremely bad for business. How can enterprises address these challenges and come out ahead with healthier happier employees and bottom lines? Respondents cited three big areas where enterprises can invest:
In the survey, 36 percent of respondents cite technology as the biggest priority to improve the overall data integration process. Investing in solutions that streamline tedious tasks such as data migration and maintenance frees up data users to focus on more impactful data projects and get a better handle on the influx of data across their organization.
Don’t just invest in your technology; invest in your people. That’s the second most critical area of focus according to survey respondents. Data professionals in an organization have a wealth of knowledge and the potential to transform raw data into a strategic asset — investing in them will yield strong results across the organization as businesses compete in the digital economy. Give them the tools to do their jobs well and head off burnout before it consumes your team.
Data architecture matters, according to 22 percent of respondents. Investing in the cloud can help remove a considerable amount of maintenance burden from data teams, and help them scale work for increased productivity. And if you’re in the cloud, your data stack should be, too. Invest in cloud-native tools and processes that take advantage of the speed and scale to free up data teams and enable them to collaborate more effectively on data science, AI/ML projects, and other analytics initiatives that will result in true business growth and innovation.
What else are data teams thinking? Find out in our ebook
These are just a few of the findings of our 2021 survey. To learn more, and to learn how Matillion ETL can help data teams solve some of their biggest challenges, download our ebook, “The Data Divide: Top Challenges Facing Enterprise Data Teams.”