2020 Matillion/IDG Marketpulse Survey: 5 Key Takeaways

November 17, 2020

Ashley Lozito

Last year’s Matillion/IDG Marketpulse survey yielded some interesting insight about the amount of data in the world and how enterprise companies are handling it. In 2019, data volumes were growing at an average rate of 63 percent per month, and organizations were coordinating data from as many as 400 different data sources.

In the report on the 2020 survey of enterprise data professionals*, released this month, we learned that preparing those immense data volumes also takes up an immense amount of time for data teams–time that they’d definitely like to take back. We learned a few other things, too: read on for the five key takeaways from the 2020 Matillion/IDG Marketpulse survey. (Click here for an infographic version of the findings)

1: Data teams spend nearly half their time preparing data for analytics

High-quality insights require high-quality data. Unfortunately, for many data teams, high-quality data requires a lot of labor. The data professionals we surveyed estimate that 45 percent of their time is spent aggregating and preparing data for analytics production. That’s nearly half of their time spent on data transformation tasks rather than more strategic tasks with higher value to the business. As part of that effort, 38 percent of respondents cite too many manual processes as standing in the way of analytics projects. In addition, 36 percent say that cleansing and preparing data is a significant barrier to getting analytics projects to production.

2: Average time spent on data prep for a typical analytics project: One week

Data teams spend an average of one week preparing data for the typical analytics project. That’s not counting the data analysis itself, reporting findings, preparing dashboards, or any of the other tasks involved with sharing insight across the business. When today’s business climate demands information as closely to real time as we can get, waiting that long for results can be detrimental to any organization. And any way to speed that process can lead to a definite competitive edge.

3: Data portability and ease of onboarding: Both key in overcoming data/analytics obstacles

We asked those surveyed to tell us what features would be most critical in helping them overcome obstacles to timely data and analytics. Data portability was cited by 57 percent of respondents as a critical quality, as data teams often need to rely on the same data across multiple environments: On-premises, cloud, and increasingly, across different cloud platforms. Ease of onboarding and use of a product was also a critical quality for 57 percent of respondents. Multiple users can increase data team agility and productivity. A full range of users across the business–data engineers, data analysts and data scientists, and business data users–can also help balance workloads and democratize data.

4: Cloud data warehouses and data lakes both matter

We also found that the majority of respondents are not thinking about data warehouses and data lakes as an either/or proposition. Forty-three percent of respondents expect to have all of their data in the cloud at some point, with 38 percent already using cloud data warehouses. While only 16 percent are using cloud data lakes, more than half (56 percent) say that they plan to implement data lakes, with another 26 percent considering it. This desire for both types of cloud environments makes clear the opportunity for technologies like the lakehouse, which leverages the features of both the cloud data warehouse and the data lake for different use cases.

5: Governance and data accessibility are on everyone’s mind

Data democratization is the eventual goal of many modern organizations, making data available to different groups to self-serve analytics. However, one of the biggest obstacles to getting any data analytics projects to production continues to be data ownership and data control issues that make collaboration difficult, cited by nearly half (47 percent) of respondents. Organizations continue to realize that establishing governance and data stewardship will lift a significant barrier to analytics productivity.