Matillion and IDG Survey: Data Growth is Real, and 3 Other Key Findings

Data volumes are expanding at an unprecedented rate. Enterprises are moving to the cloud. You hear this information over and over, and you definitely hear it from us. It’s very easy to get caught up in how many zettabytes of data we’ll all be producing by 2025, and how data and cloud computing affect us all on a global scale. But what do data growth and other issues that mean for how you do business right now? How does what you read track with the experiences you and other organizations are experiencing on a daily basis? The results of a recent Matillion and IDG Research survey are very illuminating.

Insight from 200 enterprise data professionals like you

Matillion and IDG Research recently conducted an IDG MarketPulse survey, “Optimizing Business Analytics by Transforming Data in the Cloud.” The survey polled more than 200 IT, data science, and data engineering professionals at North American organizations with at least 1,000 employees. Respondents work across several industries, including technology, finance, retail, and healthcare.

The research exposes the challenges companies face and the strategies they use to prepare data for BI and analytics, with faster time-to-value for implementing analytics projects rising as the main driver for migrating to a cloud approach. And the results validate and quantify a lot of things that data professionals have been telling us for years, about their data, their analytics processes, and what they hope to achieve within their organizations with the power of information and insight. Here are some of the key findings:

Finding 1: Data growth is hitting home for enterprises

Data growth isn’t just measured in zettabytes and it isn’t just for big global think pieces. Data professionals responding to this survey told us that, on average, data volumes are growing by 63 percent per month in their organizations. One in ten respondents told us that volumes are growing at 100 percent or more per month. Companies are at the receiving end of a firehose of data that keeps getting turned up higher. At that pace, it’s no wonder it’s so challenging to keep up, let alone turn that data into actionable insights.

Finding 2: That data is coming from hundreds (even thousands) of sources

We know that companies are collecting data from multiple data sources. These include computers, smartphones, websites, social media networks, e-commerce platforms, and now IoT devices. But the actual number of data sources is astounding. According to the survey, the mean number of data sources per organization is 400 sources. More than 20 percent of companies surveyed were drawing from 1,000 or more data sources to feed BI and analytics systems.

Finding 3: Everyone will have data in the cloud within two years

Respondents said that all told, 37 percent of organizational data is currently in cloud data warehouses. In addition, 35 percent of data is still in on premises data warehouses. And 25 percent of data resides in offsite, non-cloud data warehouses. But those numbers will shift significantly in the near future: virtually all respondents (more than 99 percent) said they will migrate data to the cloud over the next two years.

Why? Most people cited a faster time to value for implementing analytics projects. This would help businesses more effectively address common business problems. Respondents cited strategic marketing (39 percent), financial analytics and forecasting (37 percent), security risk mitigation (33 percent), and analyzing machine and IoT data (31 percent) as initiatives on their radar.

Finding 4: Transforming data to make it analytics-ready is challenging  for nearly all respondents

Enterprises are collecting lots of data from lots of places. And much of that data is going into the cloud. However, preparing those massive amounts of data for analytics is a stumbling block for nearly everyone. More than 90 percent of those surveyed said that it was challenging to some degree to make data available in a format usable for analytics. Of those, 57 percent said that transformation was highly challenging or extremely challenging. 

Actually, answers to another question may shed some light on why. Only 28 percent of respondents take an ELT approach: loading data into the cloud first, and then using cloud-optimized tools to transform it. 35 percent are using non-cloud ETL tools. And 37 percent said they were manually coding data to get it into the needed format for analytics.

There are lots of takeaways from the survey. But a big one is that what we’re talking about on a universal scale is indeed affecting real companies across industries, and real data professionals. There’s absolutely a need for simple, fast, scalable, and cost-effective data transformation. This capability can help enterprises achieve rapid time to insight and achieve dozens of critical business goals.

To see how Matillion can help your organization more effectively transform data within your cloud data warehouse, get a demo.

The post Matillion and IDG Survey: Data Growth is Real, and 3 Other Key Findings appeared first on Matillion.