Some of the best ETL advice and thought leadership comes from Matillion Partners, and we love to share their knowledge on the Matillion blog. In this post, Joris Van den Borre, Founder of Tropos.io in Belgium, shares how business data users and IT can co-create a successful model for increasing data literacy and promoting a more insights-driven organization. This blog originally appeared on the Tropos.io website.
A Simple Collaboration Model That Works
Nowadays, most organizations have programs to become more data-driven. Primarily, this is an organizational change effort, in which many of the organizations we work with are either in the process, have gone through it, or are about to embark on it.
For new business insights, one of the major obstacles that needs to be overcome is the time to market. IT-centric business intelligence teams have often struggled to deliver insights in time, while business-centric data teams have been challenged to realize economies of scale and to govern large data projects without IT.
Thanks to public clouds, organizations are now able to design an operating model between business and IT that allows for a healthy combination of both. Now, acceptable time to market for new insights, steady progress, and sustainable inflow of data product specifications based on true business demand can be guaranteed.
Our key insight? An end-to-end vision on delivering a data platform with all of the capabilities to manage, analyze, and secure data at scale. An end-to-end vision should align collaboration processes and people’s roles and responsibilities with data technology.
Business teams are in the driver’s seat
Looking at the previous trends of organizing business intelligence within organizations, we’ve noticed the trend of linking data-driven change with tool selection. We don’t feel this is the right way to go, based on our day-to-day jobs:
- Technology is an enabler, not a goal in itself.
- Business and process knowledge are embedded in business users’ ways of working, not in data engineering practices.
On the other hand, we can’t cut the IT practices out of the equation, because:
- One can’t expect businesses to both design and operate automation processes. They simply don’t have the time to govern such tasks.
- Getting secure access to data within service level agreements is closely linked to core IT operations, such as networking and identity management.
- Working in the cloud and using data creates a particularly interesting field for T-shaped professionals who span teams and practices, resulting in efficiency boosts for business teams.
Build a co-creation model that works
Assuming that our previous assumptions regarding data product building are correct – which have proven correct in the teams we’ve coached – we need to make a separation between information discovery tasks on one hand, and information distribution work on the other.
Businesses are typically better at information discovery
Want to improve data literacy and make your organization more data-driven? As a business-embedded data professional, you’ll be digging through data on a daily basis using technology. Typically, you’ll find many levels of appetite – working with data, and the technology required to work with the data. Here lies an important opportunity to get served in a better way. Ultimately, if you are in business, you should get an interface for your data that aligns with your current ways of working. Treat it like a sketchbook – a way to express your data-related thinking in a way that comes naturally to you. Although your data may export pretty well, it might be challenging to talk about your business concepts.
- You want your data answers fast, without computer hiccups and without the need to negotiate over database time with, say, Ana from Operations
- You want to be able to build new datasets by joining existing ones, and analyze the summaries
- The interface that works best for you, with your background and use case, might not work for your peers on the other team
Also, if you’re in business, it’s likely you just want to get your job done!
- You don’t want to see any tech lingo on your screen
- Nor do you want to be distracted by fancy industry buzzwords
- Lastly, if you’re building an insight that you frequently look at – say your monthly sales numbers – you just want to view the output, not the governance and control processes that ensure that those numbers are correct
Our key finding is the notion of building a collaboration model with IT. By doing so, you can hand over work that has been prepared utilizing business thinking and expressed in a language or format that is convenient for data engineers to pick up, without any structural efforts from the business side.
Data engineers are typically better at structuring processes
IT is all about providing reliable experiences. Data engineers just love to build repeatable processes that deliver quality data sets in time.
Although discovering the business value of data might not be their first focus, they do cater to some other important factors that will enhance confidence in data:
- Making sure the data quality is up to standards
- Making sure the data is tested (think of entry-related, out-of-range values that might distort your CAPEX report you’ve been pushing for)
To ensure that you get the most value out of the typically overbooked agendas of your data engineering colleagues, you need to make sure you hand over your business requirements as clearly and as specifically as possible.
Tip: the requirements.xlsx sheet that used to suffice in the past is not likely to work for much longer. Requirements have shifted – we are now demanding more output from our engineers, at a higher quality, and within a shortened time frame. Ouch!
The operating model that will likely work
The model that we’ve repeatedly put into practice is relatively straightforward.
- Create a common data platform that can retain and transform all of your organization’s data. It should be cost-efficient, require less maintenance and should be incredibly fast. One should create as few copies of data as possible on the platform in order to simplify governance.
- Select a convenient tool to combine and prepare data We’re not building data processes that should stand the test of time. We just need to prepare datasets that will either be used once for ourselves or used as an example for data engineering.
- Use whitebox technology to do the preparatory work By ‘whitebox’ we mean that the preparatory work should be done using technology that allows for the specifications to be opened up. By doing so, qualified engineers can view the internals and copy over the data set reasoning to a platform that fits their engineering paradigms.
- Put a rockstar handover process in place Technology is important, but requirements, expectations, and other important factors that are not captured in a dataset matter even more. Your product owner is a key player, as they steer the process and make sure that the business gets the most value out of their investment.
Pick accessible technology that pushes businesses forward
We’ve coached several business teams to become more data-driven by deploying business-led data products and programs. In doing so, we’ve met many profiles, ranging from data analysts to citizen data scientists.
While we have met many individuals from different levels of experience and degrees of willingness to adapt to new technologies, we have found that Matillion is the product that best sticks with most user groups.
Matillion positions itself as an enterprise-wide data transformation tool – something that your average data engineer would be happy to use. They seem to have an untapped population of analysts as a future client base too.
We noticed that some of our end users were using “extended proof-of-concept-time” from other teams to do their data prep work. The analysts, from an MS Access and Excel background, all of a sudden started a buzz about this new technology that was still under consideration. Interestingly enough, it seemed to solve many of the challenges that they had been facing for years:
- It was hard to come by new versions of their base data
- Processing large amounts of data was near impossible
- Their access to personal data (we’re in Europe, GDPR applies) was cut as they inevitably made offline copies of data sets
Interface-wise, we had a solution that was both intuitive and low maintenance for the data engineering team who suddenly saw this use case pop up unexpectedly.
The near-perfect ad-hoc integration and prototyping tool
Let’s create a case for prototyping and ad-hoc data set building at once.
Business users are already doing this with the tools they have to get the job done. We feel as though we have a great opportunity on our hands, one that will deploy the target operating model we’ve discussed before and put power in the hands of business users once again.
To make sure prototypes are governable, we either:
- Hand them over to the data engineers who specialize in improving technology efficiency. They’ll take the prototypes and align them with common data pipeline practices
- Or learn from the analysts’ logical reasoning put into the prototype, and rebuild it
Culture, capabilities and team composition will decide on which of the options is chosen. Perhaps both. Steady business value delivery is what matters here, technology should just enable it.
In Matillion, we found a technology partner who’s proven to support both – with tangible results measured in skyrocketing user adoption rates and satisfaction surveys.
Learn how Matillion ETL can help move your business forward
Want to see how Matillion ETL can help your business and IT teams work together to create a more insights-driven enterprise? Request a demo.
About the Authors
Joris Van den Borre is the founder of Tropos.io, a cloud data consultancy with offices in Belgium and the Netherlands. As a solutions architect, he designs data products and data platforms to help make organizations more data driven.
Jo Swenters, who also contributed to this article, is a product manager at Tropos.io. He helps align business priorities with engineering capacity to make business analysts more efficient.