At Matillion, we value our partners and their unique insight into the data goals and challenges of today’s businesses. Kicking off our inaugural partner guest blog, Brian Bickell from InterWorks talks about why cloud-based ELT is a cornerstone of Modern Cloud Analytics.
It’s now 2020 and much fanfare is being made about new data integration technologies and methods. Maybe you’ve heard about techniques like streaming being the new hotness when it comes to data pipelines. Streaming is exciting stuff that we should all learn more about, and you know what, I think it’s wonderful that we’re having a moment where data engineering is exciting again.
At a full-stack consultancy like InterWorks, we always want to keep an eye out for bleeding edge technologies and give our clients the best solutions available for their needs. The thing is that “best” is often subjective. Fastest, cheapest, easiest, most scalable, most comprehensive, most feature-rich – these are all prioritized differently depending on who you ask. What works for Company A might not work for Company B and so on.
Bringing this back to the world of data integration, I’m here to share why the case for cloud-based ELT in 2020, specifically with Matillion, is as strong as it’s ever been. Just to be clear, the goal here isn’t to pit batch-oriented ELT and streaming against each other in any sort of death match. Both methods have their uses. However, much like the idea of Hadoop killing the data warehouse was totally misguided hype, streaming is certainly not killing off batch-oriented transformations any time soon. In fact, I think that ELT with Matillion plays a crucial role in what we at InterWorks consider to be true Modern Cloud Analytics.
Wait, what Is Modern Cloud Analytics?
The truth is, most analytics projects fail to deliver business value. You may have encountered this reality firsthand. Whether because of technical risks like tool selection, scalability and performance, or initiative risks like failure to make a business case or even internal politics, analytics projects are easy to get wrong. Modern Cloud Analytics aims to solve those challenges by providing a repeatable schematic for success. There’s no guarantee that anything will work for everyone, but Modern Cloud Analytics was derived from years of putting these solutions in place and streamlining how they work best together.
This schematic leaves a lot of room to plug in different data sources, tools and platforms, but we do have preferences for what we feel works best in most instances, from the choice of cloud data warehouse to our go-to for BI, reporting, and analysis. As you might have guessed, Matillion takes the ELT spot. Again, everywhere is a little bit different, but we’ve found the most success with this stack.
ELT and orchestration with Matillion
So, let’s zoom in specifically on the role Matillion plays within Modern Cloud Analytics. Why do we recommend it? What utility does it have for so many of our clients? A lot of it starts with the fact that many organizations have batch-oriented processes throughout their business. Operating from that frame of reference, their primary goal is to quickly model and implement those batch processes to prove value as soon as possible. That’s what makes the most sense for their business problems.
While these processes may already be implemented in legacy ETL platforms such as Informatica, SSIS, ODI or Talend, simply swapping the data warehouse for a modern cloud data warehouse and keeping these tools intact leaves much to be desired. They do not leverage the dynamic scalability of the modern cloud data warehouse and instead perform transformations locally, requiring significant hardware investments and unnecessary movement of data. That’s one of the major draws of Matillion, especially when paired with a powerful cloud-based data warehouse. Because Matillion is oriented as a cloud-based ELT platform, it can take advantage of a cloud data warehouse’s power and perform those same tasks (and more) at greater speeds, on more modest virtual machines, without moving data out of the warehouse.
Balancing power and practicality
Could an organization completely re-architect their data environment to support something like streaming and have great success? Absolutely. The practical reality, however, is that many organizations don’t have the budget, resources, or bandwidth for such an undertaking. What’s more, we go back to the fact that even the most well-intentioned projects can fall victim to technical and initiative risks. When we start to balance risk vs. reward in that context, the ability to leverage solutions like Matillion, which can integrate into virtually any stack and scale with incredible ease, is compelling.
There are real-world examples to support this value proposition. In the case of this global mining leader, Matillion managed massive and rapidly changing data volumes while also automating data quality monitoring services. For the senior care providers at InnovAge, Matillion transformed manual ETL processes and automated them while also allowing for workflows that could move and merge data into a Data Vault structure. With cosmetics manufacturer Börlind, Matillion’s pay-as-you-go model was essential in providing speedy data transformations without tying up capital and resources unnecessarily. The commonality in all of these use cases is that Matillion delivered meaningful value with minimal risk.
What will ELT look like moving forward?
Shifts are happening in the world of data. There’s so much exciting new technology out there, but with all the newness comes the real question of which technologies will last, which will evolve, which will only have limited use cases, and which are novelties. Even the best of us can’t predict what will happen next, and if you can, your time might be better served in Las Vegas or on Wall Street. Though InterWorks always has one eye on the future, we largely take the long view on which technologies and partnerships provide the most practical value for our clients. Matillion checks those boxes for us, and looking at the rest of 2020, that value is only increasing.