What’s it like to move from an on-premises data architecture to the cloud? The answers are as varied as the organizations who have done it. As part of our recent Partner Webinar Series, the webinar, “On-Prem to Cloud: An Executive Roundtable,” Kelly Kohlleffel, Vice President of Marketing & Alliances at Hashmap (and podcaster extraordinaire), talked to data leaders at three very different companies about their experiences and lessons learned from moving their data operations from on-premises to the cloud and adopting cloud ELT using Matillion ETL.
Our three participants were:
- Ajay Bidani, Digital Enablement and Insights Manager at Powell Industries, a traditional manufacturing company based in Houston
- Tyler Wheatley, Senior Manager of Data & Analytics at Waste Management, the number one environmental services and waste management company in the United States
- Harry Hopkins, CIO of Viewgol, a startup that sells an analytics tool that helps healthcare and medical companies identify and prevent lost revenue from gaps in billing and claims systems.
Some topic highlights:
Why the cloud?
Ajay Bidani: Especially being a manufacturer, the heart of what we do is very transactional, and that’s something that most of our reporting was built around. But honestly it really wasn’t built around making the kind of hour to hour, shift type of decisions. It was more of the looking day over day, month over month, year over year. We really didn’t have an easy way to start doing that. None of our data stack or on prem one was built around that…So cloud just literally gave us the easiest chance to kind of just jump in and start trying to deliver some things and get some people thinking in those terms.
Tyler Wheatley: Scalability I think was the biggest thing…we had a very traditional on-premises data warehouse appliance. And so what’s one of the biggest problems with using an appliance like a Teradata, like an IBM the teaser product or Greenplum or something like that? Well that doesn’t scale. The only way we can scale it is to go off and buy another appliance….So we found ourselves in that situation in order to really deliver just kind of the thirst for data within the business is just explosive. People wanting to do self service reporting from all over the country.
Harry Hopkins: It’s nice being in a startup because everything feels greenfield, but then…also part of the challenge of it is prioritizing. So we had this backlog, but some important things we weren’t able to do simply because we didn’t have the data infrastructure to be done…what we found was is that we spent a lot of time optimizing our data infrastructure…we (needed) to move so that we could have greater control and also new options. So that I could return our team to focus on innovating in our product and not so much time on DevOps and infrastructure.
How pricing fosters agility and innovation
Tyler Wheatley: From my perspective, (pay for what you use is) huge because it allows you to experiment for really low costs. So you increase your rate of innovation just by trying out different products. That’s how we ended up using Matillion in the first place…Within I think day one, we were able to establish connectivity to our Snowflake environment and connect us to some of our on-prem data sources and start beginning to experiment with the products. So it’s a massive paradigm shift from what we were doing before. It’s really enabled us to move quickly and to stand up some valuable solutions for the company.
Ajay Bidani: Matillion ultimately of all the kinds of options we looked at became the easiest way to just take a step in without having to think heavily in terms of how much data it’s going to take to do all these things. Honestly, we just weren’t ready to think in those kinds of terms yet. So cost is a little bit easier for us to just kind of observe and see. And again the idea of it’s on and you’re using it, is just an easier concept for us to start with than it was to think all the way into how much data this means day to day, chart by chart, dashboard by dashboard.
Harry Hopkins: When we go into the cloud, I think one of our first focuses was, we know this will scale easily, but it will scale our successes and our mistakes just as fast…my team is just ready to jump in, like give them the tools, let them experiment. But ultimately I’ve got accountability to partners and other owners of the company that I don’t want to wake up that morning and get that bill.
Transferring skill sets to Matillion ETL and the cloud
Tyler Wheatley: I think it translates very well. Depends on what you’re coming off of, but hey SQL. So if you know SQL, if you’re coming from an existing data warehouse environment, NC SQL is not that much different. There are some nuances and tactically, but that effectively translates across some of the traditional data warehousing solutions over to Snowflake. And then of course with the Matillion, if it’s a source object, all you need to know is it’s kind of a traditional ETL skillset…it’s understanding databases, understanding how your databases perform, and it probably helps to have some scripting skills. We’re using Python within our Matillion jobs in certain places to automate capturing some new source objects and things of that nature. But that’s pretty much it.
Ajay Bidani: Tyler kind of hit it really well but SQL is a really, really core thing that kind of depends on what your sources are, but I would say especially as a traditional kind of manufacturer, again traditional databases. NoSQL is not a thing for us if it is traditionally SQL, it’s Oracle databases, SQL databases. So the SQL knowledge is really kind of our key building block from most of what we’re attempting. And that’s why in some ways I felt like as long as the cloud allowed us to just need to be SQL centric and still be able to find value we can deliver, that was the best place to start.
How Matillion changes the way we work
Harry Hopkins: Now I can free up the team to say hey just go out there knowing that it’s the cloud so we can experiment…Previously because the data was siloed in on-premise environments, it was difficult for us to have a single platform to build our RPA on that was going to integrate with all of these disparate systems. It was just kind of a nonstarter. Well now, we’ve got the data there. So now it’s a matter of giving somebody the right access and saying go experiment, and it’s low cost and it’s really low risk, from my perspective and what I’m managing.
Ajay Bidani: Sometimes I’m kind of jealous. I’ll be fair. Some of the things you’re talking about…I want to be there…I feel like we’ve made some good progress. We had some stumbling blocks along the way, but just looking forward to being able to do some of these things, enable some of these things, especially when I’m going to be doing more of that…It feels great to be in a spot where now we’re having a conversation about this where a few years ago we really weren’t. I mean, it’s kind of crazy to think about now, but the idea of being able to think about some of the things that Harry’s talking about, where before there just wasn’t room to have it at all.
Tyler Wheatley: Man that says it all, give me all the data immediately, all in one place, all governed, all secure, all accessible and with all the performance I could ever want in the world. I mean that’s what everybody wants.
Listen to the whole conversation
These are just some of the topics we talked about on the webinar. Listen to the whole recording to hear how these vastly different companies chose Matillion, what’s next on the horizon, and their best tips for starting and surviving the move to the cloud. Thanks to Kelly for hosting, and stay tuned for highlights from the rest of the Partner Webinar Series.