Given the volume and complexity of data today, and the speed and scale needed to handle it, the only place you can compete effectively (and cost-effectively) is in the cloud. If you’re starting your first data transformation project in the cloud, you’re that much closer to speeding up analytics productivity and time to insight.
To effectively transform data in the cloud, you need to create minimum viable product (MVP) that covers an end-to-end ELT project.
What is an MVP?
Let’s talk about what an MVP is and isn’t. Often, people think of an MVP as an early prototype of a product, but that’s not exactly right. An MVP is a version of a new product that allows a team to collect the maximum amount of learning about customers with the least amount of effort. Basically, you want to get a product in your customers’ hands (or your data team’s, in this case) so that you can then iterate based on their input.
Now that we’ve defined an MVP for our purposes, let’s look at some tips that can help you create an MVP for a pipeline and workflow for transforming data in the cloud. We’ve identified seven things that will help make developing an MVP much easier.
1. Adopt modern data principles
Data problems can’t be solved with technology alone. You also have to address the culture around your organization’s data practices. It’s important to look at data principles such as accuracy, transparency, data guardianship and data literacy. You may need to change the way you’re approaching data and that can involve training for all of your data professionals. After all, a big part of changing your infrastructure and processes is change management for your teams.
2. Resist the urge to divide your development team
After you’ve decided what you’re going to do for the first iteration of your MVP, it’s time to get to work. If you have more than one developer, you might be tempted to divide and conquer, developing concurrently in silos. Resist that urge.
For this iteration, you’re not trying to maximize productivity. It’s more important at this point to set up your team for successful future iterations and scalability. Get all of your developers in a room together, literally or virtually, and let them work together to develop these pipelines end to end. This collaboration will pay off in the long run because in the beginning, your developers worked together to solve problems and share understanding and knowledge.
3. Engage all of your stakeholders as soon as possible
As you begin working on the first iteration, get all of your stakeholders involved, on the data team and beyond, so you can start to develop a working relationship. Use the first iteration to educate your stakeholders on the development process so they understand what to expect. You can begin to build trust with them and expand on that as you continue to deliver iterations.
4. Break down your MVP into digestible pieces
Let’s say your MVP is to develop a KPI showing weeks of inventory remaining per product for an e-commerce system. This might sound simple, but in fact, there are a lot of moving parts.
This particular KPI actually requires three data sources: inventory, sales, and product data. For each data source, you’ll need to gain approval from the data owners, establish a connection to the data source, analyze the data to make sure it meets your needs, and then clean it up, if necessary. Therefore, it makes sense to break this MVP down into smaller chunks, working on each data source separately before combining them to make the final KPI.
5. Develop with reusability in mind
As you’re working on your MVP, look for ways to make your jobs as reusable as possible. If you develop with reusability in mind, the jobs you create now can be used as templates in the future. This approach will also help make your jobs more portable as you move from development to test to production environments.
Pay particular attention to environment variables, natural key hashes, component naming, and the notes you include in your work. If you address these components carefully, you can increase portability and make it easier on the next developer who works with your jobs.
6. Centralize your team’s ideas
Your MVP may include multiple components. And even though you won’t be able to deliver all of those components right away, you want to get everyone’s ideas and vision into one spot so that you’re all working with the same information. Centralizing everyone’s thoughts will help facilitate conversations and address requirements.
7. Review both process and product for your first iteration
When you finish iteration one, take a step back and review your design decisions. Think about whether you’d like to change your approach as you move forward into future iterations. You want to make sure that each developer feels comfortable and is ready to move on to iteration two. Another thing you want to think about at this point is how are you going to isolate the work that you just created from the work that you’ll do next. You may want to move your first iteration to a user acceptance testing (UAT) or production environment.
Watch our partner webinar: From Zero to MVP
To learn more about how to create a solid MVP, watch our recent webinar “From Zero to MVP: Setting Up ELT in the Cloud” presented by Rachel Hill, a consultant with Matillion partner Slalom, and Arawan Gajajiva, Principal Solution Architect at Matillion. You’ll learn other information on setting up an ELT MVP in the cloud and hear about an MVP that Slalom recently developed with a client. It’s great information to have before embarking on your own MVP project.