Understanding CI/CD and how Matillion leverages it

CI/CD stands for Continuous Integration/ Continuous Delivery. It allows companies to automate the migration process in a repeatable way. It involves planning and executing a process the entire team follows to prepare your development work for the automation process. 

That said, with Matillion ETL, the preparation almost exclusively involves Git Source Code Management to rely on version control and Push/Fetch/Switch Commit features as part of automation. 

Matillion leverages our time-based scheduler and Push/Fetch/Switch Commit APIs to integrate with CI/CD features such as GitHub Actions, Azure DevOps, and Jenkins.

Why CI/CD?

With the power of cloud data platforms such as AWS and Azure and the ability to optimize them with cloud data warehouses such as Snowflake, Amazon Redshift, and DataBricks, CI/CD is more important than ever. 

The resulting massive performance improvements let customers retrieve their data for reporting and data science in seconds. That's especially important considering we're in an age of acceleration that requires developing Matillion Jobs and getting them into production as quickly as possible. 

Jobs were often migrated monthly at the earliest. Sometimes even quarterly or every six months depending on the datasets and the scope of the project/release. 

Now, companies often push our releases bi-weekly and even sometimes as quickly as several times a week or even daily. This speed makes CI/CD all the more important to ensure migration integrity and quality.

How To Get Started with CI/CD

It’s crucial that you have the proper setup and steps to get started with CI/CD. Matillion makes setup easier by integrating with CI/CD platforms. 

We'll automate triggers to schedule CI/CD when your migration kicks off, automate the Push to your Git platform, and do a Fetch and Switch Commit (to ensure you’re on the right Git Branch) into your higher environments. 

From there, confirming the Push to Git is successful and triggering Matillion to Fetch is done by the Git platform to continue the CI/CD process. Here’s a more thorough step-by-step view of what the process looks like:

1. First, you need to choose a Git tool if you haven’t done so already. Github has GitHub Actions for CI/CD, and Azure DevOps has Pipelines. The rest of the Git tools (Bitbucket, GitLab, etc.) do not have their own process, so they’ll use Jenkins instead, which can be universally used with all Git tools, including GitHub and Azure DevOps.
2. Set up your Git Integration tool to connect with Matillion ETL.
3. Once this is complete (in Development, Test (Quality Assurance), and Production) and you have all your Jobs in Git, in Branches, and Committed and Merged, it’s time for CI/CD automation.
4. Plan a time that you’ll migrate; going from Development to Test, you may want to plan a daily migration. Going from Test to Production (or Test to Staging and Staging to Production, depending on how you do regression testing) will be monthly at the latest, but more often every other week, weekly, or several times a week.
5. Build a Matillion Job to do a Push. This will leverage our Matillion Push API. You can find the documentation with the API endpoint here: API v1 - Git integration for projects.
6. Next, confirm that the Push was successfully moved to Git. You will configure this in GitHub Actions, Azure DevOps, or using Jenkins. In your code for each, you will confirm a successful push, then either use Matillion’s APIs to call another Matillion Job to execute a Fetch and Switch Commit or code in those APIs (Fetch and Switch Commit APIs) directly into your Git CI/CD code. Whichever you choose, you’ll have successfully migrated your code from your lower environment to your higher one.

The ability to Migrate your Matillion Jobs efficiently is essential when companies have massive amounts of data they need to make actionable decisions. 

CI/CD with Matillion gives us this flexibility to be more productive and proactive with our company data, providing crucial insights that match the speed of doing business.

Why not have a go yourself and sign up for a free trial?