In Matillion ETL version 1.40, Enterprise customers (those on a Large or XLarge instance of Matillion) can now utilize Git. Git is a distributed version control system that is used to track code changes through the development lifecycle process. Some of our Enterprise customers have corporate policies that necessitate that all code, including their Matillion jobs, and associated items are stored in Git. This is usually for code control, business continuity, and disaster recovery. Git integration support is another way in which Matillion is responding to customer demand and integrating into mature enterprise IT environments.
Git integration: When to Use Git with Matillion
Based on the use cases we have heard from customers, Matillion Git integration support can be used:
- To commit to a remote Git repository
- To perform branching-based software development (e.g. work using common workflows from software engineering)
- As a mechanism for moving jobs between projects (as an example between Development, Testing and Production projects)
We recommend only using Git where it is mandated by your company. This implies that you are familiar with Git style branching and merging workflows and that your company has an external Git repository as part of a corporate-wide backup strategy.
Note: Matillion’s Git support will only allow for the storage of versionable items (Orchestration and Transformation Jobs) – not environmental items (Matillion environments, environment variables, project configurations like SQS or CDC).
Other Options for Version Control
If your corporation does not require Git, or you are not an enterprise customer, there are other native Matillion methods you can use to control code development, back up your work and move work between projects and instances. These are some options worth considering
- Use Matillion’s built-in Versioning system for basic version control, including tagging releases and change tracking
- Use the Server Migration Tool to move jobs and other collateral between Matillion projects without having to employ an external Git service
- Use the Matillion API to automate moving jobs between projects and environment configuration – this is useful if the user already has an external scheduler, needs to integrate the deployment of Matillion within a wider automated deployment process, or the user wishes to create a bespoke process.
What do I need to use Matillion with Git?
Before you start using Matillion with Git, there are a few prerequisites you will need to make sure that you set up. To use Matillion with Git you will need:
- At least one Matillion Instance with enterprise features enabled instance (Large or X-Large instance size)
- At least one Matillion Project not previously set up for Git
- An empty remote Git repository (e.g. BitBucket, GitHub, AWS CodeCommit, etc.)
- Credentials to access the remote Git repository
For your specific workflow requirements you may also wish to set up:
How do Git and Matillion work together?
To use Git with Matillion ETL, you’ll want to understand the underlying architecture and terms. There are six elements to Matillion Git integration:
- Matillion Project – the top-level structure containing jobs in Matillion. Projects are isolated from each other, and user access can be granted on a per-project basis.
- Matillion Version – a project contains one or more versions. These are working areas when used with Git. A version points to a single Git commit in the local Git repo (although you can repoint it to another commit).
- Local Git repo – stores Git files on the filesystem of the Matillion ETL instance, created automatically by Matillion ETL when a project is Git-enabled.
- Remote Git repo – a self-hosted or cloud-hosted Git repository that is external to Matillion ETL and set up by the user prior to Git usage with Matillion. Commits in the local Git repo can be pushed to this, or newer commits from the remote Git repo fetched to the local Git repo.
- Commit – a point-in-time copy of a Matillion version stored in it (e.g. a collection of jobs).
- Branch – a collection of one or more commits in Git with a given name.
Git Actions that work with Matillion ETL
The below diagram demonstrates how Matillion works with a Git repo. To get your code to Git from Matillion, there are a number of actions that you can use.
All Git operations are performed through the Manage Git screen. This screen acts a single view for controlling all of your Git operations. You can also view the Merge Tree and details of your previous commits.
Projects will need to be initialized in order to use them with Git. Initialise Local Repo will convert an existing Matillion Project to be Git-enabled by creating a local repo. Clone Remote Repo can copy the commits and branches from a remote repo into a local repo, and import the jobs into a Matillion project to make it Git enabled.
Below is a list of other common actions:
Basic Git actions
- Switch Commit – change the current Matillion version to point at specific import jobs from Git commit
- Commit – create a Git Commit in the local Git repository (within a specified branch) from the current state of jobs in the current version
- Create Branch – create a new Git branch from the currently selected Git branch
- Merge – merge another branch into the current branch and create a new commit (Note: this may cause a Conflict)
Remote Git Actions
- Configure Remote – supply the URL of a remote Git repository
- Fetch – perform a Git fetch operation to retrieve any new changes from the remote Git repository into the local Git repository
- Push – perform a Git push operation to send all changes from the local Git repository to the remote Git repository
By using these various Git actions, you can construct a number of different workflows to fit your business’ needs.
Watch a full overview of Git integration on Matillion:
Learn more about What’s new in Matillion ETL v1.40.