How to Integrate Source Code Control Systems with Matillion ETL

  • Ian Funnell, Lead Solution Architect
  • September 19, 2017

Business man's hand walking his fingers up wooden steps

Matillion is essentially a visual code generator and as such can be integrated with your Source Code Control System via JSON files. You can use Matillion to build workflows (called “Jobs”) in a web-browser-based design interface. Afterwards, you can run the Jobs to perform ELT tasks, such as fetching data from an external source and transforming the data inside the database.

When taking a backup, Matillion uses many different languages and formats internally meaning the generated code isn’t saved. Instead, Matillion offers a way for you to save your Job definitions into an ordinary operating-system file.

Matillion’s files are in a custom JSON-based format. These files can be:

  • Saved into your company’s backup system
  • Exchanged with colleagues, in order to move work between environments (for example Development to Test, or Test to Production)
  • Added to your company’s Source Code Control System (for example GitHub, Mercurial, SubVersion or TFS)

This blog post will explain two main methods Matillion offers for taking or restoring a backup:

Both methods use JSON files. Your use case will determine the method you choose. While the formats are not compatible with each other, files created by either method can be used with a Source Code Control System in exactly the same way.

Interfacing with Source Code Control

The following steps will guide you through how to save or restore your Matillion work to or from a Source Code Control System. Note that the steps are the same regardless of which Source Code Control System you are using.

Image graph of the steps on how to save to and retrieve from Matillion's source code control

The exchange of metadata happens like this:

Matillion's source code control metadata exchange flow going from Matillion to JSON format metadata exchange to workstation to files to source code control

Backup and Restore using Import/Export

In the main Matillion user interface, you can find the Import and Export options under the Project menu.

Screenshot image of Matillion user interface where you can find theImport and Export options under the Project menu

These options allow you to backup and restore jobs between Matillion and your workstation.

Backup and Restore using the REST API

There is no user interface for this because it’s a JSON-based API. You can use any REST client that supports basic authentication, for example cURL.

More documentation is available on Matillion’s REST API for:
Matillion ETL for Amazon Redshift
Matillion ETL for BigQuery
Matillion ETL for Snowflake

[callout-content title=”Want to try integrating a Source Code Control System with Matillion ETL? Start a free 14-day trial.”]

[button title=”Get 14-day Free Trial” link=””]