How to Integrate Source Code Control Systems with Matillion ETL

Matillion is essentially a visual code generator and as such can be integrated with your Source Code Control System via JSON files. You can use Matillion to build workflows (called “Jobs”) in a web-browser-based design interface. Afterwards, you can run the Jobs to perform ELT tasks, such as fetching data from an external source and transforming the data inside the database. When taking a backup, Matillion uses many different languages and formats internally meaning the generated code isn’t saved. Instead, Matillion offers a way for you to save your Job definitions into an ordinary operating-system file. Matillion’s files are in a custom JSON-based format. These files can be:

• Saved into your company’s backup system
• Exchanged with colleagues, in order to move work between environments (for example Development to Test, or Test to Production)
• Added to your company’s Source Code Control System (for example GitHub, Mercurial, SubVersion or TFS)

This blog post will explain two main methods Matillion offers for taking or restoring a backup:

• Using Import/Export; and,
• Using the REST API

Both methods use JSON files. Your use case will determine the method you choose. While the formats are not compatible with each other, files created by either method can be used with a Source Code Control System in exactly the same way.

Interfacing with Source Code Control

The following steps will guide you through how to save or restore your Matillion work to or from a Source Code Control System. Note that the steps are the same regardless of which Source Code Control System you are using.     The exchange of metadata happens like this:    

Backup and Restore using Import/Export

In the main Matillion user interface, you can find the Import and Export options under the Project menu.     These options allow you to backup and restore jobs between Matillion and your workstation.

Backup and Restore using the REST API

There is no user interface for this because it’s a JSON-based API. You can use any REST client that supports basic authentication, for example cURL. More documentation is available on Matillion’s REST API for: Matillion ETL for Amazon Redshift Matillion ETL for BigQuery Matillion ETL for Snowflake