Blog| Matillion ETL for Amazon Redshift

    How to Integrate Source Code Control Systems with Matillion ETL

    Matillion is essentially a visual code generator and as such can be integrated with your Source Code Control System via JSON files. You can use Matillion to build workflows (called “Jobs”) in a web-browser-based design interface. Afterwards, you can run the Jobs to perform ELT tasks, such as fetching data from an external source and transforming the data inside the database.

    When taking a backup, Matillion uses many different languages and formats internally meaning the generated code isn’t saved. Instead, Matillion offers a way for you to save your Job definitions into an ordinary operating-system file.

    Matillion’s files are in a custom JSON-based format. These files can be:

    • Saved into your company’s backup system
    • Exchanged with colleagues, in order to move work between environments (for example Development to Test, or Test to Production)
    • Added to your company’s Source Code Control System (for example GitHub, Mercurial, SubVersion or TFS)

    This blog post will explain two main methods Matillion offers for taking or restoring a backup:

    • Using Import/Export; and,
    • Using the REST API

    Both methods use JSON files. Your use case will determine the method you choose. While the formats are not compatible with each other, files created by either method can be used with a Source Code Control System in exactly the same way.

    Interfacing with Source Code Control

    The following steps will guide you through how to save or restore your Matillion work to or from a Source Code Control System. Note that the steps are the same regardless of which Source Code Control System you are using.

     

    Matillion-ETL-Source Code Control-Saving to and Retrieving from Source Code

     

    The exchange of metadata happens like this:

     

    Matillion-Source Code Control-Metadata Exchange Flow

     

    Backup and Restore using Import/Export

    In the main Matillion user interface, you can find the Import and Export options under the Project menu.

     

    Matillion-ETL-Import-Export

     

    These options allow you to backup and restore jobs between Matillion and your workstation.

    Backup and Restore using the REST API

    There is no user interface for this because it’s a JSON-based API. You can use any REST client that supports basic authentication, for example cURL.

    More documentation is available on Matillion’s REST API for:
    Matillion ETL for Amazon Redshift
    Matillion ETL for BigQuery
    Matillion ETL for Snowflake

    Want to try integrating a Source Code Control System with Matillion ETL? Start a free 14-day trial.