How to Integrate Source Code Control Systems with Matillion ETL
Matillion is essentially a visual code generator and as such can be integrated with your Source Code Control System via JSON files. You can use Matillion to build workflows (called “Jobs”) in a web-browser-based design interface. Afterwards, you can run the Jobs to perform ELT tasks, such as fetching data from an external source and transforming the data inside the database. When taking a backup, Matillion uses many different languages and formats internally meaning the generated code isn’t saved. Instead, Matillion offers a way for you to save your Job definitions into an ordinary operating-system file. Matillion’s files are in a custom JSON-based format. These files can be:
- Saved into your company’s backup system
- Exchanged with colleagues, in order to move work between environments (for example Development to Test, or Test to Production)
- Added to your company’s Source Code Control System (for example GitHub, Mercurial, SubVersion or TFS)
- Using Import/Export; and,
- Using the REST API
Interfacing with Source Code ControlThe following steps will guide you through how to save or restore your Matillion work to or from a Source Code Control System. Note that the steps are the same regardless of which Source Code Control System you are using. The exchange of metadata happens like this:
Backup and Restore using Import/ExportIn the main Matillion user interface, you can find the Import and Export options under the Project menu. These options allow you to backup and restore jobs between Matillion and your workstation.
Backup and Restore using the REST APIThere is no user interface for this because it’s a JSON-based API. You can use any REST client that supports basic authentication, for example cURL. More documentation is available on Matillion’s REST API for: Matillion ETL for Amazon Redshift Matillion ETL for BigQuery Matillion ETL for Snowflake
Begin your data journeyWant to try integrating a Source Code Control System with Matillion ETL? Start a free 14-day trial.
Manager of Developer Relations
Matillion Adds AI Power to Pipelines with Amazon Bedrock
Data Productivity Cloud adds Amazon Bedrock to no-code generative ...Blog
Data Mesh vs. Data Fabric: Which Approach Is Right for Your Organization? Part 3
In our recent exploration, we've thoroughly analyzed two key ...eBooks
10 Best Practices for Maintaining Data Pipelines
Mastering Data Pipeline Maintenance: A Comprehensive GuideBeyond ...