Skip to main content

Matillion ETL 1.61 Release: DataOps and GIT Integration Enhancements

 

The Matillion ETL data integration platform continues to get better in Version 1.61! Keep reading for highlights and details in this newest version 

DataOps enhancements 

We love supporting our Ops community with new features that help you document, test, and automate your workflows faster than ever!

NEW data structure variable

We’ve introduced a brand new variable that lets you predefine and parameterize the structure you want to extract from semistructured data in your workflows. This data structure variable makes it possible to parameterize jobs with variable payloads so they continue running and extracting the right data with no oversight needed. Thanks to this new variable, ETL developers can now:

  • Ingest data directly from API connectors with a user-defined flattening schema
  • Define data structures to be extracted from variable JSON payloads without constant manual intervention
  • Predefine table structures that can be shared and reused in all Matillion ETL jobs
  • Leverage variable data types in modern data clouds like Snowflake, Redshift and Delta Lake to store semi structured arrays in traditional structured tables 

 

New and updated source connectors

We’ve added 3 new data connectors in the 1.61 release. More connectors means better integrations for a wider range of use cases, which in turn helps you innovate with data faster.

Google BigQuery connector for Delta Lake on Databricks 

Delta Lake users can now ingest data directly from Google BigQuery to support their multi-cloud needs across various business groups or projects.

Zendesk Chat connector for Google BigQuery 

BigQuery users can now integrate Zendesk chat data and blend it into their overall analytics. Now it’s easy to start using data from your in-product customer interactions with all the other great datasets and tools available on the Google Cloud Platform.

 

Google Sheets connector for Delta Lake on Databricks 

Delta Lake customers can now pull live data directly from Google Sheets into their Databricks environment. This brings Databricks users one step closer to getting all their data into a lakehouse from wherever it lives.

 

To learn more about these new data source components, please visit our documentation page.

 

MergeManager improvements

METL 1.61 adds some important UX improvements to our new and improved GIT functionality: MergeManager. 

GIT Diff UX improvements

In addition to all of the improvements we launched in versions 1.58 and 1.59 that make it easier to compare and merge changes across multiple branches, we’ve continued to make this feature more intuitive in version 1.61. You can now see and compare changes in a Matillion job down to the component logic level, and a new conflicts counter now helps you stay on track in the midst of a big merge. These changes make it  easier than ever to collaborate on Matillion projects with colleagues around the business. 

 

Bug fixes and enhancements

We’re always tinkering under the hood to give you the best experience on Matillion ETL. Here are some of the highlights:

Universal RPM for all Matillion ETL Images

We’ve done a little housekeeping to create a single base image for all Matillion ETL deployments on every cloud platform and data warehouse. A single base image enables us to put future versions of Matillion ETL into your hands faster and increase the responsiveness of our product roadmap to your needs. Please note this will require an extra upgrade step for anyone migrating from version 1.59 or older. You will be walked through the process of installing the universal RPM in version 1.60 before proceeding to install updates from version 1.61 and beyond.

 

 

Instance migration enhancements

With version 1.60 and beyond, there are some enhancements that make migration between Matillion instances a little easier. Persisting the address of your GIT Repo during the migration process leads to fewer keystrokes and an easier process to keep your Matillion ETL instance up to date with the newest enhancements. Most important — this enhancement is based on your feedback. Thanks for that, and keep your suggestions coming!

 

Ignore NULL values with the  First/Last component

Finally, we’ve updated our First / Last sort component with the ability to ignore “NULL” values, because nobody wants to use multiple components or write custom SQL when they don’t have to.

 

Product release notes

Full Release notes are available on the Support Site: 

Ready to upgrade to Matillion v1.61? 

For more information on how to upgrade, check out our blog on best practices for updating your Matillion ETL instance.

 

How to Upgrade