Skip to main content

Matillion ETL 1.64 Release Blog

 

The Matillion ETL data integration platform received another round of incremental improvements with the release of Version 1.64! Keep reading below for the highlights included in this newest version.

 

Summary of Releases:

  • 3 new connectors

  • 1 enhancement

  • 2 new features added

 

New and Updated Source Connectors

We’ve updated 3 connectors in the 1.64 release. With these updates, we are expanding our support for data ecosystems that are ever-growing in size and complexity. 

 

API Extract Connector

We have fixed the issue in our API Extract connector with the Bearer Token  in Matillion ETL 1.64.4.

The component was not passing the configured bearer token parameter value when “Authentication Type” was set equal to “Bearer token” being sent without the “Bearer” portion of the parameter value.

A workaround was possible, by using “Authentication Type=Api Key Value” or the parameters section.

Customers that adopted the workaround can now configure the bearer token in the connector directly.

Splunk Query Connector

We have released a new version of Splunk Query connector in Matillion ETL 1.64.3. 

Due to changes in the Splunk API tables, fields and metadata, we have versioned our component to avoid introducing failures to customers using older API versions. 

If you are using the deprecated version, it will continue working. However in order to use more recent Splunk API features, you may need to upgrade the connector.

We recommend consulting Splunk API releases and moving to the new version.

To learn more about these components, please visit our documentation page.

Shopify Query Connector 

We have released a new version of Shopify Query connector in Matillion ETL 1.64.3. 

Shopify constantly updates their data model. In recent versions of their API some tables, fields and metadata have changed. In order to make newer API functionality available, we upgraded our connector to default API version 2021-10 instead of  2020-10, now classified as unstable.

You can find changes in our data model, as referenced in the technical note.

If you are using the deprecated version, it will continue working. However in order to use more recent Shopify API features, you may need to upgrade the connector.

https://shopify.dev/api/release-notes/2021-10 

Improve Collaboration with Shared Jobs and GIT

 

The Matillion ETL platform just got a step closer to seamlessly integrating DataOps with your change management solution using GIT. Previously, Matillion ETL already supported version controlled jobs managed with GIT, but the latest release now extends this functionality to also support SHARED jobs. Our updated API now enables users to bundle entire workflows into a single, shared custom component and then re-use those custom components anywhere else in the project. Orchestration jobs (and the Transformation jobs they link to) can also be shared in this manner. If a job calls another job via Run Orchestration or Run Transform then all jobs will be included in the Shared Job. Indeed, a Shared Job can include as many jobs as the user wishes. 

Integrating DataOps with DevOps in this way improves code/component re-use and reduces overall development overhead as well as complexity. This new feature encourages more modular design and standardised data pipelines, which streamlines data orchestration and maintenance activities. Integrated version control of your Matillion ETL jobs improves the isolation/separation of work, which can then be leveraged to create efficiencies in development resources spent to build & maintain data related workflows.

Improvement Overview

  • GIT support extended to SHARED jobs
  • Supports both Orchestration & Transformation jobs
  • CMD-line flexibility leverages your GIT best practices

Reduce testing costs with Snowflake Zero Copy Clone

 

Matillion ETL 1.64 adds even more benefits for Snowflake customers already using the Zero Copy Clone feature, or perhaps considering to. Our latest release now enables users to also delete clones from within Matillion ETL as part of the clean-up stage after job execution.

Snowflake clones offer a great way to save cost and time because when they are created, no actual data is copied into the new database, but instead references to the live data is used. This means that only changed data is maintained in a separate store, resulting in reduced storage requirements (cost) and time taken to configure the cloned environment.

Matillion ETL provides a simple way for Snowflake users to leverage Zero Copy Clone functionality and run jobs against “as live” data in a cloned environment. This provides the benefit of confidence that your job is running correctly against real live data, without taking the risk of actually changing any live data – while also reducing the overall cost of testing.

With Matillion ETL 1.64 we’ve made it super easy to not just create your cloned environments on the fly when executing a job, but also letting you choose how to handle the clean-up stage after the job completes. You can now select to have your cloned database automatically deleted if the job runs successfully, fails, or in both cases.

Not only does this feature keep parity with Snowflake features, but is also a great way to save our customers time and cost when testing their data workflows.

Improvement Overview

  • Snowflake Zero Copy Clones reduces cost of testing Matillion ETL jobs
  • Easily create clones on the fly from within Matillion ETL job
  • Automatically delete clones after job execution

 

For more information on this exciting new enhancement, please see our in depth blog on using Zero Copy Clone for better automated testing here.

Bug Fixes and Enhancements

 

Matillion 1.64 includes one enhancement: 

 

Grid Variables for the Calculator Component

 

The calculator component was expanded to allow users to use ‘grid variables’. This makes it possible to supply a dynamic grid to the calculator component which specifies a column and calculation.

 

This feature could be used in many ways, for example if a user has a dependent variable (e.g. the ID of a record), then they can use that value to create another column in a view or table. Users can even stack multiple calculations on top of each other, perhaps ones that are dependent on the response of an API, or multiple columns.

 

If you’d like to see Grid Variables in action alongside the Calculator Component, click here

Product release notes

Full Release notes are available on the Support Site: 

 

Ready to upgrade to Matillion v1.64? 

For more information on how to upgrade, check out our blog on best practices for updating your Matillion ETL instance.

{demandbase.company_name}, realize the value of your Cloud Data Platform
With Matillion, {demandbase.company_name} can leverage a low-code/no-code platform to load, transform, orchestrate, and sync data with speed at scale, to get the most value across your cloud ecosystem. Check out these resources to learn more.