Skip to main content

Matillion ETL 1.51 Release Notes: CYOC Updates, Error Handling, and More

 

Say hello to the very latest and greatest release of Matillion ETL yet, Version 1.51! 

Every new update of Matillion ETL improves upon the last, and in this blog you can find out more about the new features and components on offer before taking advantage of some great enterprise features in Matillion ETL 1.51.

New and updated data source components

We’ve added three new connectors in the 1.51 release. With these updates, we are expanding our support for data ecosystems that are ever-growing in size and complexity. More connectors means better support for a wider range of users and use cases, which in turn will help organizations turn their data into a strategic asset.

Google Sheets Query for Azure Synapse 

Azure Synapse users can now query Google Sheets to easily bring data from their ad-hoc analytics environments to achieve  a more programmatic analytics approach, driven by Azure Synapse Analytics. . Getting data out of the “pet project”  state and converting it into a reusable, standardized asset for the enterprise is a great capability for organizations pursuing a functional analytics program. 

Cloud Storage Unload for Snowflake on GCP 

Whether driven by a desire for process and platform interoperability or a need to use storage solutions more cost-effectively, the ability to unload Snowflake data into long-term cold storage has become one of the most popular use cases with customers. Our AWS and Azure Snowflake products already offer S3 and Blob Storage Unload components, respectively. With Matillion ETL 1.51 our Snowflake on GCP users can now also enjoy some feature parity, helping them drive down their storage costs.

 

SQL Server Output 

According to leading industry analysts, approximately 40 percent of enterprise data in 2021 lives in the cloud, and that’s projected  to grow to 61 percent by 2025. That still leaves a significant amount of data in legacy storage types, driving a need to support a hybrid cloud data environment for some time to come. With this update, we are offering SQL server output as an option. Whether for temporary data migrations or supporting your phased transition to a steady state from on-premise databases, Matillion ETL can help  smooth over your data transformation journey.

To learn more about these components, please visit our documentation page.

 

Create Your Own Connector (CYOC) improvements

 

METL 1.51 adds some important improvements to CYOC. 

 

Oauth 2.0 support

First, support for one of the world’s most popular authorization frameworks, Oauth 2.0, is here! Oauth 2.0 support emerged  as a high demand item at a recent customer focus group. This update addresses the long tail connector needs of many users, while reducing the need for authorization solutions outside of Matillion ETL. Customers can now leverage a more consistent authorization framework across a larger data ecosystem.  

In order to support the use of Oauth entries in both Wizards and Components, we extended  the “Manage Oauth” system to support custom APIs. In this release, we support both Client Credentials and Authorization Code grant types. Support of Oauth 2.0 is now available on Snowflake, Amazon Redshift, and Google BigQuery, with query only support on Azure Synapse.

Query-only support:

 

More CYOC guidance coming soon! 

In the weeks following the 1.51 release, we will provide various guides, demos and other learning opportunities to demonstrate why enterprise data teams and users should employ CYOC to break down data silos and enable a single view of data. 

 

Updated enterprise features 

 

Error handling

Matillion ETL already has a number of ways to help make jobs more resilient to errors, including explicit error handling, logging and notifications. However, as cloud infrastructure becomes more complex, often across multiple platforms, so do the ETL jobs to support that infrastructure. 

 

Feedback from some customers indicated that that failures had become more problematic to diagnose since:

  • Errors may be several layers deep within a job/subjob hierarchy
  • There may be more than one error 
  • The existing error-handling approach could entail a lot of cumbersome configuration and still lack the full details

 

A customer focus group concluded that there is a need for a single  place to configure a generic error message and the ability to push this out to a webhook. 

With this release, we introduce new error-handling functionality which enables customers to:

  • Pass information from all levels of a job to a single consolidated error message, which is then posted to a webhook of your choice. This means an error message several layers deep in a next job can be located from within a single message, then  traced back to the source of the problem
  • Configure generic error-handling rules for an entire project, which removes the need to explicitly configure error handling per component (unless specifically desired)
  • Streamline your DataOps support and maintenance routines

 

Dependency Checker

Matillion ETL will now map the dependencies of your nested and shared jobs. A warning will be provided when renaming a job will break downstream workflows.

 

Stability and Security Updates

Matillion ETL has been updated to run custom python or bash scripts as a restricted user by default. This improves the stability of your Matillion ETL instance and prevents accidental modifications to the operating environment. There is an override setting for scripts that need to run with sudo access which can be enabled for new or legacy scripts as needed.

 

New in Matillion ETL for Amazon Redshift  

 

JDBC as the default database driver

Matillion ETL has historically used the Postgres driver to connect to Redshift. This release will change the default to the Redshift JDBC driver. This shift will provide superior support for more advanced Redshift features such as advanced data types and external table control. Users will have the option to change an environment variable to still use the Postgres driver, if desired.

 

Full Release notes are available on the Support Site: 

 

Ready to upgrade to Matillion v1.51? 

For more information on how to upgrade, check out our blog on best practices for updating your Matillion ETL instance.

 

 

 

 

The post Matillion ETL 1.51 Release Notes: CYOC Updates, Error Handling, and More appeared first on Matillion.