What’s new in Matillion version 1.37?
The latest release of Matillion, version 1.37, includes a whole host of new components, new features, and enhancements to make Matillion better than ever before! Upgrade your instance today. For More information on how to upgrade, check out our blog on Best Practices for Updating your Matillion ETL Instance.
All Matillion Products
Version 1.37 of Matillion introduces four new data staging components, available across all platforms.
- Azure Cosmos Query, which connects to an Azure Cosmos Database using your Azure instance or user-defined credentials;
- DoubleClick for Publishers Query, which requires a Google-style OAuth;
- Twilio Query; and
- Apache Spark Query.
Each of these data staging components will bring in data from the relevant source and place that data into a target table, letting you extract the data that you want.
We are happy to introduce two quality of life features you’ll find in this update – Sample and SQL tabs, which are available to all data staging components.
Easily check your current configuration with the Sample tab, which allows you to retrieve a sample of your data based on the current configuration of the data staging component. You can specify the number of rows you want to retrieve in your sample with the Limit parameter, and you can discern the total row count by clicking the Refresh button next to the Row Count field.
The second quality of life component we mentioned is the SQL tab on all data staging components. Clicking this tab will show the SQL generated by the query a component is performing. In this way, you can drill down into the SQL code that sits behind Matillion’s simple UI.
With this update, moving resources from one Matillion instance to another has never been easier. Our new Migration Tool feature, which works on a similar principle to the Import/Export functions in API, offers users a simple method for migrating any number of Matillion assets directly from one Matillion instance to another.
Following version 1.37 the Migration Tool is the preferred (and safest) way to upgrade. Launch a new copy of Matillion, while still running the latest version, use the Migration Tool to move and validate the new version, before deleting the existing instance. In place upgrade may be removed in future versions.
Matillion ETL for Amazon Redshift
We are really excited about introducing the foundations of Change Data Capture (CDC) into our Matillion ETL for Amazon Redshift product. CDC is a technology for efficiently capturing changes made to a source database and applying them to a target database, ensuring your data is in-sync. Matillion’s approach to data transformation looks to leverage the cloud data warehouse for scalability. For this reason, Matillion’s CDC solution leverages Amazon Database Migration Service for Amazon Redshift.
Currently supported databases:
- Amazon Aurora MySQL
Users interested in learning more about this new feature can check out our in-depth video on CDC and our technical documentation.
More Options for Semi-Structured Data Support
The ability to tame complex semi-structured data, primarily JSON, within a Cloud Data Warehouse environment can be challenging, but Matillion is looking to simplify this, using the power of Amazon Redshift, in this case, specifically Spectrum. Users of Matillion ETL for Amazon Redshift can also now make use of Nested Data from certain file formats, such as JSON files.
Building off of new features in Amazon Redshift Spectrum, we have introduced 2 components inside Orchestration to:
- Create External Table component: Define the nested structure and relate that to the external JSON data
- Nested Data Load component: This component can flatten incoming nested data according to a user-defined structure. Unlike nested data, the flattened data can be fully-functional on Amazon Redshift.
Matillion ETL for Snowflake
In Matillion ETL for Snowflake and Amazon Redshift, this update adds a JDBC Table Metadata to Grid component, allowing you to connect to a variety of JDBC-type databases and then take the metadata from this table and place it into a grid variable. Read more about how to set up Grid Variables here.
For those Snowflake user leveraging Azure, we have introduced an Azure SQL Query component. Azure SQL Query component connects to a Microsoft Azure SQL Database.
Matillion ETL for BigQuery
The new Salesforce Incremental Load Wizard allows users to quickly and easily create incremental loads from a variety of popular database types, selecting as many data sources as you like, and use all of the columns from each data source as you see fit.
For more details on Matillion release v1.37, watch our video.