Blog| Matillion ETL for Amazon Redshift

What’s new in Matillion ETL version 1.40

Want the very best Matillion ETL experience? Each new version of is better than the last. Make sure you are on the latest version to take advantage of the features, new components, and improvements introduced in Matillion ETL v1.40.

Ready to upgrade? For more information on how to upgrade, check out our blog on How to Update Matillion – Best Practices.

New Data Connectors

Mandrill Query component

Use the Mandrill Query component to load your Mandrill data into your cloud data warehouse.

Matillion ETL New Component

Snapchat Query component

Use the Snapchat Query component to load your Snapchat data into your cloud data warehouse.

Matillion ETL New Component

Recurly Query component

Use the Recurly Query component to load your Recurly data into your cloud data warehouse.

Matillion ETL New Component

Snowflake Streams Support

Create Stream Component

Snowflake “Streams” records changes made to tables (inserts, updates, deletes) so that you can take actions on that changed data. The Create Stream component introduces support for this functionality within Matillion ETL. The Create Steam component is an orchestration feature that registers change tracking on a table and gives that stream a name.

Matillion ETL New Component

Stream Input Component

The Stream Input component, is a transformation component in Matillion ETL for Snowflake, to read changes from a stream. This component will detect changes regardless of whether the stream was created by our Create Stream component or within Snowflake.

Matillion ETL New Component

Support for Structured Data

Construct Struct Component

This component will create structs by simply selecting the columns to go into the Struct. Based on this, Matillion will create the new Struct and populate it with the specified data.

Matillion ETL New Component

Flatten Struct Component

The product now supports the ability to load and flatten Structs (nested fields) and Arrays (repeated fields).

Matillion ETL New Component

Extract Nested Data

The Extract Nested Data component allows users to unpack their nested data structures (such as JSONs) guided by a visual tree structure of the data.

Matillion ETL New Component

Aggregate component

The Aggregate component in Matillion ETL for BigQuery now has Array Agg functionality to better handle arrays.

Matillion ETL New Component

Nested Data Support

A “Define Nested Metadata” check has been added to the following components:

  • Create Table
  • Create External Table
  • Sample Grid
  • Metadata Panel

Select this option to transpose nested metadata from a simplified column layout into an intricate tree structure. Then you can add new structures nested within the tree, along with strings, integers, arrays, and other data types.

Matillion ETL Improvement

Read more about how Matillion ETL for BigQuery Supports Structured Data with Structs and Arrays

 

What’s new in Matillion: Enterprise Features

Git Integration

Git functionality lets users convert a Matillion project into a Git project. Each Matillion “version” in a Git project points at a specific Git commit, enabling development teams to devise workflows with speed and efficiency. Users can perform actions such as Commit, Create Branch, Configure Remote Repository, Fetch, and Push.

Read more about how Matillion integrates with Git.

 

Concurrent Connections

Matillion ETL for Amazon Redshift now offers users “ConcurrentConnections” to increase the number of concurrent connections to Amazon Redshift. This allows your jobs to run in parallel. As a result, some users may experience significant decreases in ETL job completion times.

 

 

 

Additional Improvements

Catalog Queries

Catalog queries have been updated to improve validation performance.

Matillion ETL Improvement

S3 Load component

To maintain parity with Snowflake on AWS 4, we’ve added properties to the S3 Load Component:

  • Stage Database
  • Stage Schema
  • File Format Database
  • File Format Schema

Matillion ETL Improvement

Query Result to Grid component now offers support for Advanced Mode (SQL query)

Now users can write custom SQL as part of their Query Results to Grid component configuration using the Advanced Mode option. This gives developers an added degree of freedom over their data transformations.

Matillion ETL Improvement

Windows Fileshare SMB2/3 Support

The Matillion ETL Data Transfer component and File Iterator component now support Windows Fileshare SMB2/3.

Matillion ETL Improvement

 

To read more about UI/UX improvements across all Matillion ETL products, check out our Matillion ETL v1.40 Improvements Blog.

 

See what’s new in version 1.40.

 

 

Full Release notes are available on the Support Site: