The latest release of Matillion, version 1.39, includes a whole host of new components, new features, and enhancements to make Matillion better than ever before! Upgrade your instance today. For More information on how to upgrade, check out our blog on Best Practices for Updating your Matillion ETL Instance.
All Matillion Products
Version 1.39 of Matillion introduces three new data loading components
Basic Mode for RDS and Data Query Components
As a Matillion user, you will already be familiar with our “Advanced” mode in both RDS Query and Database Query components. This allows you to write your own SQL when running queries. To minimize the need to write SQL yourself, just choose the “Basic” mode and configure the component’s other properties as normal. With Basic mode selected, the Matillion UI will present configurable properties for you to choose your Data Source, columns, and data source filters. This should help you get your components configured faster and therefore reduce your development time. For further assistance, just click the Help tab for Matillion’s in-client documentation.
Matillion ETL for Amazon Redshift
Version 1.39 of Matillion introduces two new data staging components, in Matillion ETL for Amazon Redshift:
It is worth noting that each of these three components only write to an External Table; some of the data, therefore, might require flattening via the Nested Data Load component.
You can now set a Default Storage Location by going through the wizard set up within the Environment section of your Matillion instance. Next time you use a data staging component, the S3 Staging Area you set will have the “Environment Default. However, you can edit the staging area manually if you wish. This makes configuring your orchestration jobs that little bit faster and more intuitive.
Matillion ETL for Snowflake
Snowflake External Tables
New to Matillion ETL for Snowflake on AWS is External Table support. External Tables in Snowflake reference data held externally, such as on Amazon S3.
Support for External Tables in Matillion ETL for Snowflake introduces two new components. First, the Create External Table component, creates an external table via a selected Stage to point at an S3 location. Users can assign a relative path within the S3 Location, and the component will find and load any data that matches the path and the File Type.
The second component, Refresh External Table, syncs your external table in Snowflake with the existing external data. After any changes are made to the external data, run the Refresh External Table component to reflect the changes.
Read our article on External Tables to learn more about how to use these components.
Snowflake Pivot component
Also new to Matillion ETL for Snowflake is the Pivot component. Use the component to “pivot” unique values from a column into rows, aggregating as you go.
Azure Blob Load Generator
Matillion ETL for Snowflake hosted on Azure instances now features the Azure Blob Load Generator tool, found in the Components pane. To begin using this tool, simply drag the icon onto the canvas and a setup wizard will load. The purpose of this tool is to bring data into your warehouse from Azure Blob Storage with simplicity and speed.
When going through the wizard, Matillion will guess the table’s schema to accurately represent the file’s metadata, reducing the amount of manual entry required by you the user.
The wizard will then simultaneously populate two components on the canvas – Create Table component and an Azure Blob Storage Load component, with their properties configured just how the generator set or guessed them. This generator tool is the simplest method for bringing data from Azure Blob Storage into a table in Snowflake.
New Queue listening and writing
The Azure Queue Listener brings a new level of automation to Matillion ETL for Snowflake users on Azure. This feature allows Matillion to run jobs dynamically based on events, instead of just on a fixed time-based schedule. The most common use case for customers is to set up Matillion to listen for when a new file arrives so it can be immediately loaded.
From the Manage Queue Configuration, you have three separate Queues: Listen, Success, and Failure. It’s highly recommended that users use a different Queue for each of these functions. Once configured you will receive your Success or Failure message corresponding to the configured Queue for each.
From your Azure portal in within your Storage account, you will be able to see you created queues in the Queue Service.
What’s new in Matillion version 1.39?
Check out our release video by David Langton, Product Director, for a guided tour of Matillion version 1.39.