Find out what’s new in the latest releases of Matillion ETL for BigQuery.

10-12-2018 [1.36] New data load components, UX improvements, autocompletion prompts and more
  • A selection of new components to connect to various services:
  • ORC and PARQUET file formats now supported in Cloud Storage Load.
  • Selected Environments are now user-specific. Users can now specify their environments independently of one another.
  • Many UX improvements including automatically connecting components on the canvas, improved variables workflow and new keyboard shortcuts.
  • Users can now freely copy, cut, and paste jobs within a project.
  • Autocompletion prompts now appear in many places when using Matillion ETL variables in code.
17-10-2018 [1.35] Data Transfer Component, New Data Load Components, Extract to New Job Functionality and More

  • New “Data Transfer” Component that boasts all the functionality of the existing S3 Get, S3 Put and Cloud Storage Put components, plus additional source and target destinations (Azure Blob Storage)
  • In addition to AWS and GCP credentials, environments can now reference Azure credentials to interact with Azure services such as Blob Storage
  • A new “Apache Hive Query” component connects to your Apache Hive data warehouse
  • A new “LinkedIn Query” component connects to your company’s LinkedIn apps
  • A new “Bing Search Query” component connects to the Bing Search API
  • A new “Bing Ads Query” component connects to the Bing Ads service
  • A new “Dynamics 365 Sales Query” component connects to the Dynamics 365
  • Allow uploading the native Microsoft SQL Server JDBC Driver (the bundled jTDS driver is often the fastest in scenarios where it works)
  • New ‘Extract To New Job’ function available by right-clicking a selection of multiple components on the canvas. Allows users to instantly create new jobs from a group of components, tidying up workflows and helping to create reusable jobs
20-08-2018 [1.34] Shared Jobs, Historic Task Viewer, Unconditional Connector, Auto Debug and More
  • Shared Jobs:
    • You can now turn your reusable orchestration jobs into their own components with their own parameters, help and Icon
    • Shared jobs can be packaged and distributed across multiple ETL instances with Import and Export
  • Historic Task Viewer:
    • Previously completed tasks can be viewed on the canvas along with any parameter errors
    • You can understand the canvas state of a job and also see the jobs contained in a Shared Job
  • An “Unconditional” connector:
    • It’s now simpler to build orchestrations where the next orchestration step is run regardless of the success or failure of the prior step. This avoids the use of extra “and” and “or” components  to achieve the same thing
  • Auto Debug” for all Data Loading components:
    • Data Loaders come with the Auto Debug property. When switched on, allows users to choose between 5 levels of Debug Logging verbosity
    • Makes it easier to retrieve logging information without console access to the Matillion ETL Instance. Include these logs in your support requests for much faster turnaround!
    • Warning: Can potentially consume large amounts of disk space. Do not leave this switched on unless directly in need of it!
  • It is now possible to import, export and modify permissions via the API
  • Users can select components within an orchestration job and ‘Extract To New Job‘ via the context menu, creating a new job from the selected components. A link to the new job (via a Run Orchestration component) replaces the selected components in the old job
  • Password and secrets used in orchestrations and elsewhere can now be encrypted using Google Cloud Key Management Service (KMS)
  • BigQuery no longer creates temporary views during validation
02-07-2018 [1.33] Grid iterator, SQL editor, user management, Open Exchange Rates API and More

 

  • Open Exchange Rates Query component connects to the Open Exchange Rates API.
  • Grid Iterator allow iterating the values of a Grid Variable, similarly to iterating through a table of values.
  • SQL Editor (in all Query components) now shows available Tables/Columns and Variables to help you author and test SQL queries from source systems.
  • A new “Notices” V1 API endpoint allows you to query the current system notifications and post new messages which notify all users.
  • New “User Configuration” and “Permission” V1 API endpoints allow user management via the Matillion API.
  • Matillion no longer requires “listAllBuckets” permission (although this is still recommended).
  • Job Variables (scalar and grid) now have a “Visibility” that determines how they are used elsewhere.
  • All variables now have a description.
  • 100+ bug fixes across all areas of Matillion ETL.
  • Trigger the execution of Orchestrations jobs via Pub/Sub messaging via a new Queue Listener feature
17-04-2018 [1.32] Grid variable components, improved matching, SQL script component, new data load components and more

 

  • Zendesk Query orchestration component for loading data from the Zendesk customer relationship system
  • Mixpanel Query orchestration component for loading data from Mixpanel product analytics system
  • Xero Query orchestration component for loading data from the Xero accounting system
  • Dynamics 365 Query orchestration component for loading data from Microsoft Dynamics CRM/ERP
  • API Profile RSD Generator
    • Accelerate the development of API Profiles using a new tool that automatically generates a basic XML “RSD descriptor” for any API endpoint, based on a sample of data returned
  • REST API Version 1 – Matillion ETL now has full API coverage:-
    • You can now read/write more assets (JDBC Drivers, credentials, SQS configuration) as well as allowing finer-control of which resources to include
    • A map of the v1 API is available here
    • The “v0” api is still available and unchanged
  • Grid Variables System
    • In addition to “scalar” (single-valued) variables, you can now define grid variables to hold lists and grids of values; use them wherever a compatible list or grid of values is required
    • Grid variables can be manipulated/modified in Python
    • You can pass values for grid variables when starting a job via SQS and/or the V1 API
  • You can now disable parts of an Orchestration job
  • Improved Matching in column mappings – Many transformation component “Column Mapping” parameters can now be automatically mapped, even when the input and output column names are similar but not identical
  • A new SQL Script component allows you to run your own SQL scripts
20-02-2018 [1.31] API profile generator, grid variables, improved matching, new data load components and more

 

  • Zendesk Query orchestration component for loading data from the Zendesk customer relationship system
  • Mixpanel Query orchestration component for loading data from Mixpanel product analytics system
  • Xero Query orchestration component for loading data from the Xero accounting system
  • Dynamics 365 Query orchestration component for loading data from Microsoft Dynamics CRM/ERP
  • API Profile RSD Generator
    • Accelerate the development of API Profiles using a new tool that automatically generates a basic XML “RSD descriptor” for any API endpoint, based on a sample of data returned
  • REST API Version 1 – Matillion ETL now has full API coverage:-
    • You can now read/write more assets (JDBC Drivers, credentials, SQS configuration) as well as allowing finer-control of which resources to include
    • A map of the v1 API is available here
    • The “v0” api is still available and unchanged
  • Grid Variables System
    • In addition to “scalar” (single-valued) variables, you can now define grid variables to hold lists and grids of values; use them wherever a compatible list or grid of values is required.
    • Grid variables can be manipulated/modified in Python
    • You can pass values for grid variables when starting a job via SQS and/or the V1 API
  • You can now disable parts of an Orchestration job
  • Improved Matching in column mappings – Many transformation component “Column Mapping” parameters can now be automatically mapped, even when the input and output column names are similar but not identical
  • A new SQL Script component allows you to run your own SQL scripts

 

Important (possible breaking change): API Profiles (“RSD’s”) that handle paging may need to be tweaked to disable “auto” paging. Please see here for more details.

Important (possible breaking change): API profile limits are now applied. Where the default of 100 is set it will now be applied. This could affect API Query Components which previously ignored that limit.

23-11-2017 [1.30] Improved Task Info Panel, Cloud Pub/Sub Component, New Data Load Components and more
  • New Cloud Pub/Sub component can send messages to PubSub during job runs.
  • New Cloud Storage Load Generator can inspect files on Cloud Storage and generate Create Table / Cloud Storage Load components.
  • Redesigned “Scheduler” user interface to simplify the management of scheduled orchestration jobs.
  • New “Task Info” panel and “Task” panel make it much easier to understand complex tasks both at run time and after job execution.
  • Matillion variables can be defined and scoped at job level making jobs much more reusable. Variables can now be passed to and returned from jobs.
  • New Quickbooks Online Query component to connect to the popular online accounting system.
  • New Square Query component to connect to the payment system.
  • New Google Custom Search component allows google search data to be ingested.
  • All data-staging components can append rows to an existing table as well as creating new tables.
05-10-2017 [1.29] New Data Load Components, Cloud Storage File Browser Enhancements, Advanced Connection Options and more
  • New Jira Query component loads data from Atlassian’s popular Software Development Platform.
  • New PayPal Query component can load payment and other data from Paypal Business accounts.
  • New ServiceNow Query component loads data from Servicenow’s IT Service Management (ITSM) platform.
  • New Stripe Query component loads data from Stripe’s payment platform
  • New Email Query component can query an IMAP based email system.
  • New YouTube Analytics component can query data from the YouTube Analytics API.
  • All query components now allow you to override the output table so you can specify an existing table to load or append to.
  • Excel Query can now load files from Google Cloud Storage, as well as Amazon S3
    • You only see S3 and/or GCS when you have credentials in the environment, otherwise they are hidden.
  • New option to drop a schema from the Environment Tree.
  • Specify a region in S3 Unload (to allow writing to buckets outside of the Redshift cluster’s region)
  • S3 / Google Cloud Storage file browser enhancements
  • Set advanced connection options during OAuth flow (e.g. to connect to a Salesforce Sandbox)
  • Warning: Manage Backups and View Audit haven’t been removed, they have been moved to the Admin menu
  • Map Values component
05-10-2017 [1.28] New Admin Menu, Table Update Component, Python Scripts Support, Enterprise Features and more
  • Improvements to Partitioned Tables support
  • You can create external tables over Google Drive files and Google Sheets
  • New Table Update component can update existing table in a transformation flow without rewriting all data
  • New iteration component Table Iterator allows you to iterate over an existing BQ dataset
  • Added support for Python scripts (Jython, Python2 and Python3) as part of Orchestration flows
  • New Except and Intersect Transformation components to compliment the Unite (Union) component
  • Enterprise Features
    • Automatic Job Documentation. Matillion ETL can automatically generate documentation for your ETL process, this tool will recursively search all your jobs and include all job detail including linked notes and descriptions
    • Auditing of User Actions with searchable Audit Log provides fine grained audit of every change to an ETL process
  • Database Query now supports IBM Netezza data warehouses via JDBC
  • All transformation components support multiple outputs.
    • Separate Replicate component not required
  • A new Admin Menu allows administrators to:
    • Access the server log
    • Apply updates to the Matillion server without using the command line.
    • Add, remove and configure users (using either an internal user database or external directory server).
    • Configure SSL