Matillion and Snowflake: Powerful ETL with No-Code/Low-Code Ease of Use

Organizations of all sizes recognize the need for analytics, and tools such as PowerBI and Tableau have made it possible for any business to implement data analysis for efficiencies in operations, sales, services, and product development. Yet, many businesses are still facing problems in gathering all of their data from the various sources used by each department and getting that data ready for analytics. Straightforward transformations to normalize and cleanse the data, such as standardizing date formats and order of fields or eliminating duplicate records, slow down the integration of data from its sources to cloud data platforms such as Snowflake that are optimized for faster analytics at scale.

Fortunately, ETL (Extract-Transform-Load) tools are now widely available to automate this process. The question for many businesses is how to choose the right ETL tool that provides them with the power and customization they need but without the difficulty of custom programming. Fortunately, tools such as Matillion provide the power of custom-programmed tools with the ability to perform complex transformations in a no-code/low-code environment. 

No More Tradeoffs: Low Code Doesn’t Mean Low Power

No-code solutions are available for many business tools today from databases such as Airtable and integration tools such as Zapier. While many companies are successfully using these tools, they’ve also gained a reputation for being much less flexible than the fully-customizable alternatives which require sophisticated programming for full customization.

When it comes to ETL, it’s obvious that companies need data connectors to a large number of different data sources, from on-premises databases to SaaS applications to file systems, to get their data into Snowflake. With so many different data sources, connecting to all of the sources, loading the data, and transforming it can be very time-consuming with many companies using bespoke coding to customize these connectors and workflows. However, it turns out that these same outcomes can be accomplished with no-code and low-code solutions. Just as website building tools have evolved to combine simplicity and full functionality over the years, so have ETL tools, with Matillion being a great example. 

Today’s  no-code/low-code ETL tools, with extensive data mapping and lineage capabilities, visual drag and drop interfaces for building data pipelines and complex transformations, and automation and orchestration of workflows, provide all of the same, if not better, functionality than legacy code-based solutions, but without the complexity.

ETL with Code Customizing: Costly Complexity

Like with most technologies, the first ETL systems on the market required a tremendous amount of coding and customization. While the data analysts were saving as much as half of their time by eliminating the data cleansing tasks, the data engineers needed to slog through a slow integration process and companies had to dedicate programming resources for the transformation of their data into the formats needed for analysis in their data warehouses and data lakes – initially on-premises data warehouses and more recently into cloud data platforms such as Snowflake – as well as for processing by BI tools. 

Even worse, all of these snippets of code written in Python or Javascript were customized separately for each database and data source, with the result being a collection of different pieces of code written at different times for each workflow. That code needed to be maintained, and if the original coder moved to another job, maintaining the connectors between the data sources and the ETL tool became a complex puzzle for the next person to solve and manage.

How It Works: Data Import and Drag-and-Drop Workflows

Matillion implements a combination of no-code, low-code tools and automation to build data pipelines and perform complex workflows that allow fit-to-purpose customization of the extraction of data from its sources, loading of data to a cloud data platform, and its transformation within that platform to fit an organization’s specific analytics needs.

The data import tool, shown below, which extracts the data from its source and imports it to the chosen cloud platform, shows the data structure in an easy-to-read human format (figure below shows structured data in JSON format). The user can re-label, re-order, and customize the data transformation and processing using a simple interface, saving time. As shown below, a built-in Java-based coding tool is also available in case a deeper customization is needed. Any coding is very lightweight and saved as part of the workflow, making it easy to manage by the next user.

Similarly, creating connectors to databases for data extractions and loading is simple with a large number of built in connectors as well as the ability to build custom connectors with Matillion’s drag-and-drop data pipeline and workflow tool. Matillion includes more than 100 pre-built data connectors from popular data sources. More importantly, Matillion’s  connector creation tool gives the user complete control for connecting and integrating non-standard databases and for creating workflows that are tailored to the needs of the organization. Similarly, for the loading of the data, Matillion includes built-in connectors to Snowflake and other popular cloud data platforms.

Matillion makes it easy even for small businesses with limited resources to create extremely powerful ETL connectors and flows for BI that can compete with the big guys. That’s not to imply that the big guys don’t use Matillion – many major global enterprises also use Matillion ETL with Snowflake. At the same time, Matillion’s pay-for-use pricing model makes it ideal for any size of organization.

Implementing Data Warehousing Best Practices

In terms of loading data to a company’s cloud data warehouse, Matillion employs best practices and includes connectors to most popular data sources and all leading cloud data platforms, including Snowflake. Companies using Matillion can leverage the native features of Snowflake directly from Matillion’s own easy-to-use visual interface and take advantage of Snowflake’s processing power through push-down integration (the processing is done within a company’s own Snowflake environment, taking advantage of the power, scalability, and flexibility of the cloud). As a result, companies are able to leverage Matillion ETL without losing any of the platform-specific functionality they’ve already invested in with Snowflake or other cloud data platforms.

Rapid Implementation Made Simple

Getting started with Matillion ETL is simple. As a cloud-native solution, Matillion is easy to deploy in an organization’s existing VPC instance (AWS, Azure or Google Cloud) and doesn’t require any local on-premises infrastructure or dedicated  resources. Furthermore, most of the necessary connectors and integrations to data sources are already built into Matillion ETL. With Matillion, there are no long-term contracts or retainers to sign. Simply use what you need and pay for what you use. 

Matillion doesn’t hold or store any data, and the data transformation workflows themselves are also stored in your company’s cloud data platform. In other words, Matillion maintains your IT security infrastructure intact, with no additional attack surface.

Power and Simplicity with Matillion

Modernizing your business intelligence infrastructure can be straightforward. Companies no longer have to sacrifice power for simplicity when it comes to Extracting, Transforming and Loading data from their databases and other data sources into their cloud-based data platforms. Cloud-native Matillion ETL makes it easy to leverage the full power of ETL in a no-code/low-code cloud based data integration and transformation platform.

If you’re using Snowflake, Matillion’s no-code/low-code ETL can dramatically improve the time from data sources to Snowflake data cloud for all of the data in your organization. You won’t have to sacrifice any of the power or flexibility for the sake of speed or due to resource constraints and you will be able to glean actionable business insights in no time.  Schedule a demo to see Snowflake and Matillion in action.