The inability of enterprises to extract and ingest data from a growing number of data sources is blocking their ability to make critical decisions. There’s a simple solution: Make it easy for more users to extract and ingest data so they can analyze it and make confident decisions faster.
Matillion Data Loader does exactly that. It empowers individuals across the business to quickly and easily load data so they can start using it right away, instead of waiting days, weeks, or even longer for another team to load it for them.
Through an intuitive, SaaS-based user interface, users can create both batch and change data capture (CDC) pipelines without coding or needing the deep technical skills of data engineering. Data engineers also love Matillion Data Loader – it makes them more productive so they can tackle higher value initiatives rather than waste time coding and maintaining pipelines or manually moving data.
Batch + CDC: A unified experience
Data is most often ingested in batches on a set schedule like once per day or once per hour. Data can also be ingested via change data capture, which streams data changes continually as they happen. There are a lot of data integration tools in the market that help you load batch data, but very few that also take on CDC pipelines.
Matillion Data Loader provides both batch and CDC. In a single, unified interface, Matillion Data Loader eliminates the burden on users of learning coding skills or multiple, complex data integration tools from different vendors. Users select the type of pipeline they want to create, choose the data source and tables to replicate, specify the data destination, and set the frequency of their loads. Without coding, users can have a production-quality pipeline up and running in minutes. It’s as quick and easy as that!
But combining batch and CDC in a single tool is only one benefit of Matillion Data Loader. There are many methods of doing change data capture, but the one method that stands above all others is log-based CDC. Log-based CDC delivers every change event in near-real time, is more efficient, and reduces the data extraction burden on the source database. This is one of the most in-demand capabilities in enterprise data integration, and it’s an integral part of Matillion Data Loader.
Log-based CDC: Never miss a change
Why is log-based CDC so critical to enterprise organizations? Many databases write low-level transaction logs as changes occur within the database. These logs are crucial for backup, disaster recovery, regulatory audits, fraud detection, and many more scenarios. Matillion Data Loader hooks into these transaction logs and ingests all change events as they happen. The changes are enriched with metadata (data about the data) and loaded into your cloud data platform. This raw data is a complete history of all change events.
Matillion Data Loader’s CDC capabilities go beyond snapshot techniques that capture data only at specific time intervals. Periodic snapshots can miss multiple changes to a data element between the snapshot windows. And adding timestamps or version numbers to database rows still does not address the issue of multiple changes occurring between the snapshots. Log-based CDC extracts and ingests every change event, as they happen, so it never misses a change. The raw data that comes in via Matillion Data Loader can provide an exact view of the source data at any point in time.
Batch + CDC Loading + Transformation: The ELT secret sauce
Combining Matillion Data Loader with Matillion ETL’s best-in-class data transformation creates a complete end-to-end data integration solution for extracting and loading raw data, transforming it into a meaningful analytics asset, and synchronizing that out to all who need it, at speed and at scale. This is cloud ELT (extract, load, transform), not ETL (extract, transform, load).
Users create, manage, and monitor both batch and CDC pipelines in one easy-to-use interface. Batch pipelines ingest data directly into a cloud data warehouse at set intervals. CDC pipelines stream change data into cloud storage in near real time. In both cases, Matillion ETL uses the compute power of the cloud to transform the raw data into a useful resource and sync it out to cloud data warehouses, analytics programs, AI/ML models, and SaaS applications where it can be put to use. This end-to-end ELT process is unique, and sets Matillion apart from other vendors offering only pieces and parts of the full picture.
Unlock a brand new world of data and insight
Imagine what your organization could accomplish by empowering more users to extract and load data in near-real time. Imagine what higher-value initiatives your data teams could work on by off-loading the time-consuming tasks of data integration. Imagine making more confident and faster business decisions, and the impact this could have on business outcomes.
By unifying batch and CDC data loading in a single tool, utilizing log-based CDC, and leveraging Matillion ETL for data transformation and syncing, you can open up a whole new world of use cases like fraud detection, marketing personalization, recommendation engines, compliance auditing, AI/ML modeling based on current and historical data, and so much more.