Skip to main content

Replay: Driving Real-Time Data Capture and Transformation in Delta Lake with Change Data Capture

Published Date
Author Rob Callaghan


The ability to capture data change for analytics is paramount for data teams to deliver business-ready insights that can make a big impact. For use cases like fraud detection, real-time marketing campaigns, operational analytics, AI/ML modeling, and others, the ability to see, understand and act on data change is a game-changer.

Change data capture (CDC) is an increasingly common technology used in real-time machine learning and AI data pipelines. When paired with lakehouse technology such as Databricks Delta Lake, it is beneficial to organizations because it can lower data processing costs and enable highly responsive analytics applications.

In June, Matillion attended the Databricks Data + AI Summit where Paul Johnson, Director of Product Management, and Paul Lacey, Senior Director of Strategic Marketing, talked about change data capture (CDC) for Delta Lake. This approach extracts data changes in a source database and ingests those changes into cloud storage in near real-time, delivering insights without spending days learning new processes. 

Their presentation provided a detailed overview of Matillion’s new CDC capabilities and how the integration of these capabilities with Delta Lake on Databricks can help data professionals manage dataset changes, making it easy to automate the capture, transformation, and enrichment of data in near real-time.

 

Watch the presentation on CDC for the lakehouse

You can view the full talk, Driving Real-Time Data Capture and Transformation in Delta Lake with Change Data Capture, below. 

Find out how you can get to business-ready data at Matillion Data Unlocked

Getting insights in near-real time enables faster decision making for business leaders. Our first ever conference, Matillion Data Unlocked, will help uncover new routes to data productivity. Attendees will learn:

  • Insights from data leaders to build productive, successful data teams
  • New data strategies and tools that can help to make data integration and transformation faster
  • Better ways to unlock the full potential of your cloud data apps with practical workshops

Save the date for Matillion Data Unlocked on October 12. Learn more