Now Available On-Demand | AI. The Future of Data Engineering - Today. Not Just Tomorrow.

Watch now
Back to all
Engage3 Logo 2
Matillion Emerald Awards Solid Logo Green Navy TRANS Background WINNER

Engage3 Automates Change Data Capture with Matillion

Schedule a demo
Engage3 Banner

Data

ingested in minutes

instead of hours

95%

less time required to make

new insights available downstream

Complete

your data pipeline

with Matillion CDC and ETL

The Challenge: A robust and reliable data platform

Based in California, Engage3 has revolutionized retail pricing for consumers, retailers, and manufacturers with its Price  Image Management platform. This Software-as-a-Service  (SaaS) solution uses carefully curated pricing data, alongside data science and predictive analytics based on Nobel prize-winning economic models, to provide timely, accurate insights to customers. Accordingly, it has become the industry’s largest competitive pricing database.

Giving customers the best competitive insights requires feeding the most up-to-date pricing data into Engage3’s systems. Some of the main inputs are from web scraping,  which is continually being accumulated. Keeping this platform up to date with the latest information is vital to ensure the quality and accuracy of Engage3’s recommendations. The company needed a simple, reliable way to quickly and continuously acquire and apply large volumes of pricing updates. To address this need, Engage3 turned to  Matillion Data Loader Change Data Capture (Matillion CDC) for its secure, no-code data pipeline automation and Matillion ETL  to dramatically reduce the time to load changing pricing data and make it available for analysis.

The Solution: Simple and effective Change Data Capture

Using Matillion Data Loader CDC, Engage3 has made its change data capture processes simple, fast, and accurate.  The company uses Matillion CDC to read data updates from their PostgreSQL database, and apply the changes to its target cloud data platform in near real-time. Matillion CDC  easily handled the volume of data, replicating every change event in 26 tables, five of which were updating 100 - 500  million rows per day.

As a hybrid SaaS solution, Matillion offers benefits in terms of efficiency and security. A local CDC agent runs in a container within the Engage3 private cloud and utilizes a cloud secrets manager for access credentials to the source and destination.  The agent monitors PostgreSQL change logs and streams change events into the Engage3 data lake, where it becomes an immutable history of all change events. Because the CDC  agent runs within the Engage3 private cloud, no data or access credentials are ever exposed externally.  This configuration is optimal for querying and taking advantage of AVRO, which is a standard open data format.  Thanks to this level of organization, Matillion CDC makes it possible to recreate an exact view of data at any point in time.

Once the raw data is in their data lake, Engage3 uses Matillion  ETL to transform the AVRO files and load analytics-ready data into their Snowflake warehouse in near real time. This end-to-end data pipeline extracts, loads, and transforms data to create timely and accurate insights for all downstream processing.

Benefits: The best competitive insights in the world

Now that Matillion Data Loader CDC and Matillion ETL have been embedded into Engage3’s operations, the fast,  incremental data feeds are the foundation of its value chain.  With Matillion in place, Engage3 is able to continuously bring in new data without friction, including extremely “wide” tables with many columns and hundreds of millions of rows.  “Scale and loading performance are no longer an issue for us,”  says Matt Belo, Senior Data Engineer at Engage3. “Overall the  solution has helped bring about a 95% reduction, from hours  to minutes, in the time required to deliver new pricing data for  downstream analytics.”

Scale and loading performance are no longer an issue for us. Overall the solution has helped bring about a 95% reduction in time required to make new insights available downstream. Matt Belo Senior Data Engineer| Engage3

What's next

Following the success of the project, Engage3 is planning on rolling out more CDC agents in 2023 to support four additional business areas. These agents will extract and load change data from their other data sources to power additional downstream data processing across the business.

About Engage3

Price optimization pioneers Ken and Tim Ouimet created  Engage3 to revolutionize the way brands and retailers engage with their shoppers. Engage3 provides the industry’s most comprehensive omnichannel intelligence solution, combining best-in-class AI-enabled web crawls, professional in-store audits, and self-serve in-store and online data collection into a single source of truth. Leveraging advanced algorithms and the Nobel Prize-winning Efficient Frontier theory, Engage3  helps retailers and brands track and optimize their Price Image to profitably accelerate revenue growth, drive incremental store trips, and increase margins.  

Engage3 serves more than 300 customers in 182 countries,  supporting $2 trillion in global retail revenue. Engage3’s leadership team is composed of former executives from  KhiMetrics, Mozenda, SAP, Revionics, dunnhumby, KSS Retail,  IBM/DemandTec, and Nielsen. Engage3 is headquartered in  Davis, California with 6 additional offices around the world.