How Engage3 uses Matillion Data Loader to automate data pipelines

Engage3 has revolutionized retail pricing for consumers, retailers and manufacturers with Price Image Management. This SaaS platform creates pricing insights using data science and predictive analytics based on Nobel prize winning economic models. The calculations use a set of pricing data that has been carefully curated over the last 40 years. It has become the industry's largest competitive pricing database. 

Keeping all this up to date with the latest information is vital to ensure the quality and accuracy of the recommendations. Engage3 needed a simple, reliable way to quickly and continuously acquire and apply large volumes of pricing updates. To address this need, Engage3 turned to Matillion Data Loader (MDL) for its no-code data pipeline automation.  

Change data capture automation

With Matillion Data Loader (MDL), Change Data Capture (CDC) is conceptually simple. MDL reads data updates from sources, and applies the changes into a target cloud data platform destination in near real time. 

For security and efficiency, Matillion Data Loader is a hybrid SaaS solution. A local CDC agent acquires data from the source system's change log. It is immediately transferred to cloud storage, where it becomes an immutable history. Because the CDC agent runs within the customer’s private cloud environment, no data or access credentials are ever exposed externally. 

The layout is optimal for querying, taking advantage of AVRO which is a standard, open format. Thanks to this level of organization, MDL makes it possible to recreate an exact view of data at any point in time.  

Optimizing Engage3's business operations

"Data drives the world," says Matt Belo, Senior Data Engineer at Engage3. 

Giving customers the best competitive insights requires feeding the most up-to-date data into Engage3's systems. Some of the main inputs are from web scraping, which is continually being accumulated.  

“Before deploying Matillion CDC, a full data refresh was needed almost every time. This was slow and not scalable, and was hurting the business.” Matt remembers it as a tremendous pain point: "It was noticeable because the data was needed hours ago."  

How Engage3 uses Matillion Data Loader to automate data pipelines

Now that Matillion Data Loader CDC has been embedded into Engage3's operations, the fast, incremental data feeds are the foundation of the value chain. With MDL CDC in place, Engage3 is able to continuously bring in new data without friction. This includes very "wide" tables with many columns and hundreds of millions of rows.  

“Scale and loading performance are no longer an issue for us,” Matt adds. “Overall the solution has helped bring about a 95% reduction, from hours to minutes, in the time required to make new insights available downstream.”  

What's next?

The raw data that Engage3 draws from various sources runs through several processing steps that increasingly add value for their users. After basic interpretation and formatting.

These steps include integrating the many data sources together. Matillion ETL is Engage3's code-optional productivity solution for data transformation and integration. 

Book an Engage3 demo to learn more about how you can take advantage of strategic pricing and grow revenue while retaining customer loyalty. 

Join Matillion’s weekly live demo or schedule a personalized demo for your team to learn more about Matillion Data Loader Change Data Capture.