Unveiling Data Observability with Kensu

Promising richer insights from unprecedented volumes of data, data teams are adopting data observability like never before. Kensu, an integral player in the realm of data observability, partnered with Matillion, a leader in data productivity. Preventing unreliable data from reaching end-users, and accelerating data incident troubleshooting to aid data teams.

Following the launch of this partnership, Laura Malins, VP of product at Matillion, met with Eleanor Treharne Jones, CEO of Kensu. Discussing all the things data observability, partner strategies and the essence of data productivity. Read on to explore the Kensu solution’s distinct features, the strategic partnership between Kensu and Matillion, and how the two complement one another in values and benefits for their customers. 

1. You lead Kensu, a 360° Enterprise Data Observability organization. How do you define Data Observability?

As data environments grow in complexity, it is becoming much more difficult for data teams to diagnose and troubleshoot data flaws. Data incidents lead to painful breakdowns and frustration between data practitioners and their end users. With Data Observability, data practitioners are given insights into the critical points within their data pipelines – allowing them to pinpoint the incident’s location and prevent it from recurring.

2. We’ve been hearing about Data Observability a lot lately as a data management trend to watch. How does Kensu stand apart from other Data Observability solutions?

Data Observability is a fast emerging space but the term can be used to describe a broad range of different solutions making it confusing for those new to the space. What separates Kensu is our platform’s ability to monitor data in-motion at the application level, which means data teams can monitor issues in close to real time and be proactive about detecting data incidents and preventing flawed data from reaching end users.

With Kensu, data practitioners are able to detect exactly what data incident has occurred, where in the pipeline it has occurred, how to fix the error, and how to prevent the error from reaching end users.

3. Can you explain the key objective of the strategic partnership between Kensu and Matillion?

This really comes down to empowering customers to confidently use their data to drive their business forward. While Matillion allows Data teams to take raw data from all of their business systems and quickly transform it into valuable business insights, Kensu ensures that teams can move faster, confident in the knowledge they have the right data checks and balances in place. By partnering to meet these needs up front rather than through sometimes painful lessons over time, we can provide even more data productivity for our joint customers.

4. What is complimentary about Matillion and Kensu for this partnership?

Both of our companies were formed after our Founders individually struggled to find the solutions they needed at the time, subsequently creating a better method and setting out to share their breakthrough with others. While Matillion has been revolutionary in the wide adoption of ELT architectures and low-code data development, Kensu is now paving the way in Data Observability. With joint headquarters in the US and Europe (well ok - the UK!) we have found a natural fit from a culture and values perspective.

In terms of the value provided to our customers, Kensu and Matillion are aligned on our mission to advance data practitioners’ data stack through trustworthy data. Both solutions offer low code, easy to start environments that allow for quick deployment and easy adoption.

5. What specific value does Kensu's 360° Enterprise Data Observability provide for current Matillion users?

Built based on customer demand - Kensu is the first solution to bring data observability capabilities to Matillion users. The Kensu platform gives users complete visibility over their pipelines and the ability to stop bad data hurting the business.

Kensu’s integration with Matillion requires no additional development to set up and instantly identifies all interdependencies between datasets and processes. Verifying that data moving through Matillion jobs with Kensu is a matter of clicks rather than time-consuming test development, and the circuit breaker integrates directly with Matillion’s drag and drop interface making all this additional confidence very easy to achieve.

6. Can you elaborate on how Kensu retrieves valuable context and information about data runs and sources from Matillion?

It’s a four step process.

1. During Data Retrieval, the Kensu Collector retrieves the latest Matillion ELT API job run data at regular intervals, ensuring observations are up to date and captures the most recent changes in users’ Matilion data pipelines.
2. This step is executed in conjunction with Data Processing, where the Kensu Collector extracts relevant information from the job run data, such as data stores, lineage details, and job information.
3. Once Data Processing is complete, the next step, ‘Snowflake Querying,’ begins, where the Kensu Collector retrieves additional metadata, such as column names, data types, and a set of metrics (ie: missing values, number of rows).
4. During the final step, ‘Aggregation and Sending,’ the Kensu Collector organizes the observations and aligns them with the appropriate job runs, data sources, and associated metadata. This data is then sent to the Kensu platform, where it can be utilized for data observability.

7. How are Matillion and Kensu’s values complimentary to help organizations become more successful?

Both organizations recognize the importance of safeguarding the soaring volumes of data enterprises interact with on a day-to-day basis. Our customers are our priority and both Kensu and Matillion are aligned in our commitment to equipping them with the solutions to bring trustworthy data for their end users.

8. Tell us about Kensu’s experience working with Matillion to develop this integration?

I have greatly enjoyed working with the Matillion team and the true collaboration we received from both the partner and the product team in the early development of the solution was a huge factor in the speed and priority we gave to this initiative and the effectiveness of the integration.

9. What does Data Productivity mean to you?

Data is extremely valuable to companies, but in order to extract that value, businesses need that data to be combined, cleansed, often translated into formats that the business can use, and even then to be truly valuable it also needs to be timely and most importantly trust-worthy. Data Productivity is taking all of the disparate and messy data that businesses are “storing” and making it actionable; taking what used to be an operational expense and turning it into an asset.