Diagnosing data issues with confidence: Introducing End-to-End Lineage in Matillion

To make confident decisions, you need to trust your data. But how can you trust something if you don’t know where it came from or what’s happened to it along the way? Data lineage is the key—it gives you a clear view of your data’s journey, from its origins to its final destination, so you can be sure it’s reliable.

In this blog, we’ll take you through a real-life example from our latest product walkthrough. Using Matillion’s Source Lineage capabilities, we’ll show how to trace data, pinpoint a problem, and fix it—all while solving a ThoughtSpot dashboard failure. With the power of Matillion’s Data Productivity Cloud (DPC), you’ll see how easy it can be to troubleshoot and keep your data on the track. And don’t miss the demo at the end of this blog, where we’ll bring it all to life by showcasing Matillion’s End-to-End Lineage in action. Let’s dive in!

Diagnosing a dashboard failure

Picture this: you're using ThoughtSpot to visualize data-driven insights, but a critical dashboard fails to display data correctly. What do you do?

The first step is identifying the source of the issue. By inspecting the dashboard details in ThoughtSpot, we can locate the Snowflake view powering the data. This view becomes the starting point for tracing the problem upstream.

Mapping the data flow with DPC Lineage

With the Snowflake view identified, we turn to Matillion Lineage to explore the flow of data. Here's how we uncover the entire path:

1. Search for the Data Source: A quick search in the Lineage tool reveals the connections between the Snowflake view and other upstream transformations.
2. Visualize the Lineage Map: The tool instantly generates a clear, interactive diagram showing how data flows from Jira ingestion pipelines, through Snowflake transformations, to the dashboard.
3. Identify Relationships: Beyond the broken dashboard; the map also highlights additional views and tables feeding other business-critical dashboards in ThoughtSpot.

This bird's-eye view ensures full traceability, making it easier to pinpoint potential problem areas.

Diagnosing the Transformation Pipeline

Next, we zoom in on the transformations within Snowflake that prepare the data for the failing view.

1. Inspect Transformation Pipeline: Each transformation pipeline is visually represented in the Lineage tool. Users can click into these nodes to examine the pipeline Metadata along with the last time this pipeline was executed.
2. Identify broken pipelines: By reviewing the pipeline's run history, we spot a recent successful transformation step that, despite appearing functional, isn’t producing the expected output.

This level of granularity ensures no step in the data flow goes unchecked.

Fixing the issue and verifying the solution

Once the problematic transformation logic is identified, we can apply the fix directly in the Designer and re-run the pipeline. After verifying that the updated pipeline operates correctly, we return to the ThoughtSpot dashboard—and success! The data now displays as expected.

This swift, end-to-end troubleshooting process showcases the power of Matillion’s Source Lineage to reduce downtime, enhance data trust, and simplify problem resolution.

Why Source Lineage Matters

Here’s how Source Lineage in Matillion transforms the way teams work with data:

• Full traceability: From the source system to the final output, gain visibility into every transformation and process along the way.
• Streamlined troubleshooting: Quickly diagnose and resolve issues to minimize disruptions to data-driven workflows.
• Enhanced compliance: Maintain detailed records of your data’s journey to support audits and regulatory requirements.

The Future of Lineage with Matillion

This is just the beginning. With Source Lineage in Matillion’s DPC, you can empower your teams to make better decisions faster and with full confidence in their data.

We’re excited to announce that we are iteratively rolling out source lineage across all our connectors. With source lineage, you’ll gain even deeper visibility into your data workflows by tracing data back to its origin systems, such as APIs, SaaS platforms, or databases.  

This feature will be released incrementally as we update and enhance our connectors, meaning source lineage will start to appear as connectors are updated. This approach allows us to deliver value quickly while continually improving and expanding our coverage.  

Stay tuned for updates as new connectors and enhancements are rolled out, bringing even greater transparency and control to your data ecosystem.