Data Lineage for Beginners

Big terms in the world of data can often feel intimidating, especially when they’re frequently mentioned but not fully understood. One such term is data lineage. If you’ve ever wondered what it means or why it’s important, this blog will explain it. We’ll not only define data lineage in simple terms but Lee Power, Senior Product Manager at Matillion, uses a great analogy that relates data lineage to something we all understand—grocery shopping. But before we dive into that, let’s start with a definition.

What is Data Lineage? 

Data lineage refers to the complete journey that data takes from its origin to its final destination, detailing every transformation and process it undergoes.

It allows us to answer some core questions, such as:

• Where did this data come from? 
• How was it transformed? 
• What processes did it pass through? 
• Where does it end up? 

Now that we have covered the definition, lets read the analogy! 

Data Lineage - A Grocery Analogy

Imagine you’re tracking fresh groceries from the farm to your kitchen table. Then you can choose to do something with it such as make a meal. Think of the data as the groceries 

In this scenario:

Source (The Farm): This is where the journey begins. Just like raw ingredients are produced at farms, data originates from sources like databases, APIs, or files. 

Transport (Logistics): The logistics of moving groceries involve trucks that go specific routes. In data lineage, this is equivalent to data pipelines that transfer data from one system to another. 

Processing and Transformation (Distribution Centers and Processing Plants): Groceries go through processing centers where they might be cleaned, sorted, or packed. Similarly, data gets transformed in various stages - cleansed, aggregated or modified as it flows through the data pipelines. 

Destination (Grocery store or your kitchen): Finally, the groceries reach the store and then your kitchen, where you use them to make meals. In data terms, this is the point where the data is ready to power dashboards or reports and is used for analysis or decision-making.

Why is Data Lineage important? 

Let’s say you made a vegetable stew, and it has a bitter taste. How would you check?

1.Check the stew

• You realize the stew tastes off, so you need to find the source of the problem 

2.Analyze ingredients

• You look at each ingredient - carrots, potatoes, onions and identify that the carrots have an odd flavour. 

3.Trace the source

• You could trace them back to the supplier and find that they came from a farm that had an issue with its soil quality affecting the taste. 

Let's say you made a Dashboard, and you notice a problem, how would you check?

1.Check the data

• You notice an issue on the dashboard. You need to find the source of the problem 

2.Analyze pipelines

• You trace the data’s path backward through transformations and pipelines all the way back to the source data. 

3.Trace the problem 

• You find that a data field from a specific database table had missing or incorrect values due to an outdated script of a data quality issue. 

Key takeaways: 

It helps teams quickly identify where errors or inconsistencies come from, ensuring that the data being used for decision-making is accurate and reliable. 

Just like tracing a bad ingredient helps improve future meal quality, tracing problematic data allows teams to fix and prevent future issues, maintaining trust and efficiency in their data processes. 

Ready to learn more? Here’s some useful further reading:

1. Diagnosing data issues with confidence: Introducing End-to-End Lineage in Matillion 
2. The Value of End to End Data Lineage, Especially for AI
3. Data Change Management – Iterate Rapidly and Track Lineage