The 1:10:100 rule of data quality: A critical review for data professionals

A significant shift has occurred since George Labovitz and Yu Sang Chang introduced the 1:10:100 rule of data quality in 1992. This rule underscored the escalating costs of poor data quality and has become more pronounced over time. 

This article explores the evolution of data quality management over the past three decades, the increasing financial implications, and the strategies necessary to navigate this challenging landscape.

What is the 1:10:100 rule of data quality? 

The 1:10:100 rule asserts that:

• The cost of preventing poor data quality at the source is $1 per record.
• The cost of remediation after data quality issues are identified is $10 per record.
• The cost of failure (i.e., doing nothing) is $100 per record.

This principle highlighted the escalating costs associated with data quality issues as they progress through an organization's data lifecycle. Fast forward to 2024, and the costs have multiplied significantly. Today, I believe the rule has evolved more into a 10:100:1000 paradigm:

• $10 per record to prevent data quality issues at the source.
• $100 per record to remediate issues after data is created.
• $1000 per record when data quality issues are left unchecked.

Why the shift from 1992 to 2024? 

In 1992, data systems were primarily on-premises, allowing greater control over data input and quality. Organizations could implement stringent rules and validations directly within their source systems. However, with the proliferation of Software as a Service (SaaS) applications, this level of control has diminished. Many SaaS systems offer limited customization, making it challenging to enforce strict data quality measures at the source.

Preventing data quality issues at the source: $10

Preventing data quality issues at the source is the most cost-effective approach. However, it has become increasingly challenging due to the rise of SaaS applications. For those fortunate enough to work with customizable systems, such as Salesforce.com or NetSuite, several strategies can be employed such as:

• Input controls: Implementing controls on input fields to ensure data accuracy.
• Restricted text fields: Avoiding open text fields or limiting their use to ensure structured data.
• Complex business rules: Building robust business rules into user interfaces to validate data before it enters the system.
• Data contracts: Enforcing data contracts to ensure that data adheres to predefined standards.

Remediation of data quality issues: $100 

While this figure is an estimate and may be high in some instances, the figure is a directionally accurate portrayal of business impact. When prevention isn't feasible, data teams must focus on remediation. This involves addressing data quality issues as data flows into data environments, such as datalakes or cloud platforms. Key strategies include:

• Data quality tools: Using ELT/ETL tools or specialized data quality tools to identify and correct issues.
• Governance platforms: Building data quality rules within governance platforms to enforce schemas and validate data values.
• Data observability: Implementing data observability platforms to monitor data as it enters landing zones, ensuring early detection and remediation of issues.

The high cost of failure: $1000

Failure to address data quality issues can be extremely costly. Poor data quality can lead to slow responses to data requests, as data engineers spend significant time cleaning and preparing data. Worse, bad data may be used unknowingly, resulting in flawed insights and decisions. The opportunity cost of bad data is substantial, affecting everything from operational efficiency to strategic initiatives. Check out Matllion’s recent ebook on Data Debt for more insights on letting data quality going unchecked in your organization.  

Conclusion

The evolution from the 1:10:100 rule to the 10:100:1000 rule underscores the growing importance of data quality management in today's data-driven world. For data professionals, it's crucial to implement robust data quality measures at every stage of the data lifecycle. Whether through prevention, remediation, or addressing failures, the costs associated with data quality issues can have a profound impact on an organization's bottom line.

By understanding and applying these principles, data engineers, analysts, and directors can ensure their organizations leverage high-quality data to drive accurate insights and informed decision-making. Investing in data quality upfront is not just a best practice—it's a strategic imperative.

Download Matillion’s Guide to Data Debt to learn more about why data debt grows over time and what you can do about it.