Intuitive UI for Debugging and Validation

Are you tired of spending hours debugging complex data pipelines? In this blog post, I'll explore how Matillion’s Data Productivity Cloud (DPC) offers a solution. Its intuitive UI and built-in validation features transform the debugging experience for users of all technical levels.

For those who come from a non-technical background, the DPC bridges the gap to the technical expertise needed to build robust data pipelines.

Personal Experience without DPC 

As a Data Engineer apprentice coming from a non-technical background, one of my greatest challenges was debugging pipelines—locating errors, understanding their root cause, and resolving them efficiently. Without a clear framework, debugging could feel like searching for a needle in a haystack, requiring hours of combing through SQL code to pinpoint issues. This process was both time-consuming and frustrating, especially when errors in one section of the pipeline cascaded into larger problems elsewhere.

Feature Introduction 

Matillion’s Data Productivity Cloud transformed my debugging experience. Its intuitive UI and built-in validation features provide immediate feedback, helping identify misconfigured components before running the pipeline. If a pipeline has unresolved issues, the system proactively prevents it from executing, offering clear error messages and actionable insights. This approach drastically reduces debugging time and minimises errors, making pipeline management more accessible, even for those with limited coding experience.

Matillion’s intuitive debugging and validation tools ensure that errors are detected and addressed early in the development process. Here are some of its standout features:

• Component Validation: Each component within a pipeline must validate successfully to ensure proper configuration.
• Pipeline Validation: The entire pipeline is validated before execution, preventing costly errors from propagating.
• Connector Feedback: Visual connectors between components display the pipeline’s status (success, fail, or continue).
• Partial Pipeline Execution: Users can run specific sections of a pipeline to test functionality and isolate issues.
• Error Messages: Clear and actionable error messages highlight misconfigured components and offer guidance for resolution.
• Data Sampling: Sampling components allow users to validate intermediate results at various stages of the pipeline, ensuring data accuracy.

Real-World Example

The pipeline below is designed to analyse customer travel habits. It performs the following steps:

• Load Data – Inputs data tables with information such as traveler demographics, exchange rates, and currency in USD.
• Transform Data – Segment data by age group and calculate spending costs.
• Data Cleansing – Splitting the city and country into separate fields.
• Currency Calculation – Convert costs into local currencies of the travel destination.

The pipeline is structured sequentially, with each component relying on the success of the previous one.

Before Validation:

Without validation, errors in one component could go unnoticed until the entire pipeline is run or fails. This wastes computational resources and makes debugging difficult, especially if the issue stems from a misconfigured job early in the pipeline.

In the example below, the Split Field component has been configured to divide the ‘Destination’ column into ‘City’ and ‘Country.’ Although the component is successfully validated, it is only when sampling the pipeline that users notice an error: the same value is being assigned to both ‘City’ and ‘Country’ (e.g., City = London, Country = London). By reviewing the output at the individual component stage, highlights the importance of the sampling feature for users of varying technical expertise, including low-code/no-code users. It enables a clear understanding of how and where the data transformation is incorrect.

After reviewing the configuration settings and adjusting the value positions for the output columns, the Split Field component is now correctly configured. The appropriate values are assigned to ‘Country’ for the corresponding ‘City’ (e.g., City = London, Country = UK).

With Validation:

Matillion’s validation tools ensure every component is checked before execution. A green tick signifies successful validation, while a red cross flags issues. Errors are displayed in the Task section at the bottom of the DPC canvas, where users can click on failed tasks for detailed error messages. For instance:

Scenario: A transformation component, Calculator, fails due to SQL syntax error for the Daily Cost Calculation. 

• Resolution: The system highlights the component with the error and provides a descriptive message, such as “SQL complication error: syntax error line 18 at position 43 unexpected ‘Daily_Trip_Cost’”. This points users to the exact location of the error, and after reviewing found the SQL code should be “Total_Trip_Cost” / “Duration (days)”, thus missing the inverted commas before the Duration (days). 

This clarity allows users to quickly fix the issue and revalidate the pipeline.

Once all components pass validation, users can confidently execute the pipeline. A green tick next to the “Run” task confirms a successful run.

Further details of the job can be found when users double click into the task row. This is particularly useful when there is an error or failure in the pipeline run, and details of which component and the error message can be viewed.

Key Benefits

• Proactive Error Detection: Errors are identified and addressed during pipeline construction, reducing the risk of downstream failures.
• Time Savings: The UI eliminates the need to sift through lengthy SQL code to locate issues, streamlining debugging efforts.
• User-Friendly Feedback: Errors are clearly marked with visual indicators and accompanied by actionable guidance, making debugging accessible to users with varying levels of technical expertise.

Importance for Teams and Organisations 

Matillion’s Data Productivity Cloud validation features are a game-changer for teams and organisations working with complex data pipelines. These tools enhance collaboration by ensuring pipelines are error-free before being shared among team members. For new starters, the intuitive design simplifies the onboarding process, enabling them to quickly identify and resolve issues without requiring deep technical expertise, thereby shortening the learning curve. Additionally, early error detection prevents wasted computational resources, particularly in large-scale pipelines, making it a cost-effective solution for organisations striving for efficiency.

Stay tuned for the final part of this blog series, where we delve into the visual, drag and drop interface of Matillion’s Data Productivity Cloud to build data pipelines, making it accessible for users without extensive coding experience.

Catch up on:

• Part One: Copilot - AI Assistance for Coding and Pipeline Configuration
• Part Two: Matillion's Auto-Documentation and Collaborative Features