Drag-and-Drop Intuitive User Interface for Pipeline Design

Matillion’s Data Productivity Cloud uses a visual drag-and-drop interface, making data pipeline design accessible to users without coding expertise. Its intuitive layout visually represents components, helping users understand data flows and bridging the gap between concept and execution.

Personal Experience without the Data Productivity Cloud 

As a data engineer apprentice transitioning from a non-technical background, the idea of coding entire data pipelines from scratch felt overwhelming. At the time, I was learning programming languages like SQL and Python, but I lacked the confidence to build a complete, production-ready data pipeline. I understood the outcomes I wanted to achieve – like removing duplicates, grouping data by date, and applying transformations – but figuring out the correct syntax and debugging code was both time-consuming and intimidating.

Manually editing code as a beginner was a taxing process, and mistakes often compounded the complexity of the task. Tools like Matillion’s Data Productivity Cloud changed the game for me, turning a daunting task into an achievable goal. Instead of focusing on coding intricacies, I could focus on understanding and improving data processes.

The Data Productivity Cloud is purpose-built to assist low-code/no-code users in building robust and efficient data pipelines. Below are some of the features I found particularly helpful on my journey to becoming a data engineer:

Components Instead of Code

Matillion uses pre-configured components that represent common data pipeline tasks (e.g., filtering, aggregating, and joining data). This eliminates the need to write complex code for every transformation.

For example, in the pipeline below, the ‘Calculator’ component has been configured to calculate the total trip cost by combining the ‘Accommodation cost’ and ‘Transportation cost.’ This intuitive design makes it simple to set up the component. The column names are readily available in the bottom-left corner of the window for selection, alongside fields to label your expression and a free-form text box to input your SQL code. 

In contrast, the screenshot below demonstrates the extensive SQL code that would be required if the pipeline were manually created.

Drag-and-Drop Interface

Users can easily drag and drop components onto the canvas to build pipelines. This intuitive design eliminates the need for manual coding and simplifies pipeline creation.

For example, in the pipeline below, the list of components is located in the side panel. You can search for a specific component, drag and drop it onto the canvas, and connect it to other components to build the pipeline.

Easier Component Configuration

Configuring components is straightforward, especially for tasks like selecting or reordering columns. Instead of manually writing SQL commands, users can reorder columns visually, reducing errors and saving time.

In the screenshot below, the ‘Traveler Gender’ column is positioned as the last column in the table but should appear as the third column, next to ‘Traveler Name.’ To reorder it, the user can click and hold the dotted square on the left side of the row and drag it up or down to the desired position.

Before:

After: 

Alternatively, the user can select ‘Text mode’ and manually reorder the columns in bulk by copy and pasting.

Visual Pipeline Representation

Pipelines are broken down into digestible, visual components, making it easier to understand the overall flow and pinpoint areas for improvement.

In the pipeline below, components segment the workflow, and their purpose can be identified by their descriptive names. As users become more familiar with the tool, they can easily recognise the function of each component - such as joins, aggregates, or calculators - based on their design.

SQL Code Preview

While using components, users can preview the underlying SQL code generated by the platform. This feature not only builds trust in the tool but also helps users learn SQL organically as they work.

For example, by selecting a component, the user can click on 'Preview SQL' in the component configuration side panel.

Efficient Pipeline Export and Import

Users can export pipelines for reuse or import them as templates for similar tasks. For example, if a pipeline requires the same transformations for different tables, the pipeline can be reused and customized, saving significant time.

For example, the user can click on the pipeline they want to export in the left-side panel, and then right-click on the same panel to import it. Once a pipeline is imported, a new copy of the pipeline will appear, and you are free to make any iterations.

Key Benefits

Matillion’s drag-and-drop features and intuitive UI offer a range of benefits that streamline data pipeline design:

1.Accessibility for Non-Technical Users:

By removing the need for extensive coding, the platform makes pipeline design achievable for users with little to no programming background.

2. Enhanced Learning Experience:

The ability to preview SQL code while using visual components helps users learn SQL in context, bridging the gap between low-code and technical expertise.

3. Reduced Errors and Debugging Time:

Visual representations and pre-configured components minimize the chances of syntax errors and provide a clear overview of the pipeline, making troubleshooting easier.

4. Improved Efficiency:

Features like reusable pipelines and drag-and-drop configuration save time and effort compared to manually coding pipelines.

Importance for Teams and Organizations 

Matillion’s intuitive design and drag-and-drop interface provide immense value for teams and organizations, enabling a broader range of users to build and manage data pipelines effectively. By empowering non-technical team members to contribute, the platform reduces reliance on highly skilled developers and fosters a more collaborative environment. The visual representation of pipelines enhances communication between team members, making workflows easier to understand and review.

In addition, Matillion accelerates efficiency through features like reusable pipelines and quick configurations, eliminating the need for repetitive manual tasks. This streamlined approach frees up time for teams to focus on strategic initiatives rather than troubleshooting or coding from scratch. These benefits translate into faster onboarding for new team members, reduced technical barriers, and a more inclusive approach to data engineering, ensuring that data pipelines are accessible and manageable across all levels of technical expertise.

Conclusion of This 4-Part Blog Series 

In this blog series, we’ve explored how Matillion’s Data Productivity Cloud transforms the landscape for data professionals, especially those from non-technical backgrounds. From Copilot’s AI assistance to auto-documentation, debugging tools, and an intuitive drag-and-drop interface, Matillion Data Productivity Cloud simplifies complex data engineering tasks, making them accessible and efficient.

As a data engineer apprentice, I’ve personally experienced how these tools reduce the steep learning curve typically associated with data engineering. Matillion Data Productivity Cloud empowers individuals and enhances team productivity by reducing reliance on highly specialized skills and fosters a collaborative environment. Its user-friendly design ensures that data workflows are efficient, reliable, and accessible to all team members, driving actionable insights and accelerating business outcomes.

Catch up:

• Part One: Copilot - AI Assistance for Coding and Pipeline Configuration
• Part Two: Matillion's Auto-Documentation and Collaborative Features
• Part Three: Intuitive UI for Debugging and Validation