Matillion's Auto-Documentation and Collaborative Features

Taking advantage of Matillion’s Data Productivity Cloud (DPC) to simplify understanding and collaboration around complex data pipelines, especially for low-code/no-code users joining technical data teams.

Personal Experience without DPC 

As a Data Engineer apprentice transitioning from a non-technical background, one of my initial challenges was understanding existing data pipelines created by other team members. Without prior technical experience, particularly in coding, I worried about my ability to integrate effectively into the team. The lack of clear documentation often made deciphering complex data pipelines a time-consuming and frustrating task.

Matillion’s Auto-Documentation feature transformed this experience. Providing clear, human-readable explanations of pipeline components, allowed me to understand intricate pipelines quickly and contribute meaningfully, even without a technical background. The documentation was straightforward, making it accessible and easy to follow.

Feature Introduction 

In Matillion’s Data Productivity Cloud, the ‘Add Note Using AI’ feature offers a powerful way to automatically generate documentation for data pipelines. This feature leverages AI to annotate each component, contextualizing its purpose and functionality based on its metadata. This significantly enhances collaboration and clarity for both technical and non-technical users.

• How it works: The AI analyses a component’s metadata—such as parameters and configuration—to generate notes for the data pipeline. Importantly, no customer data (such as table contents) is used in this process, ensuring data security and privacy.
• Secure AI implementation: Matillion’s AI functionality is hosted within its secure AWS infrastructure, and reads Matillion's proprietary Data Pipeline Language (DPL). The large language model (LLM) is not fine-tuned with customer data, and all sensitive information remains protected.
• Markdown support: Generated notes can be edited and formatted using Markdown, allowing users to customize and refine the documentation further.

How Auto-Documentation works

Creating AI Notes for a Single Component:

1. Right-click on a component and select “Add Note Using AI.”
2. Review the generated note. Options include:

• Refine: Adjust the language or content of the note.
• Elaborate: Add more detail for deeper context.
• Shorten: Make the note more concise.
• Regenerate: Create a fresh version of the note.
• Click “Add” once satisfied.

Creating AI Notes for Multiple Components:

1. Hold the SHIFT key and draw a selection box around the desired components.
2. Right-click anywhere on the canvas and select “Add Note Using AI.”
3. Follow the same steps as for a single component to refine and finalize the notes.

Editing Notes:

1. Move notes with a drag-and-drop interface.
2. Resize notes to fit the desired amount of text.
3. Click on a note to reveal the raw Markdown, enabling detailed edits.
4. Customise note colors or delete notes entirely using the provided tools.

Real-World Example

Let’s explore how Auto-Documentation improves clarity in the data pipeline below. 

Scenario: A pipeline integrates data from three sources—TravellerDataset, CurrencyUSD, and Currencies—and performs transformations. Without proper documentation, understanding the pipeline’s purpose and functionality relies primarily on component names, which often lack sufficient context. Additionally, interpreting the configuration settings within the components can be challenging, especially for those with limited technical expertise.

Before: A screenshot of the pipeline shows multiple components, but it’s difficult to infer their roles or how data flows through them.

After: By selecting all components and using “Add Note Using AI,” detailed descriptions are generated for each component. The notes explain:

• What each transformation does.
• Key metadata, such as relevant columns and operations.
• The overall purpose of the pipeline.

The final screenshot shows how Auto-documentation adds valuable context, resulting in a well-annotated pipeline that is both easy to understand and straightforward to share with others.

Key Benefits 

1.Improved Context and Understanding: AI-generated notes make complex pipelines accessible, even for users without technical expertise. Each component’s role is clearly documented, enhancing comprehension.

2. Streamlined Collaboration:

• Teams can share pipelines more effectively.
• High-quality documentation reduces reliance on handovers or lengthy explanations, enabling smoother workflows.

3. Time Savings: Automating documentation creation eliminates the need to manually write notes, freeing up time for more critical tasks.

4. Enhanced Onboarding: New team members can quickly get up to speed with existing pipelines, improving productivity and reducing ramp-up time.

Importance for Teams and Organisations 

Auto-documentation is especially beneficial for teams within organizations that frequently onboard new members, as it accelerates ramp-up time, enhances product adoption, and improves overall understanding. It also proves invaluable for data teams working on complex pipelines with multiple contributors, fostering seamless collaboration and shared clarity. Additionally, for teams experiencing high employee turnover, auto-documentation reduces friction and minimizes disruptions during handovers, ensuring continuity and smooth transitions when employees leave or join the company.

Catch up:

• Part One: Copilot - AI Assistance for Coding and Pipeline Configuration
• Part Three: Intuitive UI for Debugging and Validation