Leveling Up: Mastering Matillion Data Productivity Cloud Designer

If you've breezed through the Getting Started with Matillion’s Data Productivity Cloud blog, you're in for a treat! Get ready to amplify your skills as we delve deeper into creating projects, setting up environments, connecting with Git repositories, and building data pipelines. Ready to level up your data integration game? Here are 5 easy steps to becoming a Master of the Matillion Data Productivity Cloud!

Step 1: Creating Your First Project

A project is the backbone of your orchestration and transformation pipelines. Navigate to "Your projects" and select or add a new project. Specify project details, including project name, description, and the cloud data platform you're connecting to (Snowflake, Databricks, or Amazon Redshift). For a “Full SaaS” experience, choose Matillion managed when presented with the option. Here, Matillion will maintain your agents and secrets, allowing you the freedom to focus on designing your data pipelines. For a bit more control over your project, choose Advanced Settings. This allows you to set up a Hybrid SaaS solution leveraging Matillion agents within your own cloud infrastructure. For simplicity, choose Matillion Managed and let Matillion worry about the details! 

Step 2: Setting Up an Environment

Environments define the connection between your project and the chosen cloud data platform. When creating a new project, the initial environment for that project is defined through the project setup wizard. Provide a meaningful Environment name, then specify your cloud data platform credentials and schema specifics to confirm connectivity. Later, you will have the ability to add additional environments from the "Environments" tab in the project explorer.

Step 3: Connect with a Git Repository

Choosing Matillion managed (the Full SaaS approach) eliminates the need for connecting with a Git Repository; one is automatically provided. However, the Advanced settings within Matillion Data Productivity Cloud give you the flexibility to choose the type of code and version control repository you would like to use - Matillion-hosted (powered by GitHub) or Self-hosted (currently GitHub is supported). If choosing a Self-hosted repository, you must authorize Matillion to connect to your desired code repository. These repositories are crucial for version control and collaboration on your projects and are the backbone of the Matillion Data Productivity Cloud Designer for storing, managing, and deploying your data integration pipeline code.

Step 4: Simplify Code Management with Branches

Matillion Designer adheres to the industry's best practices by employing a branching strategy for version control. Every new project in Matillion Designer starts with a primary “main” branch. It is recommended, however, to conduct pipeline development on named development branches rather than the primary branch. Branching helps multiple users work simultaneously and independently within the same project.

You can easily add a new branch from the Branches tab and specify the Branch Name. Choose the base branch to copy from (typically the “main” branch), and then select the cloud data platform environment for the branch. Once added, the new branch will display in the Branches menu, and a simple click redirects you to the Matillion Designer interface for development within the chosen code branch.

Step 5: Begin Building Pipelines

Now comes the exciting part – building pipelines in Matillion's Designer. Click the "add" dropdown in the Pipelines panel and choose between orchestration and transformation pipelines. Enter a name for your pipeline, and start adding components to the canvas. Orchestration pipelines focus on loading data from source to target, while transformation pipelines handle data transformations within the target cloud data platform.

Congratulations, you're now ready to unleash the full potential of Matillion Data Productivity Cloud Designer. Follow these steps diligently, and soon, you'll be orchestrating and transforming data precisely and efficiently.

Take your data capabilities to the next level

As you embark on this journey with Matillion, remember that each step contributes to your mastery of data productivity. The power lies in your hands – become the hero of your data team, and let Matillion amplify your capabilities.

Ready to take your data workflows to the next level? Dive into Matillion Data Productivity Cloud Designer now. Experience the seamless integration, powerful features, and efficiency that redefine data engineering. Begin your journey with a free trial – because every step forward is a step towards excellence in the world of data. Join us and shape the future of data productivity with Matillion.