Standard and Advanced Pipeline Scheduling for Data Engineers

A crucial aspect of data engineering is managing and optimizing data pipelines to ensure smooth and efficient data processing. One of the key challenges they face is scheduling these pipelines effectively to meet business requirements and maintain data integrity. Matillion's Data Productivity Cloud now offers both Standard and Advanced Scheduling capabilities to address these challenges and enhance productivity for data teams.

The scheduling interval challenge

Data engineers often struggle with scheduling data pipelines efficiently due to the complexity of defining precise scheduling intervals. Ensuring that pipelines run at the right time and frequency is essential for timely data processing and decision-making.

Standard and Advanced Pipeline Scheduling

Matillion's Data Productivity Cloud simplifies scheduling tasks by offering both Standard and Advanced Scheduling modes.

In Basic mode, data engineers can define schedules in a simple graphical, low-code interface.

Basic mode pipeline scheduling

In contrast, advanced mode provides the flexibility of allowing engineers to fine-tune schedules using Quartz cron expressions. This enables precise control over scheduling intervals.

Advanced mode pipeline scheduling

💡 Make sure your cron expression evaluates correctly to an expression that defines the schedule. A validation message will appear if you have made a mistake.

Schedules become active after around 10 minutes from creation.

You can both monitor and manage schedules from the Schedules tab in the Designer interface of the Matillion Hub. For convenience, there is also a shortcut on the pipeline editing canvas to create a schedule.

💡 Remember that you must always select the "Publish Pipelines" option during your commit operation to make your new and edited pipelines available for scheduling.

Pushing local changes for scheduling

A normal push operation copies your changes to the remote repository. Selecting "Publish Pipelines" during a push additionally copies all your changed pipelines to your Agent so it can run them.

Conclusion

This range of flexible scheduling options enables data teams to focus on data transformation and analysis rather than getting bogged down in scheduling complexities.

By selectively using Standard and Advanced Scheduling features, data engineers can optimize their pipeline workflows and enhance overall data productivity.

Ready to streamline your data engineering scheduling tasks? Start your journey with Matillion's Data Productivity Cloud today. Sign up for a free trial and experience the power of efficient scheduling in minutes.