Conditional execution using the ‘IF’ component in Matillion ETL for Amazon Redshift

Matillion ETL for Amazon Redshift allows you to conduct conditional execution, using the ‘If’ component. This can be used to conditionally run components in your orchestration job, based on a defined condition. In this short tutorial video we show you just how easy it is to do this.

Matillion ETL for Amazon Redshift enables organizations to simplify data loading, transformation and orchestration in Amazon Redshift.

Our powerful, graphical job development environment includes a whole host of features and components, designed to make data orchestration in Redshift a breeze!

One of these components is the ‘If’ component.

The ‘If’ component

The ‘If’ component can evaluate an expression involving variables and direct the flow of an orchestration job, depending on whether the expression evaluates to ‘true’ or ‘false’.

The component has 3 output connectors:

• A Blue connector is followed when the expression evaluates to ‘true’.
• An Orange connector is followed when the expression evaluates to ‘false’.
• The usual Red connector is followed if a run time error occurs during evaluation.

Conditional execution using the ‘If’ component

In the tutorial video below, we show you an example of how you can use the ‘If’ component to conduct conditional execution in an orchestration job.

The starting point for this tutorial is with an orchestration job, similar to the one shown below.

In this example, after the RDS load has completed, if it loaded less than 1000 rows, we want to alert someone and fail the job, as there has likely been an issue.

To do this, we can define a numeric variable called ‘loaded_rows’ to take the row count after the RDS load.

Using the ‘If’ component, we can then use this loaded_rows variable and set ‘true’ and ‘false’ conditions, depending on whether the row count is greater than or equal to 1000.

If the ‘If’ component evaluates this to be ‘true’ (i.e. greater than 1000), then we can use the Blue connector to end the job as a success.

If however, the ‘If’ component evaluates it to be ‘false’ (i.e. less than 1000), then we can use the Orange connector to send an SNS message to alert someone of the failure.

We can then explicitly end this job in a failure, so that no other jobs are run as part of a flow that includes this.

Want to find out more?

Subscribe to the Matillion Youtube channel for more great videos like this one.

Alternatively, download our free guide below, for all the Amazon Redshift advice and best-practices that you need to follow.