Webhooks and Matillion pipeline monitoring

Webhooks. Now, there are all sorts of definitions you can read, from the detailed technical answer according to Wikipedia:

“A webhook is a method of augmenting or altering the behaviour of a web page or web application with custom callbacks. These callbacks may be maintained, modified, and managed by third-party users who need not be affiliated with the originating website or application.”

To Mailchimp’s high-level explanation:

“Webhooks are a way for one application to provide other applications with real-time information.”

Messaging tools, monitoring tools, and all sorts of applications will happily receive your webhook message. For me, as a Matillion user, it’s a way to push status messages to anywhere that will accept them.  As a pipeline runs, I can send messages at crucial points, be they for errors or just to provide a timestamped status for logging.

The webhook component

There’s not a lot of settings here. Simplicity isn't a bad thing. In fact, Payload Template is here just for future use, so we only have to consider the URL and the Payload.

The URL will be supplied by the destination of the message. For example, Slack would provide a URL like this: 

"https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"

For testing webhooks with minimal effort, I like to use https://webhook.site/. It’s free and saves you from having to configure the destination to see how the webhook component can work for you.

Now that we have our URL, we can enter the payload. I’m just going to use a very simple message for now

If I then run the webhook component, I will see an entry on the destination system. 

Like this:

So, that’s the mechanics of it. But how can we apply this to something useful?

Alert on job failure

Let’s assume we want to run a pipeline and just report whether it failed. You might create something like this:

The pipeline we want to run is a child pipeline, and connected to the “Failure” output of that child pipeline is our webhook message.

You can see here that I ran my pipeline, the child job failed, and the Send Failure Message (webhook component) was run.

There’s only one slight problem. The webhook was sent successfully, so the overall status of the pipeline was successful. That said, we’re going to need to end the failure path with an End on Failure component.

Something like this:

That's better. And I’m also seeing the message I entered into the webhook component displayed in the system receiving the message.

So, what we have is a setup where if the pipeline has some form of failure, we can automatically inform a monitoring system, comment in a Slack channel, or something similar.

Including variables

In the examples so far, I’ve been adding static text to the message. But if there has been a problem, it’d be useful to know details about the processes held in variables.  

I have an iterator in my sample pipeline that passes table names to the sub-pipeline. So, it would be useful to know whether the error is being seen against all table names.

In the iterator variables, I have a variable called tableName.

Let’s put that variable into our message:

If I run my “failing” pipeline, I now get the following message:

Of course, you can load all sorts of information into a message from variables. So, it might be that I wanted to run a Query Result to Scalar, load in a row count or max date/time for the table in use, and present that in the message.

Hopefully, now you can see that webhooks can be a convenient way to send messages relating to pipelines. Want to try? Start a free trial of the Matillion Data Productivity Cloud today.