This new component is a feature that many users requested and I’m excited to see the potential use cases our customers will come up with. In this blog, let’s have a quick walkthrough of the component and a few examples of what’s possible. For a more in-depth technical overview, check the Webhook Post component documentation.
What is a webhook?
A “webhook” is similar to an API. But instead of having to make a request to get data, a webhook can receive data whenever that new data becomes available. For example, if I want to get a list of my contacts, I can use an API that will send a request and get the list of contacts at that point in time. If I want an updated list of contacts, I call that API again at some point in the future.
A webhook, however, can receive data whenever an event is triggered. For example, if a new contact is added to a table, the webhook can receive the new data immediately.
The Webhook Post component in Matillion ETL allows you to post data to any webhook URL, opening up an almost limitless number of automation possibilities. Like what? Here are a couple of examples to get your imagination going…
1. Posting a message to a Slack channel
Using a Slack webhook, you can post updated information in a Slack channel when it comes in. First, we need to set up your incoming webhook in Slack. Once we have an incoming webhook URL for Slack, we can configure the Webhook Post component in Matillion ETL to send a message to that channel.
One common scenario is to send some information about what has happened during a data load Orchestration Job to a Slack channel. To do this, we’ll first set up a payload template so we can reuse the message format across our projects.
From the Project Menu, we select the new Manage Webhook Payloads item. Then we create a new payload template by giving it a name and entering the payload text into the editor.
Slack has a handy tool for testing your message markup.
Now that we have a payload template saved, we can use this in our Webhook Post component:
Now we just need to hook the Webhook component up to our data loading Orchestration Job, as seen below.
Then we can map the export variables that we use in the payload as shown here:
When the job completes, we then get the following message in our Slack channel:
2. Automating the start and stop of EC2 workflows
You can also automate the start and stop of EC2 using a webhook. For this example, we are going to use the [Custom] Payload option in the Webhook component. This is useful when you want to have a static payload that does not inherit changes from the payload template.
Many of our users take advantage of the elasticity of cloud compute resources and stop their instances when workloads complete. We used Zapier to create a no-code workflow which allows us to turn off the instance at the end of a job.
In Zapier, set up a “Zap” to catch a webhook and then start/stop an EC2 instance.
Then in your Matillion ETL job just set up a Webhooks Post component with a custom payload like the example shown below:
We can turn this into a useful reusable component that we can place into any job by converting that to a Shared Job. (And remember, you can always export your Shared Jobs to Matillion Exchange so other customers and partners can try them.)
For data architects, the ability to post to webhooks eliminates the need for workarounds or custom coding to integrate with an almost unlimited number of downstream applications.
For data and ops teams, the ability to receive notifications via webhooks allows them to respond quickly to errors and other events.
We are very excited that this widely requested component is now available and hope you are, too. Give us your feedback and tell us how you are using the Webhook Post component via the Matillion ETL community.