Amazon Redshift and Bedrock: Where your data meets AI and why this matters

We have just announced support for Amazon Redshift for the Matillion Data Productivity Cloud. Simultaneously we have also announced support for building AI-powered data pipelines using Amazon Bedrock’s Anthropic models. For anyone who wants to combine the power of AI and the power of Amazon Redshift’s data warehouse engine, this is a game changer.

Everyone is figuring out AI

It’s a year since the world was introduced to large language models (LLMs) that could be used and accessed by everyone. The hype has been enormous, the progress rapid; from the student AI researcher through to the top echelons of government, everyone has been racing to understand how to fit the technology into our lives. Nowhere has this been as critical as in business, where the competitive margin could be the difference between success and failure.

The time for playing is over. The time for material progress is now.

The time for playing is over. It’s time for businesses to wrap up the promising looking PoCs and turn the promise into real-world production value.

The most promising area to make progress is where data meets AI.

Anyone who has worked with AI and Machine Learning Algorithms of all types (not just generative models) will already know that the hard part in the real world is getting the quality of source data to drive and train the model. Our customers have for years been gathering and processing their data using Matillion atop data warehouses like Amazon Redshift. 

In fact, Amazon Redshift was where it all started for us. I have very distinct and fond memories of launching Matillion ETL for Amazon Redshift at re:Invent 2015. You can trace the lineage, experience and depth (as well as a good chunk of the code) directly forward to our launch of Amazon Redshift support in the Data Productivity Cloud.

In hindsight, the success of that product was predicated on the two tectonic trends happening at the time: Growth of the cloud (especially cloud data warehouses) and growth of data. As 2023 adds the power of generative AI to that mix, it’s thrilling to me that we once again find ourselves positioned right at the intersection, this time of data and AI… but that’s enough about my excitement, what does this mean practically?

Practicalities

Amazon Redshift is a tried and tested data platform with a long history powering data warehouses and the transformations that feed them. In conjunction with Matillion’s Data Productivity Cloud, data engineers can be incredibly productive working with structured and semi-structured data.  

Bedrock is the new kid on the block but comes with the excellent Anthropic models that, in our testing, have outperformed OpenAI’s models on the sort of real-world business tasks that customers actually want to take into production. 

 Let’s talk about a few examples of those. Here are my favourite five:

• Categorisation - mapping free text inputs to a label from a defined list, e.g. standardising something like job titles.
• Summarisation - producing a short headline summary of longer text reports, possibly filtering for specific topics of interest.
• Extracting specific calls to action from sources such customer comments, meeting minutes or call transcripts.
• Documenting code and/or translating what it does for a non-technical audience.
• Classification, such as sentiment analysis or asking yes/no natural language questions of unstructured data, e.g. “Did the call centre agent follow the standard script on this call?”

The common thread across all of these is that Amazon Bedrock can unlock the unstructured data that is already part of but probably mostly ignored in existing data pipelines. When used in conjunction with Amazon Redshift, it’s possible to create AI-powered pipelines that immediately improve the performance and quality of the data that will drive your business.

If you’re ready to start taking your AI-powered use cases to production, get started with DPC on Redshift and Anthropic today! Don't miss the chance to supercharge your data team and AI & ML projects. We invite current and prospective Matillion customers to sign up for our AI preview to stay informed about the latest advancements and to get early access to the AI functionality.