Be at the Forefront of AI: Introducing Matillion’s New AI Capabilities for Data Engineering

Welcome to the latest innovation from Matillion – our new AI Prompt and Vector Store components. These components are designed to enhance your data engineering capabilities by leveraging the power of large language models (LLMs). Whether you're a seasoned data engineer or just starting out, AI is here to bring more value to your pipelines and unlock new possibilities in data transformation for AI and analytics projects.

Leveraging the Power of Large Language Models

Our AI Prompt components use LLMs to provide intelligent responses to user-composed prompts. Combining inputs from your source table with user prompts enables you to process data in innovative ways. With retrieval augmented generation (RAG), you can further optimize outputs for accuracy, efficiency, and contextualization in your data workflows.

OpenAI Prompt Component

Our Open AI Prompt component integrates seamlessly with the OpenAI platform, allowing you to tap into the capabilities of GPT-3.5 and GPT-4 for data processing tasks. By leveraging OpenAI's advanced generative AI technology, this component enables you to generate responses to user-composed prompts with unparalleled accuracy and speed.

• Open AI Prompt: Docs, Video

Azure OpenAI Prompt Component

Built on the Azure cloud platform, our Azure OpenAI Prompt component brings the power of LLMs to your Azure-based data pipelines. With seamless integration and robust performance, this component empowers you to harness the full potential of AI-driven data processing in your Azure environment.

• Azure Open AI Prompt - Docs, Video

Amazon Bedrock Prompt Component

For users leveraging the Amazon Web Services (AWS) ecosystem, our Amazon Bedrock Prompt component offers a reliable and efficient solution for incorporating LLMs (i.e., Anthropic Claude, Meta LLama2, Cohere Command, Amazon Titan, etc.) into your data workflows. Built on AWS infrastructure, this component provides seamless access to AI-driven prompt processing capabilities, enabling you to optimize your data pipelines with ease.

• Amazon Bedrock Prompt - Docs, Video

Get Started with Prompt Engineering

Enhance your data engineering efforts with AI Prompt Engineering, transforming data processing and analysis across OpenAI, Azure, and AWS environments. Our components add valuable data context to pipelines, leveraging Language Model (LLM) technology to generate responses to user prompts. Matillion integrates smoothly with leading LLMs such as OpenAI, AWS Bedrock, and Azure Open AI, offering flexible output options in text or JSON formats while ensuring effortless storage in your cloud data platform. Our model-agnostic approach ensures adaptability to emerging LLMs, facilitating swift integration within a week of market release and empowering your organization with unparalleled data-driven insights and innovations. 

Want to learn more about how to get started with prompt engineering - read our ebook Getting Started with Prompt Engineering: Insights and Tips From a Data Scientist

But that’s not all! Let’s Talk about RAG

We've also introduced two new components tailored to enhance your AI capabilities and help you implement RAG in your organization. 

Pinecone Vector Upsert

With the Pinecone Vector Upsert component, you can effortlessly convert data from your cloud data warehouse into embeddings and store them as vectors in your Pinecone vector database. These vectors can then be used in retrieval augmented generation, enabling you to add even more context and accuracy to the insights coming out of LLMs, integrate valuable insights into analytics, and drive innovation in your AI projects.

Watch our video to see this component in action!

Pinecone Vector Query

The Pinecone Vector Query component allows you to ingest text search strings and retrieve text data associated with similar vectors in your Pinecone vector database. By harnessing the power of vector similarity search, you can quickly look up relevant information and make informed decisions based on your data.

Learn more - see in action!

Want to learn more about RAG? Check out our RAG for Dummies blog!

Empowering Data Engineers Everywhere

"Our new AI Prompt components enable data engineers to leverage prompt engineering when designing data pipelines in the Data Productivity Cloud. This allows for the transformation of data using natural language prompts in low-code, graphical components, helping you to speak the language of your business in data pipelines. Anyone can integrate the reasoning capabilities of LLMs to transform and enrich data, taking data processing to the next level. For instance, turning unstructured data into structured content is now more accessible: making data business-ready faster with AI thanks to these new components." – Cyril Sonnefraud, Principal Product Manager at Matillion

Auto Documentation Now Generally Available

In addition to our AI Prompt components, our Auto-Documentation and Matillion’s AI Note feature is now generally available. With AI Note, you can easily annotate your data pipelines with explanations and business documentation, making it easier than ever to share insights around pipeline literacy and strengthen collaboration.

Simply select the component(s) you want to document on the canvas, right-click, and select "Add note using AI." You can then refine the note as needed, adding more detail or reducing it to suit your needs.

Learn more about AI Note in our documentation. 

Getting Started

For more information on each component, check out the documentation and watch our hands-on configuration videos:

• Prompt Components
  • Open AI Prompt: Docs, Video
  • Azure Open AI Prompt - Docs, Video
  • Amazon Bedrock Prompt - Docs, Video
• RAG Components
  • Pinecone Vector Upsert - Docs, Video
  • Pinecone Vector Query - Docs, Video

Unlock the Power of AI in Data Engineering

With Matillion's AI-powered data engineering components, you can boost productivity, level up your analytics projects, and build pipelines for new AI use cases. Discover unprecedented productivity, add data context with prompt engineering, get better responses from LLMs with RAG, and open up new use cases with AI applications.

Learn more about Matillion's approach to AI and join us on our journey to enhance data engineering with the power of AI.

Watch our event “AI. The Future of Data Engineering” to learn more. 

Are you ready to transform your data engineering with AI? Get a demo from one of our AI experts today!