Testing Large Language Models for Customer Service Copilot: Improving Quality, Reducing Costs, and Transforming Processes

Artificial Intelligence (AI) is redefining how businesses operate. Traditionally, customer service operations rely on manual workflows—analyzing customer issues, researching solutions, and crafting professional responses. While effective, this approach is slow, costly, and resource-intensive.

Imagine resolving 10 times more customer cases at just a fraction of the cost—now made possible with Matillion’s Data Productivity Cloud (DPC).

In this blog, we explore how to integrate Large Language Models (LLMs) into your data pipelines using the Data Productivity Cloud. Through a practical example, I’ll demonstrate how you can automate customer service processes, evaluate AI-generated responses, compare costs, and assess the overall impact. By the end of this post, you’ll see how easily organizations can replicate this workflow to drive efficiency and cost savings.

The Power of Large Language Models in Customer Service

Customer service is often seen as a human-first industry, requiring a nuanced understanding of issues and personalized responses. However, LLMs like ChatGPT and Claude Sonnet have emerged as powerful tools capable of automating large portions of this process. These AI models can:

1. Interpret and summarize complex customer issues.
2. Retrieve relevant documentation or information from knowledge bases.
3. Craft professional, helpful, and context-aware responses.

Despite their potential, not all LLMs are equal. Differences in cost, response quality, and accuracy can significantly impact business outcomes. To explore these variations, I tested four LLMs:

• ChatGPT-4 (ChatGPT-4-1106-preview)
• ChatGPT-4o
• ChatGPT-4o-mini
• Claude Sonnet v3.5

I built a complete ETL pipeline in Matillion DPC to evaluate these models systematically. This pipeline was designed to handle customer case issues from start to finish, transforming raw data into actionable, AI-generated responses.

Version 1 - Using the Amazon Bedrock Prompt Component for Claude Sonnet v3.5

Version 2 - Using the OpenAI Prompt Component for ChatGPT-4, ChatGPT-4o, and ChatGPT-4o-mini

ETL Pipeline Overview: Automating Customer Service with Matillion DPC

The pipeline comprises seven key stages that seamlessly process customer issues, integrate AI-generated insights, and deliver polished responses. Let’s break down each stage:

1. Input Customer Cases

The process begins with a dataset containing unresolved customer support cases. Each case includes:

• A unique Case ID.
• The customer’s query or issue.
• Any available metadata (e.g., timestamp, priority).

This table acts as the foundation for subsequent transformations.

2. Generate Keywords Using LLMs

Using Matillion’s Amazon Bedrock and OpenAI Prompt Components, we extract relevant keywords from each case. These keywords serve as contextual anchors, helping the AI model focus on the core issue.

How It Works:

• Input: The customer query.
• User Context Prompt: A tailored instruction asking the LLM to identify ~5 keywords summarizing the issue.
• Output: A JSON object containing extracted keywords, stored in a new column.

Documentation for Configuring Prompt Components:

• Amazon Bedrock Prompt
• OpenAI Prompt

3. Join Keywords to Customer Cases

In this stage, the generated keywords are joined back to their corresponding case IDs. This ensures a seamless connection between the input data and subsequent transformations.

4. Retrieve Context Using Pinecone Vector Query

Using Matillion’s Pinecone Vector Query Component, we query the company’s knowledge base to retrieve relevant documentation. This stage enriches the customer case with additional information, such as step-by-step instructions, API references, or troubleshooting guides.

Key Steps:

1. Ingest keywords as search strings.
2. Query the Pinecone vector database for matching content.
3. Return text snippets associated with the most relevant vectors.

Documentation for Pinecone Vector Query:

• Pinecone Vector Query Component

5. Join Context to Customer Cases

The extracted context is merged with the original customer cases, creating a comprehensive dataset that includes both the problem statement and supporting information.

6. Generate AI Responses

Using the same Prompt Components as in Stage 2, the LLM generates a detailed, step-by-step email response for each case. The prompt is carefully crafted to ensure:

• Professional tone.
• Step-by-step clarity.
• Accurate and relevant advice.

Prompt Details:

The User Context includes instructions to:

• Reference extracted keywords and contextual data.
• Include step by step instructions to guide the customer.
• Provide helpful links or resources where applicable.

7. Deliver Final Responses

The AI-generated responses are joined back to their corresponding Case IDs, ready for review or direct delivery to customers.

Testing the Pipeline: Results and Insights

To assess the pipeline’s performance, we tested it on a set of 20 sample customer queries. Below is an example of a customer query used: 

“I’m struggling with pagination while using the API Query Profile in Matillion ETL. What type of pagination should I use?”

Responses to the 20 sample customer queries from the four LLMs were evaluated by a Matillion Solution Architect based on usefulness, accuracy, and completeness. Below is an example of the feedback provided for the specific customer query quoted above: 

To better illustrate the differences in responses from the four LLMs to the customer query mentioned earlier, their AI-generated replies are presented below:

Cost Analysis: Which LLM Delivers the Best ROI?

Scalability is crucial for customer service automation. To determine cost efficiency, we calculated the average cost per case for each LLM based on token usage (input + output) for the 20 sample customer queries and vendor pricing.

Switching from ChatGPT-4 to ChatGPT-4o-mini yields a cost reduction of $1,690 per year while receiving the highest rating for response quality out of all of the tested models.

Note: Pricing is subject to change based on vendor updates.

Benefits of Matillion’s Data Productivity Cloud for LLM Integration

Key Advantages:

1. Improved Customer Satisfaction: Faster, more accurate responses enhance the customer experience.
2. Operational Efficiency: Automate repetitive tasks, freeing up team resources for complex issues.
3. Cost Savings: Dramatically reduce operational costs by using efficient LLMs.
4. Scalability: Handle higher volumes of cases without increasing headcount.

Challenges:

• Manual Review: AI-generated responses still require human oversight to ensure accuracy and relevance.
• Continuous Optimization: Regular testing of new models is necessary to keep up with technological advancements.

Conclusion: Transform Your Customer Service Today

This project illustrates how Matillion’s Data Productivity Cloud empowers organizations to revolutionize customer service by integrating LLMs. The result? Reduced costs, improved efficiency, and happier customers.

Ready to unlock the potential of AI-driven customer service? Explore how Matillion can help your organization streamline operations and achieve measurable impact.