AI Judges: Finding the Best Model for Evaluating Gen AI Outputs

Exploring how dedicated small language models can be deployed to evaluate and score the quality large language model comprehension.

Generative AI models can produce impressive outputs, but evaluating the accuracy or quality of these is hard and frequently requires manual analysis. That often becomes the rate limiting step of any effort to optimise a Gen AI enabled process or pipeline. That is especially challenging given the range of options available to explore – variations in temperature, RAG strategies, and of course new models coming available every month.

One option available is to use one Gen AI model to rate or score the output of another. This could be a general purpose large language model or a specialised small language model that has been specifically trained to rate the output of other models. This approach is referred to as using an AI Judge. 

In either case we are asking the AI Judge: “given this <contextual input> and this <question> how likely is it that the <other model’s answer> is an hallucination?”. In practice this translates to asking: “is that answer a logical response to infer, given the context?”

The Use Case

In previous Matillion blogs we have explored the use of Gen AI models within a data engineering context to create structured, categorical data based on asking questions about unstructured, free text inputs. Specifically, in that case, to classify free text reviews of consumer electronics and identify which ones highlighted actionable defects to fix. 

Example inputs:

This is a subset of the 50 reviews we used to test models in this exercise. 

With those shown here, ideally we would want our LLM to highlight the 1st, 4th, 5th and 6th lines as having actionable feedback – the Bluetooth speaker’s tinny sound and Bluetooth connection dropping and the reviews complaining about poor tasting coffee and messy pouring from the coffee maker. For those rows we want the model to answer ‘yes’ when asked ‘is there a call to action?’ and ‘no’ for the other rows.

Evaluation Methodology

In order to test the accuracy of possible AI Judges, we collected these reviews, and the previous model answers, and asked ‘given this <review>, is the other model’s answer (yes or no) likely to be an hallucination?’. As with the previous blog we also have a human analyst’s answer to if the review highlights a defect.

That allows us to compare:

• The LLM’s answer
• The AI Judge’s response
• The human’s answer

all of which are yes/no answers.

Gpt-4o-mini results

The classification is based on if the AI Judge correctly detected an hallucination, which can be seen as a mismatch between the human and LLM answers. 

This gives us a confusion matrix on how well gpt-4o-mini is able to do as an AI Judge:

The confusion matrix can be used to generate quality scores: 

In this case the precision is good, meaning if gpt-4o-mini suggests an hallucination it is likely to be one, but the recall is poor – it’s missing many of the errors. To combine the two we will look at the overall F1 score as a composite measure. For those unfamiliar with precision, recall and F1, an overview can be found here. 

Having set up the pipelines in the Data Productivity Cloud to run evaluation using our prompt components for AWS Bedrock and OpenAI, and with this structure for scoring, we were able to evaluate how several common large language models do in this role. 

We also wanted to evaluate how a specialised small language model performs at the same task. For this we used Collinear, for whom this is a core offering. To connect to their Veritas model, we configured a custom connector from the Data Productivity Cloud to talk to their API. For more detail about how custom connectors work see here.

Matillion’s Findings

The results were interesting:

Sonnet and Haiku performed poorly in the role of an AI Judge in this scenario, erring heavily on the side of declaring a result to not be an hallucination or error. Gpt-4o was the strongest performer overall at an 83% F1 score, but Veritas delivered a very respectable 73%.

That was for a zero shot scenario, where no examples were given as part of the prompt. As a next iteration, prompts were updated to include a few examples of what we considered an accurate response vs an hallucination. 

That produced highly varied effects on the results – with the few shot examples:

• gpt-4o-mini became a little less cautious, trading some precision for a huge improvement in recall and hence overall F1 score
• gpt-4o goes now errs too much on crying ‘hallucination’, generating many false positives dropping its overall F1 score – becoming less useful than its mini version!
• Claude Sonnet improves significantly – with examples it becomes less cautious, reducing false negatives and raising the F1 score a lot
• Claude Haiku gives up and says nothing is an hallucination/error
• Veritas improves to put itself at the top of the heap, improving both precision and recall (as opposed to a trade-off), resulting in a 93% F1! 

So with a few examples (few shot) the specialised model really shines for this use case. 

The Collinear team also fine-tuned a Veritas model off some of this very small data set, which produced an improvement on its raw performance but interestingly did not do as well as the simpler few shot approach.   

I’ll hand off to Collinear’s Nazneen Rajani to talk more about it:  

Collinear’s Findings

Collinear’s Veritas is a reliability judge designed to ensure factual accuracy in AI-generated content. Unlike most existing models, Veritas unifies hallucination detection across multiple formats, such as Question Answering (QA), Natural Language Inference (NLI), summarization, and dialogue. Having support for multiple formats is essential in business environments, where AI shows up in many different ways from customer service teams using chatbots, to knowledge workers querying documents, to compliance teams verifying policy alignment. Traditional hallucination detectors, optimized for single formats, force organizations to juggle multiple specialized models, increasing complexity and costs. And while some might consider using large language models like GPT-4 or Claude for fact-checking, Veritas offers a more performant, faster, and cost-effective solution.

Veritas models, available in sizes ranging from approx. 400M to 7.5B, can be customized either through few-shot examples or domain-specific fine-tuning. Our work with Matillion explored both paths to find the best approach. While Veritas can be fine-tuned with just 25-30 examples, our base models proved remarkably robust even without customization. Veritas achieved a 96% F1 score using few-shot examples, significantly outperforming GPT-4o's 0.83. Even our Veritas Nano model reached 93.8% F1, offering enterprise-grade reliability detection at a fraction of the cost.

A fine-tuned Veritas starting with just 25 real-world examples from Matillion with the input format of <query, binary answer, is factual label>, reached a F1 of 84% with just 1 epoch and a small LoRA rank of 64. We note that, counterintuitively, the fine-tuned Veritas was slightly worse than the few-shot Veritas due the format and number of data samples. The Matillion test data was checking for binary indicators of a defect in the product (as opposed to fact checking for grounded data).

These results speak for themselves – Veritas delivers superior hallucination detection through an easy-to-use SDK, whether you need quick deployment with few-shot examples or a custom-tuned model for your domain.

Costing 

From the Matillion side, we also looked at comparative costing, based on the tokens consumed for the test data set and token costs as published at the time of the study (Nov 2024). 

So significant differences in cost appear based on the model chosen and the smaller and/or more specialised models are an order of magnitude or more cheaper than the general purpose large language models. 

Conclusions

As with many things, Gen AI is a field where picking the right tool for the job brings significant rewards in terms of quality, cost and often speed. 

Being able to quickly evaluate model output will be key to rapid progress in projects in this field and believe the combination of two technologies – small, efficient and specialised evaluation models along with the ability to build large and small language model enabled pipelines will enable customers to quickly select the best model and parameters for every use case. 

Furthermore this combination allows our users to monitor that their current production model  remains the best in class option in a rapidly advancing field without the costs and delays of extensive manual reviews. 

If you would like to try Collinear's suite of AI judges, including reliability and safety models, please visit www.collinear.ai or reach out to nazneen@collinear.ai.

Authors

Julian Wiffen

Chief of AI and Data Science at Matillion

Julian Wiffen leads a dynamic team focused on leveraging machine learning to improve products and workflows. Collaborating with the CTO's office, they explore the potential of the latest breakthroughs in generative AI to transform data engineering and how Matillion can play a pivotal role in supporting the needs of those working with these cutting-edge AI models.

Meghana Arakkal Rajeev

Founding Member of Technical Staff at Collinear.AI 

Meghana Arakkal Rajeev is a Founding Member of Technical Staff at Collinear.AI, where she works within the core R&D team to design and deploy enterprise-grade LLM judges – specialized AI systems focused on safety, reliability and values alignment.