Visit Matillion AI Playground at Snowflake Data Cloud Summit 24

Find out more

How to Choose Your GenAI Prompting Strategy: Zero Shot vs. Few Shot Prompts

Generative AI language models have dramatically increased the range and scope of natural language processing tasks that are available to data engineers.

Large language models (LLMs) produce output in response to a natural language input known as a prompt. The prompt contains general instructions alongside some specific context material to be evaluated. For data engineers, the context is likely to be unstructured text sourced from an operational system or process. In practice, it's also very likely that many rows of unstructured text will need to be processed one by one.

The general instructions in a prompt can rely entirely on the innate abilities of the LLM. Alternatively the instructions in the prompt can be written to include some examples of inputs and the desired outputs. This choice defines the difference between zero-shot and few-shot prompting.

What is Zero-Shot Prompting?

The formal definition of zero-shot prompting is to present a task to a generative AI model without giving any specific training on how to perform that task. The model must use its pre-trained knowledge to generate an answer.

A simple example of a zero-shot prompt would be to ask an LLM what the capital of France is. Here's how it looks with Mistral 7B Instruct in the Amazon Bedrock chat playground.

It's a trivial example, but there are two important points to notice nevertheless:

  • The prompt contains no examples
  • The LLM relies exclusively on its pre-training to understand the question, and to answer it

I'll move on to a more sophisticated example now. Imagine I had a free text record of someone's job title – the kind of thing you might capture from a survey or during a recorded meeting. However, business operations require a more formal job title definition so they can branch accordingly, and a zero-shot prompt with an LLM can enable data engineers to bridge that gap.

Here's a zero-shot prompt that you might use to classify an arbitrary job title such that it fits into a predefined list.

Once again no examples are provided in the prompt: the LLM just has to work out the best fit for itself. The actual job title to be analyzed has been taken from one row of input and inserted into the prompt.

For data quality and reliability, note that the prompt allows the model to choose "Other" if a job title does not fit well into the predefined list.

What is Few-Shot Prompting?

Few-shot prompting also involves just sending a single prompt to an LLM. But the crucial difference is that a few examples are added to the prompt to help the LLM understand and complete the task.

Here's an example of a few-shot prompt for log analysis and error detection. Once again the LLM is Mistral 7B Instruct, running in the Amazon Bedrock chat playground.

The above few-shot prompt contains three prior examples that show the LLM how to respond to the task. Important things to notice are:

  • The prompt does include examples
  • To generate the appropriate output, the LLM relies on its understanding of language and on the style and structure of the examples
  • The actual task - captured from a logfile - has been inserted towards the end of the prompt, in an identical way to zero-shot prompting

To continue my earlier example about job titles, it's going to be important operationally to classify job titles as either Individual Contributors or Managers.

Here's a few-shot prompt that turns an LLM into a binary classification engine for job titles:

The prompt contains general instructions and several specific examples, which I have highlighted in yellow. This helps an LLM to produce more accurate classifications.

Zero-Shot vs Few-Shot prompting

The choice between zero-shot and few-shot prompting when dealing with generative AI depends to some extent on the specific requirements of the task you want to perform.

Here's a list of factors you need to consider:

Task Specificity

  • Zero-Shot: best for generalized tasks that do not require domain-specific knowledge relevant to the nuances of the task
  • Few-Shot: preferable for more specialized tasks, especially where additional context and examples can significantly influence the output's accuracy and relevance

Data Requirements

  • Zero-Shot: no need for additional examples; relies solely on the model's pre-training
  • Few-Shot: relies on the prompt providing a small number of example inputs to guide the model in the right direction

Accuracy

  • Zero-Shot: can sometimes provide less accurate - or more generalized - answers due to the lack of specific context
  • Few-Shot: typically provides more accurate answers for tasks with high specificity, thanks to the examples which give context to help refine the model's output

Scalability

  • Zero-Shot: highly scalable, as it does not require specific preparation for tasks
  • Few-Shot: slightly less scalable, as each new task type might require creating new prompt examples. Might also be vulnerable to semantic schema drift, potentially making the specimen responses less accurate or relevant

Conclusion

Zero-shot prompting is ideal for a very large category of general inquiries, where training data is not needed or is unavailable or where creating it would be impractical. Zero-shot is less labor-intensive and great for general queries.

Few-shot prompting is preferable for tasks that are more specific and benefit from rather more nuanced understanding.

To choose the most effective prompting strategy, data engineers considering these methods must evaluate the trade-off between ease of implementation, the availability of example data, and the need for task-specific accuracy.

The Matillion Data Productivity Cloud contains components that mediate between your cloud data platform and your large language model. All the moving parts are handled in a single place, which means you can work most efficiently. Furthermore, mundane tasks like repeatedly injecting the task for row after row of data are handled automatically.

If you would like to try few-shot and zero-shot prompting for yourself on your own data, you can download and use these Matillion Data Productivity Cloud pipelines that run unstructured text classification on job title samples.

Ian Funnell
Ian Funnell

Data Alchemist

Ian Funnell, Data Alchemist at Matillion, curates The Data Geek weekly newsletter and manages the Matillion Exchange.