How to Evaluate, Compare, and Optimize LLM Systems

Examine evaluation as a concept for LLM-based systems. We'll start with a simple example and build up to evaluating a system here; we QA documents.
We'll also dig into how to leverage hyperparameter optimization (HPO) to find a better-tuned LLM-based system.

Table of Contents

Try out the code here $\rightarrow$

Straightforward LLM Evaluation

We might have to fine-tune the LLM on our "private" dataset for our particular use case. In this case, evaluation is usually straightforward - we have a ground truth dataset against which we evaluate our fine-tuned model, mostly using standard metrics. However, fine-tuning should not be the first thing we should consider, given it's expensive and takes time. To continue our example above, our medical system might be fine-tuned on patient data or medical research not available to out-of-the-box LLM architects.
LLMs are powerful, but with well-thought pre/post-engineering, we can build LLM-based systems that, in many cases, might perform well enough. Building such a system has gotten easier thanks to the tools like Langchain, LlamaIndex, and others. However, it is still tricky to find the right components (more on this later) and to evaluate the system properly.

1. Eyeballing

2. Human Annotation, a.k.a. Supervised Evaluation

The Calculator, aka a simple LLM-based system

Supervised Evaluation

Hyperparameter Optimization

maths_prompt_template_1.txt

The following is the mathematical expression provided by the user.
{question}

Find the answer using the BODMAS rule in the {format_instructions}:

The following is the mathematical expression provided by the user.
{question}

Find the answer in the {format_instructions}:

You are an expert mathematician. You can solve a given mathematical expression using the BODMAS rule.
BODMAS stands for Bracket, Orders of Indices, Division, Multiplication, Addition and Subtraction. The computation should happen in that order.
The dorder is as follows:
B: Solve expressions inside brackets in this order -> small bracket followed by curly bracket and finally square bracket.
O: Solve the indices such as roots, powers, etc.
D: Divide the numbers which are given
M: Multiply the numbers next
A: Sum up the next numbers
S: Subtract the numbers left in the end

The following is the mathematical expression provided by the user.
{question}

Think about it step-by-step. Don't skip steps.

When ready with the answer return in the {format_instructions}:

Observations

There are three accuracy bands - low (10-40%), medium (60-70%), and finally high (88%+). The three bands are positively correlated with the cost. Higher accuracy is achieved by spending more money.
GPT4 rules them all. The GPT4 model gives the highest accuracy.

LLMs Evaluating LLMs

Try out the code here $\rightarrow$

1. Generate Eval Dataset Using An LLM

It's scalable. We can generate a vast number of test cases on demand.
It's flexible. The test cases can be generated for special edge cases and adapted to multiple domains, ensuring relevance and applicability.
It's cheap and fast. LLMs can quickly collate information from multiple documents at a far lower price.

Langchain's QAGenerationChain

You are a smart assistant designed to come up with meaninful question and answer pair. The question should be to the point and the answer should be as detailed as possible.
Given a piece of text, you must come up with a question and answer pair that can be used to evaluate a QA bot. Do not make up stuff. Stick to the text to come up with the question and answer pair.
When coming up with this question/answer pair, you must respond in the following format:
```
{{
    "question": "$YOUR_QUESTION_HERE",
    "answer": "$THE_ANSWER_HERE"
}}
```

Everything between the ``` must be valid json.

Please come up with a question/answer pair, in the specified JSON format, for the following text:
----------------
{text}

2. Metrics

1. LLMs as a Metric?

2. Standard Metrics

Exact Match: For each question-answer pair, if the tokens of the model's prediction exactly match the tokens of the true answer, exact_match is 100; otherwise, exact_match is 0. One can imagine that each token matching is a rare occurrence for a stochastic system. This metric should be taken with a grain of salt for our use case.
F1 Score: This is a well-known metric that cares equally about the precision and recall of the system. Precision is the ratio of shared tokens to the total number of tokens in the prediction. Recall is the ratio of shared tokens to the total number of tokens in the ground truth.

Hyperparameter Optimization

method: random
name: random_qa_full_sweeps
parameters:
  embedding:
    values:
      - SentenceTransformerEmbeddings
      - OpenAIEmbeddings
      - CohereEmbeddings
  llm:
    values:
      - gpt-4
      - gpt-3.5-turbo
      - text-davinci-003
      - command
      - command-light
  prompt_template_file:
    values:
      - data/qa/prompt_template_1.txt
      - data/qa/prompt_template_2.txt
  retriever:
    values:
      - Chroma
      - TFIDFRetriever
      - FAISS
program: qa_full_sweeps.py

Observations

OpenAI models are performing better than Cohere model. The top F1 scores (50+) come from the OpenAI model family.

Click on the check box below to select the experiments for that model. The resulting F1 score is the mean across the experiments.

What can be improved in the evaluation strategy?

Maybe use a better metric than the F1 score. Maybe using some semantic similarity metric like the one proposed in "Semantic Answer Similarity for Evaluating Question Answering Models" (arXiv:2108.06130).
Since gpt-3.5-turbo is performing better on average, it would be good to update the evaluation set to include the following:
More QA pairs are generated using Cohere family of models (command and command-light).
QA pairs generated using all the LLMs that are further scrutinized.

How about evaluating the retrieval system separately?

Conclusion

Related Resources

A Gentle Introduction to LLM APIs

In this article, we dive into how large language models (LLMs) work, starting with tokenization and sampling, before exploring how to use them in your applications.

How to Run LLMs Locally With llama.cpp and GGML

This article explores how to run LLMs locally on your computer using llama.cpp — a repository that enables you to run a model locally in no time with consumer hardware.

COURSE: Building LLM-Powered Apps

A free, interactive course from Weights & Biases

Prompt Engineering LLMs with LangChain and W&B

Join us for tips and tricks to improve your prompt engineering for LLMs. Then, stick around and find out how LangChain and W&B can make your life a whole lot easier.