LLM evaluation: Metrics, frameworks, and best practices



Table of contents

Key metrics for LLM evaluation

Traditional Statistical Metrics

Perplexity: A common metric for language models that measures how well the model predicts a sample. Formally, it is the exponential of the average negative log-likelihood of the test data. Intuitively, a lower perplexity means the model finds the test data less "surprising" (and thus is a better fit). For example, if an LLM has a perplexity of 20 on a corpus, it means it's about as uncertain as if it had to pick among 20 equally likely options for each word on average. Lower is better, indicating the model can predict words more confidently​. Perplexity is often used during training or to compare language models, but by itself doesn't tell if outputs are useful or correct - just that they match typical language patterns.
BLEU (Bilingual Evaluation Understudy): A precision-based metric originally developed for machine translation. BLEU checks the overlap of n-grams (continuous sequences of n words) between the model's output and one or more reference texts​. A higher BLEU score means the generated text shares more common phrases with the reference, implying it's closer to the expected output. For instance, a BLEU-4 score (up to 4-gram overlap) is often reported for translations. BLEU works well for comparing against a specific reference wording, but it can penalize valid rephrasings or creative answers that deviate from reference wording. It's less useful for open-ended generation where there isn't a single correct output.
ROUGE (Recall-Oriented Understudy for Gisting Evaluation): A set of metrics mainly used for summarization tasks. ROUGE-N is similar to BLEU but recall-focused - it measures how many n-grams from the reference are present in the generated output​. There's also ROUGE-L, which measures the length of the Longest Common Subsequence (LCS) between the output and reference, capturing sentence-level structure similarity. In summarization, a high ROUGE means the model output captures a lot of the same information (words/phrases) as the human-written summary. Like BLEU, ROUGE is easy to compute and useful when reference texts are available, but it may not reflect coherence or factual correctness.
METEOR (Metric for Evaluation of Translation with Explicit ORdering): Another metric for translation and summarization that aims to improve on BLEU. METEOR uses a flexible matching strategy: it aligns the generated text and reference, allowing matches not just on exact words but also on stems and synonyms​. It then computes a score based on precision and recall of these alignments, with penalties for things like word order differences. METEOR often correlates better with human judgment than BLEU in language generation tasks because it credits the model for using different words that mean the same thing. It produces scores typically in 0–1 (or 0–100%) where higher is better.
F1 Score: Commonly used for evaluation in tasks where the output can be seen as a set of items (e.g., extracted entities, or QA exact answers). The F1 score is the harmonic mean of precision (what fraction of the model's outputs were correct) and recall (what fraction of the desired outputs did the model produce). It balances the two, and is high only when both precision and recall are reasonably high. In many information-retrieval or extraction-style evaluations (like retrieving relevant documents or predicting a set of keywords), F1 is used. For example, in a QA context, if we consider an answer correct when it contains certain key facts, precision is the fraction of the answer's facts that are correct, and recall is the fraction of the reference facts covered - F1 combines these. A perfect F1 (1.0 or 100%) means the model found all the right information with no extra incorrect info. (Mathematically, F1 = 2 * (precision * recall) / (precision + recall))​

They often require a reference output (ground truth) for comparison. For open-ended tasks like open-domain conversation or creative writing, references are hard to define.
They mainly assess surface similarity. A model output could have different wording but equal meaning as the reference - BLEU/ROUGE might score it low despite it being a fine answer. Conversely, an output could get a high score by copying the reference even if it's irrelevant or incoherent in context.
They don't directly measure attributes like factual accuracy, logical coherence, or stylistic appropriateness. For instance, a grammatically perfect sentence that is factually wrong could still score well on BLEU if it overlaps with reference words.
Optimizing solely for these metrics can lead to overfitting to them (the model learns to game the metric rather than truly improving quality). Despite these issues, BLEU, ROUGE, etc., remain widely reported in papers as a rough proxy for quality on tasks with reference texts (translation, summarization, etc.). They are best used in combination with other evaluation methods.

Model-based evaluation metrics

BERTScore: A metric that uses pre-trained language model embeddings (from BERT) to evaluate text similarity. Instead of exact string matches, BERTScore aligns each word in the candidate output with a word in the reference based on their vector similarity in BERT's semantic space​. It then computes precision, recall, and F1 at the token embedding level. Intuitively, if the model output uses different words that convey the same meaning as the reference, BERT's contextual embeddings for those words will be close, yielding a high BERTScore. This metric has been shown to correlate better with human judgment on tasks like summarization than BLEU or ROUGE, since it captures meaning, not just exact wording. Scores are often reported as precision/recall/F1 (or just F1) in the range 0 to 1 (higher is better).
COMET: A learned evaluation metric (created by Unbabel for machine translation) that goes even further by training a neural network specifically to score MT outputs. COMET (Crosslingual Optimized Metric) takes in the source input, the model's translation, and a reference translation (or in some versions, just source and translation) and outputs a quality score​. It is trained on human-rated translation examples, so it learns to predict scores that mimic human evaluation. Because it's learned, COMET can pick up on errors that simple overlap metrics miss (like mistranslations even when some words overlap). It has become a new standard in academic MT evaluations, often reporting higher correlation with human judgments than BLEU. There are also variants of COMET for different settings (reference-free evaluation, etc.). While COMET is mostly used for translation tasks, the approach of training a model to evaluate text can extend to other tasks (and some have done similar for summarization, etc.).
BLEURT, PRISM, BARTScore, and others: These are similar learned metrics. BLEURT fine-tunes BERT on a collection of synthetic and human-rated data to predict a score for a candidate vs reference. PRISM uses a multilingual model to measure probability of the candidate given the reference and vice versa. BARTScore uses a pre-trained seq2seq model (BART) to score how likely the candidate is a paraphrase of the reference by reconstruction likelihood. All these aim to leverage the power of large models to judge text quality beyond exact matches.
GPT Score / LLM-based Scoring: With the advent of powerful LLMs like GPT-3 and GPT-4, a trend is to use the LLM itself as an evaluator. For example, given a task prompt, one can ask GPT-4 to rate the outputs of two models or to score a single output on a scale (perhaps with some instructions on what constitutes a good answer). An example of this is G-Eval, a framework where GPT-4 is prompted with a detailed rubric and chain-of-thought reasoning to grade an NLG output​. The prompt might say, "Evaluate the following answer for correctness and clarity" and GPT-4 will produce a score along with an explanation. G-Eval has shown higher agreement with humans by using GPT-4's understanding to evaluate content, especially when it uses chain-of-thought prompting (making the LLM explain its reasoning) to increase reliability. Essentially, the LLM is being used as a judge.

They can inherit biases from their underlying models. If the model (like BERT or GPT-4) has certain blind spots, those can reflect in its evaluations.
They require careful calibration. An LLM used for scoring might need prompts that ensure it evaluates on the right criteria. Different prompt phrasing could lead to different scores​.
There's a risk that models could overfit to these learned metrics if used in training (e.g., optimizing a response to please GPT-4 as a judge might lead to overly verbose answers because the judge model likes more explanation).
Still, these metrics have become very popular for research because they often align better with human preferences than BLEU/ROUGE. For example, if a new summarization model has a higher BERTScore and higher GPT-4 judge score than a baseline, it's a strong signal it actually produces more human-like summaries (though one would verify with human eval as well).

Human evaluation methods

Preference Tests (Pairwise Comparison): Evaluators (often crowdworkers or domain experts) are shown outputs from two different models (or a model vs a reference) for the same prompt and asked which they prefer. This yields a preference percentage - e.g., "Model A's answer was preferred 70% of the time over Model B's." Pairwise tests are very useful when deciding between model variants. They are simpler for annotators than giving absolute scores, and they directly tell you which output is better in a relative sense. Many companies (e.g., OpenAI, DeepMind) use pairwise comparisons extensively - for instance, to fine-tune models with Reinforcement Learning from Human Feedback (RLHF), humans compare outputs to train a preference model. Even for evaluation, a win rate of one model over another is a strong metric (OpenAI's GPT-4 paper uses this to show GPT-4 is preferred over GPT-3.5, for example).
Likert Scale Ratings: Annotators rate an output on a fixed scale for certain criteria. For example, rate from 1 to 5 on quality, or 1 to 7 on how informative or harmful the response is. This allows more fine-grained evaluation on multiple axes. In a chatbot evaluation, you might have people rate overall satisfaction, accuracy of the answer, politeness, etc. The average ratings (and distribution) indicate where the model stands. Likert scaling provides more detail than a binary win/lose comparison, but it can be harder to get consistency - one person's "4" might be another's "3". Clear guidelines and training examples for raters help mitigate this.
Scalar or Ranked Outputs: Sometimes outputs are ranked from best to worst if comparing many models at once, or given a numeric score like 0–10. This is similar to Likert but not bounded to labeled categories like "strongly agree" etc. For instance, an evaluator might score a summary on a 0–100 scale for readability. Typically, though, bounded Likert scales are used to maintain consistency.
A/B Testing with End Users: In live systems, an ultimate test is deploying model A to a fraction of users and model B to another, then using implicit or explicit feedback to judge which performs better. For example, you might deploy two versions of a search query answering model - one uses a new LLM, one uses the old - and see which version yields higher user engagement or satisfaction ratings in a real setting. This is a form of human evaluation at scale (the "users as judges" approach). It's very powerful because it measures actual real-world success criteria (like user clicks or task success). The downside is that it's post-deployment - you only do this with models you believe are safe enough to test with real users.

Accuracy/Correctness: Does the output contain factual errors or mistakes relative to a gold answer or known truth?
Relevance: Is the output on-topic and addressing the user's query or the prompt?
Fluency/Naturalness: Is the text well-formed, grammatically correct, and natural-sounding?
Coherence: Does the output make sense as a whole? (This is especially for long outputs - e.g., is a story's plot coherent, does an essay's argument follow logically?)
Usefulness: For applications like assistants, was the answer helpful?
Harmfulness/Toxicity: Does the output contain offensive or unsafe content?
Fairness: If evaluating for bias, have humans check outputs for biased assumptions or stereotypes.

Custom task-specific metrics

Factual Accuracy Metrics: For tasks where factual correctness is crucial (e.g., Q&A, knowledge base retrieval, summarization of factual documents), teams devise ways to score how many facts are correct. This could be as simple as exact match accuracy for closed questions (did the model output exactly the expected answer?) or more complex like Precision/Recall on Knowledge Graph facts. For summarization, there are metrics like FactCC and Q² which involve automatically checking the summary against the source text for consistency. Another approach is to use an information retrieval system: take each claim the LLM makes in its output and see if you can find support for it in a trusted database or Wikipedia. If not, that claim might be inaccurate. These metrics try to catch hallucinations. For instance, TruthfulQA is a benchmark that asks a set of questions designed to see if the model will produce truthful answers or fall into common misconceptions - the score is the percentage of questions answered truthfully.
Coherence and Consistency Metrics: To measure if a long passage is coherent, one might use a discourse coherence score. Academic work has proposed metrics using entity grids or transition probabilities of topics. There's also Coh-Metrix in linguistics which computes various coherence indicators. For consistency (especially in dialogue or storytelling), one might check if the model contradicts itself. For example, if in a story a character's hair was described as blonde in one paragraph and black in another, that's an inconsistency - a custom script could flag that. In dialogues, checking that the model's answers don't conflict with things it said earlier is another consistency check. These are often rule-based or require semantic parsing of content. While not trivial to automate, even simple checks (like looking for negations or contradiction words) can help quantify logical consistency.
Specificity / Relevance: In dialogue systems, a metric called USR (Unsupervised and Reference-free evaluation) includes components for checking if the response is specific to the prompt (not a generic "I don't know" or off-topic). Evaluating relevance might involve comparing the embedding of the response to the embedding of the query - if they're very unrelated, the answer might be off-topic. Such a metric could be used to penalize meandering or evasive answers.
Length-based Metrics: Some tasks require concise answers, others require elaboration. For example, a summarization might be evaluated on compression ratio (how much shorter than the original while retaining info). Or a code generation task might measure lines of correct code produced. If brevity or verbosity is a quality concern, one could include a simple metric like output length or compression rate.
User Engagement Metrics: In interactive systems, one indirect metric is how users behave. E.g., for a help chatbot, one metric might be containment rate: the percentage of conversations that the bot handled without needing a human handoff. That's not a traditional "NLP" metric, but it's a real success metric for that system. Or time to resolution, user satisfaction ratings at end of chat, etc. These require a deployed system to measure, but they are key for practical evaluation.
Safety Metrics: If evaluating for safe behavior, one might count the number of unsafe outputs in a test set (perhaps using a classifier to detect them). For instance, run the model on a battery of provocative or sensitive prompts and use a toxicity detector to see how many toxic responses occur. The "metric" could be something like "% of outputs flagged as toxic" - lower is better. Similarly, bias can be evaluated by specific tests: e.g., pose the same question about different demographic groups and use a sentiment analyzer on the responses to see if there's a disparity. The metric could be the difference in sentiment between groups - ideally zero for a fair model.
Code-specific Metrics: For LLMs that generate code, standard software metrics are used. E.g., pass rate on unit tests (provide some test cases and see if the generated code passes them), or functional correctness on known programming challenges (like the percentage of problems solved on Codeforces or LeetCode). In these cases, an output is "correct" if it produces the expected result when executed - a very clear-cut metric.

Evaluation methodologies

Automated benchmarks and datasets

GLUE and SuperGLUE: General Language Understanding Evaluation (GLUE) is a collection of multiple English tasks (such as sentiment analysis, paraphrase detection, question answering, textual entailment, etc.). A model's performance is measured by its accuracy or F1 on each task, and often an aggregate score is reported. SuperGLUE is a harder version with more challenging tasks and an unbalanced class distribution to discourage trivial solutions. These benchmarks were widely used pre-LLM era (for models like BERT, RoBERTa, T5) and remain useful to check if LLMs can handle basic language understanding​. Researchers can use benchmarks like GLUE/SuperGLUE to compare how well different LLMs handle a variety of language tasks​. Modern LLMs (like GPT-3+) often perform extremely well on these, sometimes near human-level, but they are still good sanity checks for foundational capabilities.
MMLU (Massive Multitask Language Understanding): This is a benchmark introduced to test knowledge and reasoning across 57 diverse subjects, from history and literature to mathematics and biology. It is structured as multiple-choice questions for each subject. The goal is to see what the model knows and can reason about, likely from its training data. MMLU is quite challenging; models take this like a big trivia/knowledge exam. It has become a standard to evaluate knowledge depth - for example, GPT-4's high MMLU score was cited as evidence of its broad knowledge. It essentially measures how well an LLM can solve problems or answer questions in domains it supposedly learned during training.
HellaSwag, PIQA, WinoGrande, etc.: These are targeted benchmarks for common-sense reasoning and logic. HellaSwag is about picking the best continuation of a story snippet (testing common-sense grounding). PIQA (Physical IQA) asks common-sense physical reasoning questions ("how would you use X to do Y"). WinoGrande is a large Winograd Schema dataset testing pronoun resolution with common sense (to evaluate understanding of situations to resolve ambiguities). LLMs are evaluated on accuracy on these. They probe whether models have the kind of everyday reasoning humans do. Initially, these were challenging, but larger models have gotten much better at them.
BIG-bench (Beyond the Imitation Game): A collection of very diverse and creative tasks (around 200 tasks) contributed by the research community, specifically designed to be difficult for current models. Tasks range from elementary math to logical deduction puzzles to translating ancient languages. BIG-bench is meant to probe capabilities that we don't typically test. Models are evaluated against these tasks (often with custom metrics per task). A model's overall BIG-bench performance gives an idea of its emergent abilities or lack thereof on unusual challenges. It's useful to identify blind spots; for example, a model might do well on standard benchmarks but fail at a BIG-bench task that requires complex planning or understanding jokes.
Holistic Evaluation of Language Models (HELM): This is a newer initiative (from Stanford) not just a dataset but a framework. HELM defines a broad set of "scenarios" (tasks and domains) and metrics across categories like accuracy, robustness, calibration, fairness, bias, toxicity, and efficiency​. The idea is to provide a comprehensive leaderboard where models are compared on many axes simultaneously. For example, HELM reports not just how good a model is at a task, but also how calibrated its probabilities are, how it performs when input is noisy, how fast it is, how often it produces offensive outputs, etc. This addresses the need for multidimensional evaluation beyond a single score. HELM is more of an evaluation platform than a static benchmark; it's updated as new models and metrics come in.
OpenAI Evals / Custom Benchmarks: Many organizations build their own internal benchmarks relevant to their product. OpenAI, for instance, has an openai-evals framework where you can specify custom evaluation logic (like a Python script that checks if ChatGPT's answer contains some key info). These are often tailored to specific use cases (like coding, math word problems, etc.). Over time, some of these become public or de-facto benchmarks. For coding, there is HumanEval (writing correct programs for given spec), MBPP (Massive Multitask Programming Problems), etc., which measure functional correctness of code outputs.

Human evaluation techniques and challenges

Use a held-out dataset: If you have a set of questions or tasks that are especially important (or were used in development), take a fresh set that the model hasn't been adjusted on, and have humans evaluate outputs on those. This ensures an unbiased measure on novel inputs.
Crowdsourcing: Platforms like Amazon Mechanical Turk, Appen, or Scale AI are often used to get human judgments. You prepare a guideline (what to look for in the response) and examples of good and bad outputs. You then have multiple annotators rate each output. It's important to have multiple annotators per item to detect consensus or flag disagreement. One common approach is to have 3 to 5 people rate each output and then either take the majority vote (for preference tests) or the average rating.
Expert evaluation: For specialized domains (medical, legal), you may need people with expertise to judge correctness. E.g., a medical answer needs a doctor to verify if it's correct. This is slower and expensive, but necessary for high-stakes domains. Often a smaller sample is evaluated by experts due to cost.
Red teaming: This is a type of human eval where experts try to break the model - i.e., find inputs that cause harmful or nonsensical outputs. Red teamers deliberately probe the model's weaknesses (safety loopholes, logical failures). While not a "metric" per se, the findings from red teaming (like "model gave dangerous advice when asked X") are used to qualitatively judge if the model is safe enough.

Preference win rates (e.g., Model A was preferred in 60% of comparisons against Model B).
Mean ratings and standard deviation on each criterion.
Inter-annotator agreement, to see if the evaluation was consistent. For classification-like evaluations, metrics like Cohen's kappa or Krippendorff's alpha might be calculated. High agreement gives confidence in the results; low agreement suggests the task was ambiguous or guidelines not clear.

Cost and Scalability: As noted, human eval doesn't scale well. Doing a thorough human eval for every model tweak is impractical​. This is why teams often reserve human eval for final comparisons or periodic checks, and rely on automated metrics in between (we'll discuss hybrid approaches shortly).
Time: Getting human annotations can take days or weeks (from setting up the task, running it, to cleaning the results). Models might have changed in the meantime. So there's a latency in the feedback loop.
Quality control: Crowdsourced annotators might misunderstand instructions or not pay full attention. It's important to include test questions (with known correct answers or obvious expected judgments) to filter out low-quality work. Many eval setups include some "sentinel" items - if a worker gets those wrong, their other answers might be thrown out.
Subjectivity: For some tasks, even humans don't agree on what's best. E.g., evaluating a story's "creativity" is subjective. One way to handle this is to pick more objective questions for humans ("Did the story have a clear ending? Yes/No") rather than a vague "How creative?". Alternatively, accept the subjectivity but ensure you have enough samples that an average emerges.
Annotator biases: As mentioned, people have biases. Some might be more lenient, some more critical. If your annotator pool isn't diverse, you might get skewed results. For example, if all annotators are fluent English speakers from one country, their judgment of what's "polite" might differ from another culture's. It's good practice to have a diverse annotator set if evaluating something like politeness or general helpfulness (assuming a global user base for the model).
Task framing: How you pose the task to annotators can influence results. If you say "Model A was tuned to be more concise. Which is better?" - you've primed them to favor conciseness. So you must frame neutrally, like "Here are two answers. Please indicate which you prefer and briefly why." Also, instruct them to consider multiple aspects (or a primary aspect) depending on what you want. Sometimes separate groups of annotators are used for different aspects to avoid conflating criteria in one person's mind.

LLM-as-a-judge approaches and their limitations

LLM judges can approximate human preferences reasonably well in certain domains, especially if the criteria are well-defined. They are extremely consistent in how they apply the given rubric (no annotator mood swings or fatigue, unless the prompt itself has randomness).
They are very fast and cheap once you have the model (especially if using an open-source LLM as the judge, or an API like GPT-4 which, while not free, can evaluate thousands of pairs much faster than humans could).

Inconsistency and prompt sensitivity: LLM evaluators can be inconsistent. For instance, phrasing the evaluation prompt slightly differently might change the outcome, indicating they are not entirely robust evaluators​. Also, if asked to score outputs independently, they might not maintain a stable scale (one day a response is "8/10", another day the same response might get "7/10" because these models don't have a persistent rubric memory unless enforced via prompt). Some work suggests that having them do pairwise comparisons (A vs B) is more stable than absolute scoring.
Biases and alignment: LLMs carry biases from their training. If asked to judge outputs, they might have a bias towards more verbose or formal answers, or they might reflect biases in content (e.g., prefer answers that align with majority opinions in training data). One study noted that LLM-based evaluations are prone to similar biases as human ones, since the LLM was trained on human-written content and human preferences. Moreover, if the model being evaluated and the model judging have similar weaknesses, the judge might not catch certain errors. For example, if both have a propensity to think a subtly false statement is true, the LLM judge won't flag the other model's false statement as wrong.
Honesty of explanation: If an LLM judge provides a reasoning, it might produce a plausible-sounding explanation that isn't actually the true reason it preferred one output. It could potentially rationalize a choice that was made due to some spurious pattern. So reading LLM judges' explanations needs caution - they are not guaranteed to reflect the real "thought process" (since they don't have one in the human sense, they just generate likely text). However, these explanations can still be informative and even out-of-line explanations can signal a problem in the judging process.
Ranking vs rating: Empirical evidence suggests LLM judges perform better when choosing between outputs than when giving an absolute score​. In a pairwise setup, models like GPT-4 can identify which answer is more correct or better written with reasonable accuracy. When asked to just rate a single answer on a scale, they might exhibit scale drift (e.g., tend to rate things higher overall than a human would, or compress everything to a narrow band). To mitigate this, some frameworks use Elo ratings or TrueSkill by doing many pairwise comparisons and deriving a ranking for models from those.
Self-evaluation: If one uses the same model to evaluate itself, that can be problematic. It might be overly lenient or simply mirror its own confusion. A model might think an answer it wrote is good because it cannot see the mistake. Using a stronger or at least different model as the judge is advisable. For example, using GPT-4 to judge outputs from GPT-3.5 or another model. Or use an ensemble of judges (GPT-4, plus some rules, plus maybe another LLM). There's research on "ensembling" LLM evaluators or calibrating them with some human feedback to improve reliability.
Tricking LLM judges: There have been cases observed where a poorer answer can fool the LLM judge by, say, including an authoritative tone or even a fake reference. For example, if one answer boldly states a (wrong) fact and the other answer is unsure, the LLM judge might prefer the confident (but wrong) answer if its prompt isn't carefully instructing it to check factuality. Essentially, an LLM judge might be swayed by the style of the answer. If not told to prioritize correctness, it might favor a more verbose, polite answer over a terse but correct one. Crafting the judge prompt to avoid these pitfalls is an area of ongoing tweaking.

One approach is calibrating the LLM with a few examples of human judgments (few-shot prompting or fine-tuning on a small eval dataset) so it better mimics human criteria.
Another is agreement verification: for important evaluations, have multiple LLM judges (different prompts or models) and only trust the result if they agree with high majority. If they conflict, that example might need human review.
Techniques to reduce position bias: e.g., present outputs in random order or even have the LLM judge do a blind evaluation by mixing the text (though that's hard).
Using different prompts like first asking the LLM judge specific questions ("Is answer A factually accurate? Is answer B factually accurate? Which one followed instructions better?") and then aggregating those answers into a final decision. This structured approach can make the judging more systematic.

Hybrid evaluation methods

Human-in-the-loop + Automation: Use automated metrics or LLM judges to pre-screen or rank outputs, then have humans focus on tricky cases. For example, if you have 1,000 outputs to evaluate, you might use a toxicity classifier to automatically mark any clearly toxic outputs (and fail the model if any are found), use an LLM judge to score coherence and flag the lowest 100 outputs, and then have human evaluators carefully review those 100 (plus a random sample of the rest to spot-check). This way, human effort is concentrated where the model likely has problems, improving efficiency.
Blending qualitative and quantitative: While you might present overall automatic metric scores to summarize performance, you also include qualitative analysis from humans. Many research papers do this: "Model X achieved a BLEU of 0.25, close to Model Y's 0.27. To further understand, we conducted a human study: raters preferred Model X 55% of the time, citing more factual consistency, even though it had slightly lower BLEU." The combination gives a fuller picture.
LLM-assisted human eval: Interestingly, LLMs can even help human evaluators. An example: when humans evaluate long text (like a 5-page story by the model), it's hard to keep track of everything. An LLM could summarize the story or highlight potential issues (like "In paragraph 3, the model contradicted something from paragraph 1"). The human then makes the final judgment using these pointers. The human is still the judge, but the AI helps speed up the process.
Iterative eval and refinement: Hybrid evaluation can be part of the development cycle. For instance, a workflow could be:
Develop a new model version.
Run automated tests/benchmarks (perplexity, BLEU, etc.) - if it's worse than before on these, go back (regression).
If it passes, run an LLM-judge comparison against the old model on a variety of prompts - get a quick sense of wins/losses.
If the new model seems better or comparable, do a small human eval on key metrics to confirm quality.
If all looks good, deploy to a small user group and monitor real interactions (perhaps with metrics like user rating, or measuring if the new model answers questions that the old failed). This real-world data serves as further evaluation.
If that is positive, then fully launch the model.
In this cycle, automated and human evaluations happen at different stages with different scopes (offline vs online, small-scale vs large-scale).
Continuous feedback loop: In production, combining human and automated evaluation in a loop can maintain model performance. For example, some customer service bots allow users to rate answers or say "this was not helpful". Those signals can be aggregated (automated analysis of human feedback) to find failure patterns. Then, either humans or an LLM can go through those failure cases to categorize them (e.g., "Many failures are about technical questions the bot couldn't handle"). This can inform the next training iteration or prompt adjustment. Here, user (human) feedback is part of evaluation, and automated analysis of that feedback helps identify where to improve.
Ensembling evaluators: Another hybrid idea is to use multiple evaluation methods on the same outputs and treat the evaluation result as a combination. For instance, you might require that a new model should improve on at least one automatic metric without degrading others, and also be preferred by human judges. This multi-pronged criterion ensures you're not blindly optimizing one metric. If an output passes all automated checks (no toxicity, high similarity to reference, etc.) but humans still dislike it, that flags something the metrics missed.
Task-specific hybrid metrics: Sometimes a metric itself can be hybrid. For example, consider evaluating a dialogue system. You might create a composite score that includes a semantic similarity part (via BERTScore to a reference answer), a penalty for any rule violations (via regex or classifier, e.g., if it said something forbidden), and a small bonus if it uses polite wording (via a list of polite phrases). Such a metric mixes automated components (embedding similarity, classification) that reflect human priorities (accuracy, safety, politeness). The final "score" tries to approximate a human's holistic judgment. If tuned well, such composite metrics can be useful to rank models before a final human check.
Benchmark + human synergy: Use benchmarks to diagnose weaknesses, then human creativity to design new tests. For example, if a model aces all standard math problems but users still find it struggles with tricky word puzzles, evaluators can create new adversarial examples (perhaps by hand or with an LLM's help) to add to the evaluation set. Over time, this enriches the automated benchmark to include those human-identified edge cases.

Test it on a set of 100 known Q&A pairs (from a medical dataset) and measure exact match or F1 (automated).
Also measure BERTScore or semantic similarity to the reference answers (automated).
Have a medical professional review 50 of its answers for correctness (human).
Use an LLM like GPT-4 to double-check those answers as well (LLM judge).
Compare the professional's review with GPT-4's review: if they largely agree, we might trust GPT-4 to screen more questions.
After deployment, monitor user feedback: any low-rated answers are collected. Periodically, have a human go through those to see what went wrong (e.g., was the answer wrong, or was it correct but user didn't understand?).
Feed those findings back into improving the model and also into our evaluation set for the next version (closing the loop).

LLM model evaluations vs. LLM system evaluations

You use the same prompt format for all models (maybe a zero-shot prompt or a few-shot prompt).
You don't use external tools; you rely on the model's training knowledge.
You measure direct metrics like accuracy, BLEU, etc., on whatever task output the model produces.

A specific prompting strategy (maybe a well-crafted instruction or few-shot example context to steer the model).
Possibly a retrieval-augmented generation (RAG) setup, where before the model generates an answer, the system retrieves relevant documents from a database and feeds them into the prompt.
Post-processing or filters that modify the model's raw output (for example, formatting the answer, or censoring certain content).
User interaction loop: e.g., a chatbot that remembers past conversation (so the prompt includes chat history, which is part of the system state), or a tool-using agent that can call external APIs (like a calculator) as part of producing the final answer.

A standalone model is usually evaluated on static benchmarks under controlled prompts, whereas a full system is often evaluated on simulated or real user scenarios, possibly interactively or with custom test harnesses.
In a system, the prompt engineering is considered part of the "solution". A weaker model with excellent prompt engineering might outperform a stronger model with a poor prompt on certain tasks. So system eval measures the synergy of model and prompts.
Systems often have additional metrics beyond the model's output quality. For instance, in a RAG system, you might measure the quality of the retrieved documents (did the retriever find the correct info?). In a multi-turn system, you might measure conversation length or success rate of completing a task (like booking a flight through a dialogue).
Evaluating a system sometimes means treating it like a black box - give it an input (which might be complex, e.g. a user conversation or a knowledge-base query) and check the final output or outcome.

Prompt Engineering: How you prompt an LLM can drastically change its performance. So when evaluating systems, you must ensure you're using the best or intended prompt. For example, GPT-3 might get a question wrong with a naive prompt, but if you add "Let's think step by step" and a follow-up format, it might get it right. In a system, that prompt tweak becomes part of the system's design. So the evaluation of the system would show a much higher accuracy than evaluation of the base model with no prompt. This means system evaluation can sometimes be more optimistic because you're leveraging prompt hacks to cover the model's weaknesses. Evaluation should document the prompt used, because a different prompt is almost like evaluating a different model. Essentially, the prompt is an integral part of system behavior - it's like code. Thus, system eval may involve a phase of prompt tuning and then locking that prompt when testing. If comparing systems, one should ideally allow each system to use its optimal prompt rather than forcing a common prompt (since the goal is end quality, not model fairness, at that point).
Retrieval-Augmented Generation (RAG): In RAG systems, the model isn't expected to have all knowledge internally; instead, it pulls in information from an external source (like Wikipedia or a company document store). The evaluation of a RAG system must consider both retrieval performance and generation performance:
You might measure retrieval metrics like Recall@K (did the correct answer appear in the top K retrieved documents?). If the retriever fails, the model likely will output wrong info no matter how good it is.
You also evaluate the final answer: was it correct and well-formed? If the model had the relevant doc, did it use it correctly?
Sometimes, you evaluate the attribution: does the system properly cite the source of information from retrieval? If that's a requirement (like Bing's search-powered chatbot cites sources), the eval might check if the sources given actually support the answer.
A challenge is that if a system gets something right, was it because the model knew it or because it found the info? And if it got it wrong, was it model hallucination or retriever error? To pin this down, evaluators often do ablations: evaluate the retriever alone on a dataset (information retrieval metrics) and evaluate the generator given gold documents (to see the model's capability if correct info is provided). This way, you can tell if a system's failures are due to retrieval misses or generation mistakes. For instance, a study might report: "Our RAG system answered 85% of questions correctly. When the correct document was retrieved, the model answered 95% of those correctly, but it failed completely if relevant docs were missing. The retriever had a Recall@5 of 90%, indicating most failures were due to a few queries where retrieval failed." That gives insight that improving retrieval might boost the system more.
Pipeline Optimization: Many systems have a pipeline of steps. For example:
Pre-process user query (maybe classify its intent).
Retrieve documents if needed.
Formulate a prompt to the LLM with those docs.
Get the answer and post-process (e.g., remove any disallowed content).
Maybe have a final check (like a smaller model that verifies the answer isn't obviously wrong or offensive).
Evaluating such a pipeline might involve unit tests at each stage (evaluate the classifier in step 1, evaluate retriever in step 2, etc.) and overall tests at the end. It's similar to software testing: unit tests for components plus integration tests for the whole system. For LLM systems:
You might evaluate how well the pipeline handles different categories of queries. For example, does the system correctly route math questions to use a calculator tool? You'd test some math queries and see if step 1 classified them as math and step 3 included calculator usage. This is not just evaluating the LLM, but the decision logic around it.
Prompt optimization in the pipeline: Many pipelines involve prompts at multiple stages (one prompt to format user query for retrieval, another prompt template for the final answer). Each of those prompts might be tuned and should be fixed during eval. If you update a prompt, you should re-evaluate because it can change behavior a lot.
Integrated metrics: System evaluation might use composite metrics. For example, for a QA system, a metric like answer accuracy with source citation might be defined: the answer is only considered correct if it not only answers correctly but also cites a source document that contains that answer. This kind of metric evaluates the system's ability to not just be right but justify the answer. A base model alone wouldn't be tested on that, but a system would, because source citation is a system feature.
User Experience factors: A system evaluation often brings in metrics that matter to UX: latency (does the system respond quickly enough?), consistency (does it maintain persona/tone over a conversation?), handling of context (does it remember what the user said earlier?). For example, evaluating a chat system might involve multi-turn dialogues to see if it remembers context correctly. A base model might be good at single-turn QA, but evaluating it in a conversational system context might reveal it forgets instructions given two turns ago unless carefully managed. So the system (with a conversation history in the prompt or a memory module) is evaluated on these longer interactions. You could measure something like: in a 5-turn conversation, how often does the model need the user to repeat info vs it recalls it (context retention rate).

Standalone eval: test the model on trivia questions without any help - maybe it gets 30% correct because it's missing some facts in its parameters.
System eval: let the model use a web search (RAG) - now maybe it gets 80% correct because it can look up answers. The system is much better even though the model is the same, just augmented with a tool. So if you only looked at the standalone evaluation, you'd underestimate what the deployed system can do. Conversely, if the system has flaws (like slow retrieval making it laggy or a prompt that sometimes malfunctions), the system eval will catch issues that a standalone eval would miss.

Online vs. offline evaluation strategies

Offline evaluation (pre-release & controlled testing)

Conducted on historical or specially prepared data.
Environment is controlled - the inputs are known, the evaluation criteria are predefined.
No active user interaction; it's a one-way evaluation (we feed inputs, observe outputs, and measure against ground truth or desired properties).

Safety and Risk Mitigation: You're not exposing unvetted model behavior to users. If the model has serious flaws (e.g., it might produce offensive content or very wrong answers), you catch that offline rather than harming users or a company's reputation. It's like crash-testing a car dummy before letting a human drive.
Reproducibility: Each model variant can be evaluated on the exact same dataset, making comparisons fair and statistical significance easier to compute. Offline tests are repeatable - if the evaluation is automated, you can run it any number of times.
Depth of Analysis: You can afford to have detailed evaluation datasets and procedures that might not be possible live. For example, you can include many edge cases, long inputs, rare scenarios, etc., to really stress test the model. Offline, you can analyze the outputs thoroughly (even manually inspect them if needed).
Coverage: You can ensure certain important categories of inputs are covered in your offline tests (even if they're rare in real traffic). For instance, if you want to be sure your assistant can handle questions about medical advice safely, you include those in an offline test set, rather than waiting and hoping a user asks it.
Iterative development: Offline eval allows rapid experimentation - you can test many model variations in parallel on the same dataset and immediately see which is best, without deploying each to users. This speeds up development.

Not fully representative: No matter how good your test set is, it may not capture the full diversity or distribution of real user queries. Models might excel offline but then encounter unforeseen inputs in production and stumble. For instance, maybe none of your test questions had slang, but real users use slang. Offline eval might miss that.
Static: Offline tests often have a single correct answer or a static notion of what's good. They might not capture interactive aspects (like user might reformulate a question if the answer was unclear - offline eval typically doesn't simulate that back-and-forth).
Overfitting to eval: If developers tune the model or prompts heavily to ace the offline dataset (which they know), there's a risk of essentially overfitting to the test set. The model might be specifically good at the test questions but not generally (akin to teaching to the test). That's why having large and varied eval sets and not leaking them into training is important.
Lack of real user judgment: Offline metrics might say a model is better, but users might not feel that difference. For example, a new model could score higher in accuracy but might have a tone that users dislike; that wouldn't show until online evaluation.

Running the model on the entire validation portion of a dataset like XSum (summarization) and calculating ROUGE scores.
Using a set of 1000 curated prompts and comparing the average human rating of outputs (via a hired evaluation team) between two model versions - done entirely in-house before any deployment.
Adversarial testing: creating a list of trick questions or inputs known to cause problems, and checking how the model handles each (perhaps in a checklist style evaluation, marking pass/fail for each scenario).
Evaluation harnesses: Tools like EleutherAI's Language Model Evaluation Harness or HuggingFace's evaluation libraries allow running a suite of tasks offline easily. A developer might use these to get a quick report card of a model on dozens of benchmarks offline.

Online evaluation (Live deployment & user feedback)

Happens with real users or live data streams. Users may not even know they are part of an evaluation (if it's A/B testing two versions, all they see is the product working as normal).
The criteria of success are actual user behavior or feedback: clicks, ratings, conversion, time spent, error rates, etc., depending on the application.
Online eval is continuous - as long as the product is live, you can collect evaluation data (whereas offline is a one-shot on a fixed set).

Real-world validation: Online evaluation captures everything about the real usage: the distribution of queries, the context of use, and users' true preferences. It answers the ultimate question: does the model actually make the user experience better? There have been cases where a new model is objectively smarter, but users preferred the older one for some reason (maybe the new one was slower or too verbose). Only online testing would reveal that.
Uncovering unknown unknowns: Users are creative and varied. They will likely find patterns of input or edge cases that developers didn't think of. Monitoring the model online can surface new failure modes (e.g., "Oh, users keep asking this weird question and the model gives a bad answer - we never tested that offline!"). It keeps the model honest because it's meeting the full complexity of reality.
Measuring business metrics: If the LLM is part of a product, you might have key metrics like user retention, task success rate, customer satisfaction scores, revenue, etc. You can correlate model changes with movements in these metrics. For instance, if a virtual assistant model improvement leads to users needing to call customer support 10% less often, that's a huge win that only online deployment would show.
Continuous improvement: Online eval allows for an ongoing feedback loop. You can continually gather data - for example, logs of interactions - which can be used to improve the model (through fine-tuning or prompting changes). It's not just evaluation, but also data collection for training. Many deployed systems periodically retrain on logs (after filtering/labeling) to get better. In that sense, evaluation and training merge: user feedback becomes training data (with proper handling).

Risk: Deploying a not-fully-proven model to users can cause harm. If the model says something offensive or incorrect to a user, that's a real cost (it could upset users, cause misinformation, etc.). That's why online evaluation is often done incrementally - e.g., start with 1% of users (canary deployment) to limit potential damage.
Noise and Variability: User behavior has a lot of variance and external influences. Say you deploy a new model version and at the same time some unrelated event causes more people to use the service, or a holiday week changes usage patterns - it can be tricky to isolate the effect of the model change. That's why carefully designed A/B tests (with randomization and statistical analysis) are needed to draw conclusions. You need enough sample size to get statistically significant results on metrics, which can take time.
Delayed or Implicit Feedback: In offline eval you get an immediate score. In online, feedback can be implicit. For example, if a user is unhappy with an answer, they might just ask the question differently or abandon the chat - you have to infer dissatisfaction from such signals. Explicit feedback (like thumbs-up/down buttons) is great but not all users will use them. Also, some outcomes are long-term (did the user come back next week? That could indicate they found the AI useful or not). So, connecting a model's performance to these indicators can be complex and often requires careful logging and analysis.
Constant Distribution Shift: The real world isn't static. Users' queries today might be different from a month later (trends, news, seasonality). The model might gradually perform worse if the world changes (e.g., new slang comes up, new events that the model doesn't know about). Online evaluation is necessary to catch this drift. It's a challenge because you have to decide when the model's performance has meaningfully changed - which might trigger a need for an update. Essentially, you're never "done" evaluating online; it's an ongoing process.
User Privacy and Ethics: Using real user data for evaluation must be done carefully to respect privacy. If analyzing logs, ensure personal data is handled properly (anonymization, opt-outs if required). If doing experiments, often users should be informed in terms of service (many companies have a clause that says something like "we may test different versions of the service to improve quality"). Particularly for sensitive domains, there might be ethical oversight needed.

Average response length, average response time.
Percentage of sessions where users rephrased the question (could indicate the model didn't answer well initially).
Frequency of escalation (if it's a support bot, how often it fails and hands off to a human).
User ratings over time.
Any flagged content (if there's an automated filter that flags possible policy violations, monitor how often that triggers).

Stage 1: offline eval (must pass criteria).
Stage 2: internal testing (employees or a small trusted user group).
Stage 3: online A/B test with a small percentage of users.
Stage 4: gradually ramp up to more users if metrics look good.
Stage 5: full deployment if proven.
Even after full deployment, keep a fallback ready (maybe the previous model) in case something unexpected shows up and you need to roll back.

Offline advantages: controlled, fast iteration, lower cost, no user risk.
Offline downsides: may not reflect actual user needs or model's real usage performance.
Online advantages: real feedback, measures what ultimately matters (user happiness, success).
Online downsides: risk, complexity, slower to get results (need enough users/time), requires robust monitoring infra.

W&B Weave in LLM Evaluation

Overview of W&B Weave and its features

Tracing LLM Execution Flow: Weave can capture the series of steps an LLM app takes - for example, the prompt given to the model, the model's output, and any intermediate computations. If your application uses a chain-of-thought or a tool-using agent, Weave can log each function call the agent makes. Developers add one line of code or a decorator, and Weave will log the inputs/outputs of those functions (which constitute the LLM's reasoning trace)​. This results in a trace tree that you can inspect. In other words, Weave helps you visualize and inspect the execution flow of your LLM, including intermediate steps and final outputs​.
Automated Logging of Prompts and Responses: With Weave, you don't have to manually print and copy-paste prompts and model answers to analyze them. It automatically logs all prompts sent to the LLM and the responses received, along with metadata (like model name, timestamp, parameters such as temperature). This is extremely useful for evaluation because you can review exactly what was asked and what was answered for each test case. You can compare prompts and see how slight prompt changes affect output.
Support for Popular LLM Libraries: Weave integrates with libraries and APIs like OpenAI's API, Anthropic's API, LangChain, LlamaIndex, etc. If you use those, Weave can hook into their calls without much extra code​. For example, if you use LangChain to manage a conversation, Weave can capture each part of the chain. This means as you evaluate a system that uses these tools, Weave will record each component's input/output (e.g., what was retrieved vs what was generated).
Logging Custom Metrics: You can log arbitrary values or metrics to Weave (since it's built on W&B's logging). For instance, after generating an answer, you could compute a BERTScore or a toxicity score for that answer and log it. Weave will store those metrics alongside the trace. This way, you can later filter or sort outputs by these metrics. For example, find all cases where the toxicity score was high, and then examine those outputs in the UI.
Comparison and Benchmarking: Weave provides a UI where you can compare different runs. A run might correspond to an evaluation of a model or system version on a set of inputs. If you have run A (model v1 on test set) and run B (model v2 on same test set), the UI can show you aggregate metrics of both, and also allow side-by-side comparison of outputs for each input. You might have a table where each row is a test prompt, and columns show model v1 output vs model v2 output, and maybe a human reference or score. This makes it easy to spot where one model did better than another. Essentially, Weave can act as a benchmarking dashboard for all your model versions and prompt variations.
Visualization and Search: Because it logs data in a structured way, you can query it. For instance, filter to only the examples where the model's answer was incorrect (if you have a flag/metric for correctness). Or search for a keyword in the model outputs (maybe to find all instances of a certain mistake or phrase). The UI also offers charts - e.g., a distribution of a metric. If you logged, say, answer lengths or latency, you could get a histogram or time-series plot of those. This helps in evaluating efficiency metrics or variability.
Traces and Panels: In the Weave UI, a trace of an LLM call (especially if multi-step) can be visualized. For example, if an agent had a conversation with itself to reason (scratchpad), Weave might show that chain-of-thought. This can be used in evaluation to understand why the model gave a certain answer. Maybe you see in the trace that it retrieved the wrong document, leading to a wrong answer - that analysis is valuable for improving the system. Weave's ability to pinpoint where things went wrong (prompt, retrieval, generation) is an evaluator's friend.
Collaboration and Reports: Since Weave is hosted (or can be self-hosted), team members can log in and see the evaluation results. You can share a link to a specific comparison or make a report. This is great for keeping track of evaluations: e.g., a report might summarize "Model v2 vs v1 on our evaluation set - see examples where v2 still fails." This serves as documentation and can be revisited later.

Using Weave for automated tracking, benchmarking, and visualization

Setup Weave in your code: You initialize Weave at the start of your script or application (e.g., weave.init(project="LLM_eval", run="experiment_42")). If you have functions that call the LLM or do important steps, you annotate them (Weave provides an @weave.op decorator for Python functions)​. This tells Weave to automatically log calls to that function. For example, if you have a function get_answer(prompt): ... that calls the model and returns an answer, decorating it will log its input (the prompt) and output (the answer). This minimal code change yields a lot of logged info.
Run your evaluation dataset through the model: Suppose you have 100 test questions. You call your get_answer function on each (perhaps in a loop). As it runs, Weave captures each invocation. You might also log a metric after each answer (like wandb.log({"accuracy": correctness}) if you have a way to score it). All this data (prompts, answers, metrics) gets sent to the W&B backend.
Examine results in Weave UI: Now, on the W&B dashboard, you open the run. You might see an interface where each call is recorded as an object. You can click on a specific input to see details. For example: Prompt = "Who is the CEO of OpenAI?" -> Model Output = "Sam Altman." You might have logged a correctness = True because that's correct. Weave lets you scroll through all examples easily. If any output looks wrong, you can tag it or make a note. This is far more efficient than manually checking logs because you have a nice list and possibly reference answers if you logged those.
Benchmarking multiple runs: Say you try a new prompt format or a new model. You do another run with the same test questions. Now in Weave, you can compare run A vs run B. The interface could show something like:
Question 5: "What is the capital of France?"
Run A answer: "Paris."
Run B answer: "The capital of France is Paris."
Both are correct (accuracy True), but maybe run B is more verbose. If brevity was a goal, you note that.
Or maybe on another question, Run A was wrong, Run B is right - you can filter to just those, to see what types of questions improved. This comparison helps quantify improvements and also qualitatively understand them (by reading actual outputs side by side).
You can also look at overall metrics: say run A got 80/100 correct, run B got 90/100 correct. Weave would show that (since you logged accuracy per question, you can compute aggregate easily in the UI or in a report). So you have both the high-level metric and the granular examples to support it. This kind of analysis is critical in LLM eval, where a single overall score often doesn't tell the full story.
Visualization: Suppose you also logged the response time or token usage for each answer (Weave can capture model outputs and you can subtract timestamps to get latency, etc.). You could then create a plot in the W&B UI of "token usage per answer sorted by prompt length" or something to evaluate efficiency. Or a bar chart of accuracy broken down by category (if you label your questions by type). Weave allows custom panels, so you might make a panel that filters your examples by category and shows accuracy. For instance, "Science questions vs History questions accuracy" if you labeled them. If you see the model is much worse in Science, you know to focus there. These visual analytics are hard to do manually but straightforward when all data is logged structurally.
Case studies / LLMOps pipeline integration: In an LLMOps context (continuous integration of LLMs), you might set up that every time you train a new model or change a prompt, an evaluation run is executed and logged to Weave. Over time, you accumulate runs (like a history of experiments). Weave becomes a repository of model behavior. If someone asks "did we fix the issue with question X in the latest model?", you can literally search for that question in the latest run and see the answer. Or compare the answer now vs 3 versions ago - all conveniently accessible. This historical benchmarking is valuable for ensuring you're making progress and not re-introducing old errors.
Example from the field: An AI startup building a coding assistant could use Weave to track how each new model version handles a set of programming problems. They log for each problem: prompt, code output, did it run successfully, any errors, etc. In Weave, they can quickly spot which problems each version fails. They might notice version 2 started failing on something version 1 solved - maybe due to a regression in how it handles a certain instruction. This insight lets them fix the issue before release. Without Weave, they might have missed that regression.

Organization: all your eval data in one place.
Benchmarking: easy comparison across runs.
Visualization: charts and tables for metrics and outputs.
Traceability: ability to trace errors to specific steps (thanks to logged traces).
Collaboration: share results with team, add notes, build a knowledge base of model behavior.

Use cases: Weave in action for LLMOps

Use case 1: Chatbot Quality Monitoring - A company has a customer support chatbot powered by an LLM. They want to continuously evaluate and improve it. They set up Weave in their staging environment with test conversation transcripts. Each nightly build of the bot is run through a suite of 50 simulated conversations (covering various topics like billing issues, technical support, etc.). Weave logs the conversation flows: user message -> bot reply (with chain-of-thought hidden reasoning, since they use a prompt that lets the bot reason internally). The next morning, the team checks the Weave dashboard. They see a particular scenario (password reset) where the bot's performance dropped (it gave a confusing answer). They inspect the trace and realize the knowledge base retrieval brought an outdated article. This directs them to update their knowledge source or adjust the retrieval query. In the same dashboard, they notice overall a slight improvement in politeness scores (maybe measured by a classifier) after a tweak they did. They compile these findings in a Weave report to share with the product manager. This workflow repeats, with Weave ensuring they catch issues early and have data to show improvements.
Use case 2: Prompt Engineering with Weave - A developer is trying to optimize the prompt for a summarization task. They have 5 candidate prompt templates. Instead of manually trying each on a few examples, they create a loop that runs all 5 prompts on a set of 100 articles and logs results to Weave (each prompt variant as a separate run, or all in one run with a group tag). In Weave, they can now compare the summaries side by side for each article. Perhaps Prompt C yields the shortest summaries but sometimes misses key points, while Prompt A yields longer but more complete summaries. They also logged ROUGE and a human preference score for a subset. Weave shows Prompt B had the highest ROUGE, but when they look at human preferences (they had 3 coworkers quickly rate a sample via Weave's UI or an exported table), Prompt D was actually preferred for readability. With all this evidence, they decide on a final prompt that balances things. Weave helped manage the complexity of testing many prompts and aggregating the results.
Use case 3: Regression testing - As models are updated, it's vital to ensure they don't get worse on important cases. One can use Weave as a regression testing harness. Suppose you have a list of "critical test cases" (maybe 200 hand-picked questions that your model must handle correctly). Every time you update the model, you run these through and log to a Weave run labeled with the version. Weave can then quickly highlight if any test that used to pass is now failing. Maybe version 1 got case #37 correct, but version 2 got it wrong. That will show up as a difference when you compare runs. If not caught, that regression could have gone into production. With Weave, you catch it and fix it (maybe by integrating a specific prompt adjustment or adding that scenario to training). Over time, you build a robust regression suite, and Weave's interface makes checking it far easier than manually diff'ing outputs.

Implementation example: Evaluating LLM outputs with Weave

Step 1: Instrument the code with Weave

import weave
weave.init(project="trivia-bot-eval", name="model_v1_eval")

# Suppose we have a function that calls the LLM API
@weave.op()  # This will log calls to this function
def get_answer(question: str) -> str:
    # Call the LLM (pseudo-code, could be OpenAI API or others)
    response = llm_api.ask(question)
    return response

Step 2: Run evaluation and log metrics

# Example test set
test_questions = [
    {"q": "Who is the CEO of OpenAI?", "expected": "Sam Altman"},
    {"q": "What is the capital of France?", "expected": "Paris"},
    # ... more questions
]

import wandb
correct_count = 0
for item in test_questions:
    ques, expected = item["q"], item["expected"]
    answer = get_answer(ques)
    # Simple accuracy metric:
    is_correct = (expected.lower() in answer.lower())
    correct_count += 1 if is_correct else 0
    # Log the result as an artifact of this call
    wandb.log({
        "question": ques,
        "model_answer": answer,
        "expected_answer": expected,
        "correct": is_correct
    })
    
# After loop, log overall accuracy
accuracy = correct_count / len(test_questions)
wandb.summary["overall_accuracy"] = accuracy
weave.finish()  # ensure all data is flushed

Each question triggers get_answer, which Weave logs (the prompt and the result).
We then use wandb.log to log some fields. Notice we log question, model_answer, expected, correct. Since these have the same keys each time, W&B will form a table of these logged items.
We also set a summary metric for overall accuracy.

A table of all logged examples, showing question, model_answer, expected_answer, correct (True/False). Because we also used @weave.op, the trace of get_answer is recorded too (which in this simple case just shows the function input and output; if there were sub-steps they'd show up).
Weave might automatically highlight where correct is False (since that's a boolean metric).
We have a summary accuracy number for the run.

Step 3: Visualizing and interpreting results in Weave

We might see an overall accuracy of, say, 90%.
We can browse the table: see each Q, model's answer, expected answer, and whether it was correct.
We notice, for example, one that was marked incorrect: Question: "Who is the CEO of OpenAI?" - Expected "Sam Altman" - Model Answer: "OpenAI's CEO is Sam Altman as of 2021." It contains "Sam Altman", so maybe our simple check would count that as correct (depending on implementation). If it was incorrect, maybe the model gave the wrong name.
If a model gave a wrong answer, we can click that entry. The trace might just show the function call. Not much internal detail here (since it's a direct call). But if it were a more complex pipeline, we could see where it went wrong (maybe retrieved wrong info).
We could add a filter: correct == False, to show only the questions the model got wrong. This quickly gives us a feel for what types of questions those are. Suppose the model got "Who is the CEO of Google?" wrong (just hypothetical) - maybe it gave an outdated name. That tells us it has outdated knowledge in some areas.
We could then plan to fine-tune the model or adjust the prompt to fix that.
We can also sort by question or such, or search. If we search "France", we'd find the France question and see the answer.

We run the above again, perhaps with name="model_v2_eval".
Now in the project, we have two runs. Weave can compare them. For instance, we could see model_v1 vs model_v2 on each question:
If model_v2 fixed some previously wrong answers, those will show as now correct.
If model_v2 introduced any new mistakes, that will show too.
Suppose model_v2 had overall accuracy 95%. We confirm that improvement by seeing those additional correct answers highlighted.

A summary of Weave benefits

Systematic logging: You won't forget to record an output or metric; everything is logged, avoiding anecdotal evaluation.
Reproducibility: You can rerun the evaluation anytime and compare with past runs. Weave keeps a history.
Insight into model internals: If you log intermediate steps (like chain-of-thought, retrieved docs, etc.), you gain interpretability, which is a best practice to understand model errors (and not just treat it as a black box).
Visualization and collaboration: Plots and tables are generated for you, and you can discuss them with team members, which is important for consensus on model quality.
Efficiency: It cuts down the time spent writing custom evaluation scripts and building reports. That time can be used to expand evaluation (more test cases, more metrics) or to actually fix problems found.

Best practices and common pitfalls in evaluating LLMs

Best practices for robust LLM evaluation

Define Clear Evaluation Objectives: Start by deciding what qualities you care about in the model, and choose metrics and methods aligned with those. Is factual accuracy paramount? Is user satisfaction the ultimate measure? Different goals might require different eval setups. For example, if truthfulness is critical (like in a medical assistant), include a factual accuracy evaluation and maybe penalize any hallucinations strongly. If creativity is desired (like a story generator), you might rely more on human subjective evaluation than on any single metric. Being clear on objectives helps avoid "chasing a metric" that doesn't truly reflect success
Use Multiple Metrics (Holistic Evaluation): No single metric will capture everything for an LLM. A best practice is to evaluate along multiple dimensions.* For instance, when assessing a chatbot, measure:
Task success: did it answer correctly? (accuracy/F1 or human judgment)
Fluency: was the language well-formed? (maybe grammar score or human rating)
Helpfulness: did it address the user's intent? (human rating)
Safety: did it avoid toxicity? (toxicity classifier score)
Efficiency: was it concise enough? (length or user feedback) By looking at a dashboard of all these, you get a holistic picture. Often there are trade-offs (one model might be more correct but also more verbose than another). With multiple metrics, you can balance those or at least be aware of them, rather than inadvertently optimizing one at the cost of others. The HELM framework specifically advocates multi-metric evaluation for transparency​.
Combine Automated and Human Evaluation: As discussed, automated metrics are invaluable for scale and speed, but human evaluation is the gold standard for many aspects. Best practice is to use automated methods for the properties they can reliably measure (e.g., BLEU for closeness to a reference, or a classifier for toxicity detection), and use human judgment for subjective or nuanced aspects. For instance, use an automated metric to filter obviously wrong answers, then have humans evaluate the remaining for quality. Or use an LLM judge to rank outputs, but then have a human verify the final top choices. This hybrid approach harnesses the efficiency of automation and the nuance of human insight.
Ensure Fair and Representative Test Data: Your evaluation is only as good as the test data. Make sure to use diverse and representative datasets that cover the range of inputs your model will see. If your model will be used globally, include questions or prompts from different cultures, languages (if multilingual), and user groups. If certain edge cases are critical (e.g., questions about rare diseases for a medical bot), include those in eval even if they're rare in the wild. Beware of test sets that are too narrow or too easy. It's a good practice to include some challenging, adversarial examples in evaluation to probe the model's limits. Moreover, continually update your test sets as new failure modes are discovered (keeping a core stable set to measure progress over time, but adding new sections for new kinds of inputs).
Maintain Evaluation Consistency (when needed): When comparing model versions, try to keep evaluation setup consistent - same test questions, same prompts - so you attribute differences to the model change, not the eval change. On the other hand, if you improve the evaluation method (realize a better metric or additional test cases), apply it going forward (and possibly re-evaluate older models with the new method for reference). Clearly version-control your evaluation code and data, so you know with what criteria a model was judged. Reproducibility in evaluation is important for trust - if someone reruns your eval on the same model, they should get the same results.
Emphasize Realism in Evaluation: Whenever possible, use real user data (respecting privacy) in evaluation. For example, log a sample of actual user queries (anonymized) and use those as an eval set. This ensures you're evaluating on what people actually ask, not just neatly curated questions. If real data is scarce (like for a new product), create realistic simulations. For instance, simulate dialogues that a user might have. The closer your eval is to reality, the more confidence you have that good eval results will translate to happy users. Additionally, consider performing beta tests or user studies as part of eval - that is, give the model to a small group of target users and get qualitative feedback. This can reveal issues that metrics might not, such as "the tone is off-putting" or "it answered correctly but I didn't understand the explanation."
Evaluate for Bias and Fairness: Given the importance of fairness, incorporate specific tests to detect bias. A best practice is to include contrastive prompts: e.g., evaluate answers to "Tell me about a doctor" vs "Tell me about a female doctor" vs "Tell me about a male doctor" and see if there are unjustified differences or stereotypes. Or test the model with names from different ethnicities in a similar question and see if it behaves differently. Use established benchmarks or create your own scenarios to cover protected attributes (race, gender, religion, etc.). If possible, quantify bias (e.g., "toxicity of completions when prompted with identity X vs Y"). Ensuring your evaluation framework has a bias audit component is key to catching unfair behavior. Moreover, fairness in evaluation results themselves (ensuring your human raters are diverse, etc.) is also a consideration.
Track Progress Over Time: Continuously track metrics across model versions and time. Maintain a spreadsheet or (better) a Weave/W&B project or similar where each model's eval results are logged. This helps identify if you're hitting diminishing returns on a metric or if a recent change caused a dip somewhere. It also helps when communicating to stakeholders - you can show how the model has improved (e.g., "We reduced factual error rate from 15% to 5% over the last 3 months"). It also ensures lessons learned are preserved, avoiding repeating the same mistakes (if a model had an issue and it was fixed, tracking helps ensure it doesn't resurface unnoticed).
Set Thresholds and Expectations: For metrics that matter, set target thresholds. For example, "the model should achieve at least 90% on these critical Q&A pairs" or "no more than 1 in 100 responses should be rated as unsafe". These act as guardrails. If a model doesn't meet them, it shouldn't be considered for deployment. If it exceeds them, maybe focus on other aspects next. Having such standards helps prevent the allure of "cool new model with lower performance" from taking over, and it formalizes the acceptance criteria. Of course, pick reasonable thresholds based on user needs and current model abilities (often start with something attainable and raise it as models improve).
Incorporate User Feedback into Evaluation: If your LLM is deployed, leverage user feedback (explicit ratings, or implicit signals like whether the user re-asked the question) as part of ongoing evaluation. For example, if users can rate answers from 1–5 stars, use the average star rating as a metric (with appropriate caution since users can be inconsistent). Look at what fraction of sessions are rated good vs bad and track that. Even in offline eval, you might use transcripts of when users said "This isn't what I needed" as evaluation prompts to see how a new model handles them. Basically, closing the loop between deployment and eval ensures your evaluation stays relevant to actual usage.
Perform Error Analysis Regularly: After each evaluation, do a thorough error analysis on a sample of failures. Classify the errors: how many are factual mistakes, how many are because the question was ambiguous, how many are due to formatting issues, etc. This not only guides improvements (which area to work on) but also ensures you truly understand what the metrics are telling you. Two models might both get 90% accuracy, but their 10% errors might be very different - one might hallucinate wildly occasionally, while the other consistently fails on math problems. Error analysis brings such patterns to light. Document these findings, as they become part of the evaluation knowledge base for your project.
Balance Quantitative and Qualitative Evaluation: Numbers are essential, but also read the outputs. Always allocate time to read through a set of outputs from the model, both good and bad ones. This qualitative feel can catch things metrics won't - like if the answers are technically correct but unnecessarily verbose or lacking empathy, etc. Many teams do a "blind eval" where evaluators read outputs from two models side by side without knowing which is which and give preference judgments. This is a qualitative method yielding quantitative results (preference %). Use metrics to narrow down candidates, but use human eyes to make final calls, especially if deploying to users.
Test in the wild (if possible): If feasible, do a limited release (to internal staff or a small user group) and treat that as part of evaluation. Sometimes models behave differently under real user pressure (distribution shift, or users intentionally stress-testing it). Observing this early, as part of eval, can save headaches. In these controlled beta tests, gather transcripts and issues reported, and fold that back into refining the model and the evaluation process for next time.
Consider the Cost-Quality Trade-off: In evaluation for production, it's not just about quality metrics. Consider inference cost and latency. A model might be slightly better but twice as slow or expensive (especially relevant when choosing model sizes or using ensemble vs single model). Evaluation should incorporate these concerns - e.g., measure response time and maybe have a requirement like "must answer within 2 seconds 95% of the time". Sometimes a smaller model might be acceptable if it's much cheaper, depending on business needs. So, treat efficiency metrics (speed, memory, cost per query) as part of the eval criteria, not separate. W&B Weave or similar can log those too for each run.
Iterate on Evaluation Itself: Treat your evaluation setup as a living project. After each round, ask: did our evaluation miss anything important? Are there new metrics or tests we should add? Continuously improve your evaluation methodology. This might mean adopting new metrics that research shows correlate better with human judgment, or expanding human eval guidelines as you learn what to look for. Avoid becoming complacent with one set of metrics - as models get better, you might need to raise the bar or add more challenging tests (e.g., once factual accuracy is high, maybe focus on more subtle things like consistency or style).

Common pitfalls to avoid in evaluating LLMs

Overfitting to Benchmarks: Focusing too narrowly on improving specific benchmark scores can lead to a situation where the model's improvements are just exploiting quirks of the test data, not truly general progress. This is an instance of Goodhart's Law: "when a metric becomes a target, it ceases to be a good metric." For example, a model might learn to do extremely well on a known dataset like TriviaQA by memorizing lots of trivia during fine-tuning, but that doesn't mean it's better at open-domain QA generally. Or a summarization model might be tuned to maximize ROUGE by copying whole sentences from the source (which ups overlap), producing high ROUGE but possibly a less readable summary. To avoid this, use a variety of eval sets and metrics so you're not optimising one number at the expense of overall ability. Also, periodically do sanity checks on truly new data that the model wasn't tuned on. If a model's benchmark score jumps suspiciously with minor changes, double-check on a fresh eval set. Keep some eval data secret from even the team if possible, to truly test generalization. 
Ignoring Outliers and Failure Modes: A pitfall is to just look at aggregate metrics and ignore the "long tail" of issues. An LLM might have a 98% success rate, but the 2% failures might include serious issues (like producing harmful content or very egregious errors). If you only pay attention to the high-level metric, you might miss that. For instance, an AI assistant might do great on normal questions but occasionally produce a very offensive joke due to some trigger phrase. If you only track average performance, you won't catch that single bad case which could be a big problem if it reaches a user. Solution: inspect the failures, even if they are few. Use evaluation specifically targeted at finding any catastrophic failures (red-teaming). In other words, don't be lulled by high average scores; ensure you also examine worst-case behavior. One way is to track metrics like "# of critical failures in N trials" and aim to make that zero, even if it doesn't show up in average accuracy.
Equating Human-like output with Correctness: LLMs can produce very fluent, confident-sounding answers that are wrong. Evaluators might be fooled by this if not careful (even automated evaluators like other LLMs can be fooled). Don't assume an answer is correct just because it's well-written or detailed. This is why having ground truth references or reliable checking methods is important for factual tasks. A related pitfall is using only superficial metrics like BLEU which might be satisfied by a fluent answer that uses the right words but doesn't truly answer the question. Always double-check factuality separately. This is also a caution when using LLM-as-judge: the judge might give a high score to an answer that sounds good but is actually incorrect in content. Mitigate this by explicit criteria focusing on correctness and by manual spot checks.
Inadequate Human Eval Design: If you do human evaluation without careful design, you can get misleading results. Common issues: unclear instructions to raters, too few raters (leading to noise), raters not representative of target users, or asking them the wrong questions. For example, asking "Is this answer good?" might get inconsistent responses. Instead, breaking it down ("Is it correct? [Y/N], Is it sufficiently detailed? [1–5], etc.) can yield more actionable data. And ensure multiple people evaluate each item to average out individual bias. Another pitfall is not training the annotators - if the criteria are complex (e.g., "logical consistency"), give examples of consistent vs inconsistent outputs so they know what to look for. Neglecting these can result in unreliable human eval where you either think a model is better or worse than it is due to annotator idiosyncrasies. Always review the inter-annotator agreement; if it's low, your human eval may need redesign.
One-off Evaluation: Evaluating a model once and considering the job done is a pitfall especially in dynamic environments. Models can drift (if fine-tuned more, or if used in an online setting where context changes). If you don't continuously evaluate, a change in input distribution or a slight model update might degrade performance and go unnoticed. Best practice is continuous or at least frequent re-evaluation, as we discussed. Treat evaluation as an ongoing process, not a final exam the model passes and that's it. Many failures in deployed AI come from assuming past performance will continue without monitoring (e.g., a model's accuracy drops as new slang emerges or new info after its training cutoff date becomes relevant).
Not Involving Domain Experts when needed: For specialized applications, a common pitfall is relying on layperson evaluation when you need an expert's eye. For example, evaluating a legal document summary model with crowdworkers might be problematic since they can't judge if important legal points were preserved. Similarly, medical answer correctness should ideally be evaluated by medical professionals. It's understandable that experts are costly, but for critical aspects, at least a sample should be reviewed by an expert to set a quality bar. Perhaps use crowdworkers to filter obvious cases and have experts judge borderline ones. If domain expert eval is impossible during every iteration, at least have them evaluate major versions to validate the model. Failing to do this might mean you think a model is fine (because laypeople thought the answer sounded good) but an expert would notice a subtle but crucial error.
Neglecting User Experience in Evaluation: Purely technical evaluation sometimes misses UX aspects. A model might get the right answer but in a way that's not user-friendly (too verbose, too terse, or not sufficiently polite). If your end-users care about style/tone, your eval should too. For instance, a customer support bot's evaluation should include something like a politeness/tone assessment. If you only eval accuracy, you might deploy a bot that comes off as rude or robotic. Always tie evaluation to the end-user experience: consider things like response time (users hate waiting), format correctness (if the app expects JSON, is it always valid JSON?), and user engagement metrics. For example, maybe model A and B have similar accuracy, but in an online test users asked follow-up questions 30% more with model A - maybe because its answers were less clear. That indicates model B provided more complete answers. So incorporate such signals. Not doing so is a pitfall where a model that's "better on paper" isn't actually better for the user.
Over-reliance on Automated Scores: We touched on this but to reiterate: metrics like BLEU, ROUGE, or even learned ones like BERTScore are imperfect. Don't make decisions solely on small differences in these if they aren't corroborated by human judgement. For example, chasing a +0.5 BLEU could be meaningless in terms of actual quality. Sometimes teams get into local optimum chasing a metric up and down with tweaks that don't really improve real output (like adjusting phrasing to please BERTScore). Use automated metrics as a guide, not gospel. A healthy approach is: if metric and human eval agree, you can be confident. If they disagree, investigate why. Perhaps the metric is penalizing something that doesn't matter, or the model is improving in a way the metric can't detect. In such cases, trust the human more and consider improving your metric.
Ignoring Confidence/Uncertainty: LLMs often don't have calibrated confidence. A pitfall is evaluating them only on correctness of outputs, and not whether they signal uncertainty appropriately. For some applications, it's important that if the model is unsure, it says so or defers. If you evaluate just on giving an answer, you might push the model to always answer even if it's guessing (leading to fluent but wrong answers). It's a good practice to evaluate calibration: e.g., when the model says "I'm not sure," is it actually on a question it would have gotten wrong? And if it's confident (doesn't hedge), is it usually right? Some evaluations measure this (Brier score, or checking correctness conditional on expressed certainty). If not explicitly evaluated, a model might look great (because it attempts every answer and gets most right) but the ones it gets wrong, it states wrongly with confidence. In many real scenarios, a wrong but confident answer is worse than a humble non-answer. So consider that in eval.
Not Evaluating Robustness: A model might work on clean input but fail if input is slightly noisy or unusual. A classic pitfall: not testing how the model handles typos, different wording, or adversarially phrased inputs. For example, if all your test questions are well-formed, you might miss that the model fails when a question has a typo or is phrased colloquially. It's wise to include some perturbed inputs in evaluation to gauge robustness. Similarly, test variants of the same question (paraphrases) - a robust model should answer them all correctly, not just one phrasing. If you only test one phrasing, you might overestimate model's consistency. Another aspect is multi-turn consistency in a conversation: if you only test single-turn, you may not catch that the model contradicts itself across turns or forgets info. Expand eval to those scenarios if relevant. Essentially, test the model's ability to handle input variations and maintain output stability. Otherwise, minor changes at runtime might lead to major drops in performance that you never saw in eval.

Hands-on tutorial: Implementing an LLM evaluation pipeline

Step 1: Set up the environment and data

An LLM access (could be OpenAI API, or a local model via Hugging Face).
wandb library for Weave.
Any other evaluation helper libraries (maybe nltk for BLEU, or transformers for BERTScore, etc., if needed).

# Sample evaluation data
evaluation_data = [
    {
        "question": "Who wrote the novel '1984'?",
        "reference_answer": "George Orwell"
    },
    {
        "question": "What is the largest planet in our solar system?",
        "reference_answer": "Jupiter"
    },
    {
        "question": "In what year did the World War II end?",
        "reference_answer": "1945"
    },
    # ... (more items)
]

Step 2: Initialize W&B Weave for logging

import wandb
import weave

# Initialize a Weave run for this evaluation
wandb.login()  # if not already logged in (requires W&B API key)
weave.init(project="LLM_evaluation_demo", name="model_v1_eval")

Step 3: Define the LLM query function (instrumented for logging)

import openai
# Set your OpenAI API key if using OpenAI (or set environment variable)
openai.api_key = "YOUR_OPENAI_API_KEY"
@weave.op()
def get_model_answer(question: str) -> str:
    """Query the LLM to get an answer for the question."""
    # Here we use a simple prompt format: just the question.
    # In a more complex scenario, you might add a system prompt for context or instructions.
    prompt = question
    try:
        response = openai.Completion.create(
            engine="text-davinci-003",  # or another model
            prompt=prompt,
            max_tokens=100,
            temperature=0  # deterministic for evaluation
        )
        answer_text = response['choices'][0]['text'].strip()
    except Exception as e:
        answer_text = f"<ERROR: {e}>"
    return answer_text

We send the question as the prompt to the model.
We set temperature=0 for deterministic output; that's often done in evaluation to get repeatable results.
We limit max_tokens to ensure answers aren't too long.
We capture any API errors and return them as a placeholder answer (so we can catch if any queries failed).

Step 4: Running the evaluation loop and logging metrics

Exact Match (correct or not, by a simple string comparison or regex).
BLEU score for each answer (though in single-answer case BLEU might be trivial, but just for demonstration).
Answer length just as an additional piece of info.

import nltk
from nltk.translate.bleu_score import sentence_bleu  # If using NLTK BLEU, ensure the resources are downloaded
# nltk.download('punkt')  # if needed for tokenization

results = []  # to store results if we want to use them in code later
total_exact_matches = 0
for item in evaluation_data:
    ques = item["question"]
    ref = item["reference_answer"]
    # Get model's answer (this call is logged by Weave)
    model_ans = get_model_answer(ques)
    
    # Compute Exact Match (case-insensitive containment for simplicity)
    is_exact_match = (ref.lower() == model_ans.strip().lower())
    total_exact_matches += int(is_exact_match)
    
    # Compute BLEU (here using a simplistic approach with the reference as single reference list of tokens)
    ref_tokens = [ref.split()]  # list of one reference list
    ans_tokens = model_ans.split()
    bleu = sentence_bleu(ref_tokens, ans_tokens, weights=(1, 0, 0, 0))  # unigram BLEU for simplicity
    
    # Simulate a human rating: let's say 5 if exact match, 3 if answer contains ref as substring, 1 otherwise
    if ref.lower() in model_ans.lower():
        human_score = 5 if is_exact_match else 3
    else:
        human_score = 1
    
    # Log to W&B
    wandb.log({
        "question": ques,
        "reference": ref,
        "model_answer": model_ans,
        "exact_match": is_exact_match,
        "bleu_unigram": bleu,
        "human_score": human_score,
        "answer_length": len(model_ans.split())
    })
    
    # Save result in list too (if we want to print or analyze in Python)
    results.append({
        "question": ques,
        "reference": ref,
        "model_answer": model_ans,
        "exact_match": is_exact_match,
        "bleu": bleu,
        "human_score": human_score
    })
# After loop, log summary metrics
accuracy = total_exact_matches / len(evaluation_data)
wandb.summary["exact_match_accuracy"] = accuracy
wandb.summary["average_bleu"] = sum(r["bleu"] for r in results) / len(results)
wandb.finish()

For each question, we got the answer via get_model_answer. That function call is logged by Weave.
We then logged several fields:
The question, reference answer, and model answer (so we have context in the logs).
exact_match as a boolean (True/False) if the model's answer exactly matches the reference (case-insensitive exact match, for simplicity). In a real QA eval, exact match or F1 (for partial) is common.
bleu_unigram: since reference and answer are short, we used unigram BLEU (which basically measures precision of tokens, similar to accuracy of content words).
human_score: a made-up metric to mimic human evaluation; in real life, this would come from an annotator. We decided to correlate it with correctness in this simplistic scenario.
answer_length: just number of tokens in model's answer, to see if the model is verbose or concise.
We logged these for each example using wandb.log(). This means in Weave, each example is like a row with these columns. We've effectively created a mini dataset of Q, ref, answer, and metrics.
We also logged some aggregate summaries at the end: overall accuracy and average BLEU.
We finished the W&B run to make sure data is saved.

A table of logged examples will be visible, showing all those fields. We can sort or filter by them. For instance, filter exact_match == False to see which questions the model got wrong.
We can also see the get_model_answer traces for each call. If we click on a specific example, it might show the trace input and output (which in this case is just question -> answer text).
The human_score we logged can be compared with exact_match to see if our simulated human agreed with correctness mostly.
The aggregate exact_match_accuracy summary is displayed (in the run overview or summary panel). If it's, say, 0.66 (66%), that's the model's accuracy on this small set.
average_bleu similarly.

for res in results:
    print(f"Q: {res['question']}")
    print(f"Model: {res['model_answer']}  | Reference: {res['reference']}")
    print(f"Exact Match: {res['exact_match']}, BLEU: {res['bleu']:.2f}, Human Score: {res['human_score']}")
    print("----")

Step 5: Interpreting the results

Q: Who wrote the novel '1984'?
Model: George Orwell  | Reference: George Orwell
Exact Match: True, BLEU: 1.00, Human Score: 5
----
Q: What is the largest planet in our solar system?
Model: Jupiter is the largest planet in our solar system.  | Reference: Jupiter
Exact Match: False, BLEU: 1.00, Human Score: 3
----
Q: In what year did the World War II end?
Model: World War II ended in 1945.  | Reference: 1945
Exact Match: False, BLEU: 1.00, Human Score: 3
----

The model answered all questions with the correct information (as seen by BLEU 1.0 - all key tokens present, and by the content).
However, it didn't exactly match the reference for two because it phrased the answer in a full sentence (which our exact_match strict check marked as False). So exact_match_accuracy might show 33% (1/3 correct by strict matching) whereas in reality the model got them all correct, just with extra words.
Human score (our simulated one) gave partial credit (3 out of 5) for those full sentence answers since they still contained the right info.
This tells us something important: our exact match metric was too strict. The model's outputs "Jupiter is the largest planet in our solar system." technically contain the correct answer "Jupiter". A human evaluator would consider that correct. So we realize we should use a looser metric (like consider it correct if the reference is a substring of the model answer, or use F1 overlap as is common in SQuAD eval). This is an example of refining evaluation after seeing results.

exact_match_accuracy was low (0.33) which might initially look bad, but BLEU was high (1.0 average). BLEU being 1.0 indicates the answers had all the reference words (in this case references were one-word, so BLEU 1.0 means that word was present).
This discrepancy between exact_match and BLEU/human indicates that the model is actually correct but our exact match metric underestimates it.
So, we'd adjust our evaluation: maybe use a metric like F1 (which in these cases would be 1.0 since all words overlap ignoring order).
This shows why having multiple metrics and examining outputs is good; a single metric could be misleading.

If the model had gotten something wrong, we'd see it in the logs (exact_match False and answer not containing the reference). We'd note what kind of question it was. Suppose the model said "George Orwell" incorrectly as "Orwell George" (just a weird example) - exact match would fail. We'd categorize that as a formatting issue, not a knowledge issue.
Or if it gave a wrong year for WWII end, that's a knowledge error.

Evaluate on a larger set or different category of questions.
Perhaps test some adversarial phrasing: e.g., "The novel '1984' was written by who?" to see if the model still gets it. We could augment our evaluation_data and rerun.

Step 6: Integrating Weave visualization and reports

In the W&B app, build a simple dashboard: one panel showing overall accuracy (we logged that summary), one panel listing the examples table. Weave might automatically provide some of this.
Use Weave's filter to focus on incorrect cases.
Perhaps create a report (link this very blog post) summarizing: "Model v1 got X% exact, but after refining metric it's effectively Y% correct. It tends to answer in full sentences. All answers were factually correct. Next, we will test more open-ended questions or multi-sentence answers."

Step 7: Example code integration for eontinuous Evaluation

Possibly accept a model identifier or prompt variations.
Run this evaluation.
Log results to Weights & Biases.

Step 8: Extending the pipeline

Add more metrics, like a toxicity check using Perspective API or a HuggingFace model, log toxicity_score for each answer.
Evaluate speed: measure time for each get_model_answer call and log it (Weave can capture it or we do manually).
If we had multiple models to evaluate (say GPT-3 vs GPT-4), we could run this pipeline twice and then use Weave to compare.
For human eval at scale, one could set up a mechanism where after logging outputs, a separate process queries a human labeling service and then logs their ratings back (perhaps updating the human_score field later or logging a new entry with human feedback).
Integrate with a CI: you could script this to run nightly and perhaps automatically flag if accuracy drops below a threshold.

Define evaluation set and metrics.
Run model and collect outputs.
Log results in an organized way (using Weave/Weights & Biases).
Examine outputs to ensure metrics align with actual quality.
Use insights to adjust model or evaluation (or both).
Repeat for new model versions, and compare results to track progress.

Future directions in LLM Evaluation

More Fine-Grained and Dynamic Evaluation: Instead of coarse metrics like single scores, future evaluation might produce richer profiles of model behavior. For example, generating a "capability report" for a model: highlighting strengths and weaknesses across many dimensions automatically. Projects like Google's Eval++ or others aim to have models evaluate models across nuanced criteria. One idea is letting models generate test questions for each other (an approach to expand eval data automatically). As LLMs get more involved in evaluating, we'll see more dynamic evals that adapt to the model: e.g., if a model is very good at one category, the evaluation system might automatically start asking harder questions in that category to find the boundary of capability.
AI-Assisted Evaluation and Interpretability: We've started using LLMs as judges, but future work may make them more aligned evaluators. For instance, research on prompting an LLM to not just score but also detect specific issues (like logical fallacies or biases in an output) will improve. There's a concept of 'universal evaluator models' that could be trained on large amounts of human feedback data to evaluate any output on various traits. If successful, such evaluator models (or ensembles of them) could rapidly accelerate evaluation - imagine having an AI panel that gives detailed reviews of each output: factuality grade, style grade, bias flags, etc., which correlate well with what a human expert would say. This could be the equivalent of unit tests for every generated sentence. We're not there yet, but progress in this direction continues.
Continuous and Real-Time Evaluation: As models may be updated frequently or even on-the-fly (online learning), evaluation may become continuous. Instead of big static eval events, tools will continuously monitor streams of outputs and compute metrics in real-time. For example, a deployed chatbot could have a live dashboard of its performance: updated satisfaction score as conversations happen, drifting topic success rates, etc. This moves towards the idea of "evaluation pipelines in production", where the model is constantly being evaluated against key criteria and triggers alarms if something goes off. Already, companies do monitor for spikes in bad outputs; future systems will likely automate more of this (like anomaly detection on output quality metrics). This ties into MLOps - just as we monitor uptime or latency, we'll monitor correctness and safety.
User-Centric and Contextual Evaluation: We might see evaluation that is more user-centric. For example, personalization: a model's performance might be evaluated differently for different user groups or preferences. If an assistant adapts to a user, evaluation might need to account for user satisfaction on a personal level. Also, context-awareness: evaluating how well models handle context lengths of 10K tokens vs 1K, or how they adapt to different tones when user sets a tone preference. Standard evals now don't cover these well. Future eval frameworks might include simulating a variety of usage contexts (like switching personas, languages mid-conversation, etc.) to stress test adaptability.
Benchmark Evolution and Meta-Evaluation: New benchmarks will continue to emerge to test aspects like reasoning (BIG-Bench tasks, math benchmarks like MATH or GSM8K), common sense (ASDiv, etc.), multimodal understanding (as vision+language models rise). But an interesting direction is meta-evaluation: evaluating the evaluation methods themselves. For instance, researchers are exploring how well various metrics correlate with human judgment​.* This research will guide which metrics are trustworthy. We might end up using learned metrics that have been validated extensively. Also, community-driven evaluation platforms (like Dynabench from Facebook or Eval Harnesses on HuggingFace) allow continuous updating of benchmarks with new challenging examples - likely these will grow, making evaluation a more community-driven, dynamic process.
Interpretable Evaluations and Model Understanding: There's a push for interpretability of models (like understanding the internals or why a model arrived at an answer). In the future, evaluation might include interpretability checks: for example, evaluating how well we can predict a model's errors by looking at its internal activations, or evaluating how consistent its reasoning path was. Techniques like TCAV (Testing with Concept Activation Vectors) or probing models with certain inputs to see if they have knowledge embedded could become part of eval. For example, instead of only asking the model questions, one might also probe its embedding space or attention patterns to evaluate if it "knows" a fact (even if it didn't express it correctly). This is quite researchy, but as interpretability tools mature, they could enter the evaluation toolkit, especially for safety-critical systems (where you might want to eval not just output, but whether the model's internal state contains risky concepts, etc.).
Robustness and Adversarial Testing: Future evaluation will put even more emphasis on robustness. Models might be evaluated against adaptive adversaries - other AI systems designed to find their weaknesses. For instance, an adversary model could try to produce inputs that trick the LLM into a wrong or harmful answer, and the frequency of failures under this stress-test is recorded. This is somewhat happening in red-team events (like Anthropic's adversarial training setups) and will likely formalize into benchmarks. The CLOUDS framework (Checklist for Language Understanding with Dynamic Substitution), or other adversarial methodologies, might become standard in eval suites to ensure models aren't just good on average but also in worst-case.
Ethical and Regulatory Evaluation: As AI gets regulated, we might see standardized evaluation criteria mandated by law or policy. For example, a data protection regulation might require evaluating that a model doesn't reveal personal data from training (so an eval test for memorization of personal info might be required). Similarly, bias and fairness might be legally required eval components for certain applications (e.g., any AI used in hiring must undergo fairness testing on certain demographics). We might see the equivalent of FDA trials for AI in healthcare - evaluation protocols that models must pass before deployment. This will drive the development of more rigorous, unbiased evaluation processes, and probably more transparency (like evaluation reports being published or models coming with a "Nutrition label" of eval results across key dimensions).
Human-AI Team Evaluation: In many applications, AI will work with humans, not replace them entirely. Evaluation will then also consider the human-AI team performance. For example, in a medical diagnosis scenario, an LLM might assist a doctor. The evaluation of such a system might involve seeing how much it improves the doctor's accuracy or speed, and also if it sometimes misleads the doctor. So future evaluation might include user studies where humans use the AI and we measure outcome differences. This is more a UX evaluation but crucial: some models might not independently be super accurate, but if they provide useful hints that a human can interpret, the team's performance is high. Conversely, a model might be accurate but phrased in a confusing way that leads humans to make mistakes combining it. So evaluating human-AI interaction outcomes is a frontier, involving disciplines of HCI (Human-Computer Interaction) in the eval loop.
Synthetic Data for Evaluation (and Training): Synthetic data generation by LLMs themselves can help address the lack of test data in certain domains. We might see processes like:
Use a strong LLM to generate many question-answer pairs on niche topics, then use those as an eval set for another model (with careful vetting).
Or generate adversarial test cases as mentioned.
Synthetic data could also fill gaps: e.g., generate a dataset of polite vs impolite responses to test style transfer capabilities.
One must be cautious because an LLM might inject subtle biases or errors into synthetic data, which could skew evaluation. But techniques like instructing the model to generate diverse and unbiased test cases, and then maybe having a human review or another model filter them, are being explored. In essence, LLMs generating their own challenge sets is a concept (sometimes called Bots red-teaming bots). This can dramatically expand the scope of eval beyond what humans alone could come up with.
Cross-Modal and Embodied Evaluation: As LLMs merge with other modalities (images, video, robotics), evaluation will also cross modalities. For example, evaluating a model that describes images - you need image captioning benchmarks, visual question answering benchmarks. Or an AI agent that uses an LLM for planning in a virtual environment - you need to evaluate success on tasks in that environment. The future likely holds integrated evals: e.g., an AI assistant that can see and talk might be evaluated on a scenario like "Describe what you see and answer questions about it while being factual and not biased." This requires combining NLP metrics with vision metrics. New benchmarks like VQAv2 (for vision Q&A), or EAI (Embodied AI tasks in simulators) will play a role, and LLM eval frameworks will incorporate those for multimodal models. Essentially, evaluation is broadening as AI systems become more general.
Benchmark Fatigue and New Paradigms: There's recognition of benchmark fatigue - models topping out on benchmarks quickly (superhuman on many NLP tasks now). This means we need new evaluation paradigms. One idea is interactive evaluation: testing models in interactive environments or dialogues where questions aren't i.i.d from a test set but depend on the model's previous answers. This can test consistency and depth of understanding better. Another idea is stochastic or distributional evaluation: measuring not just the best output, but the distribution of outputs a model can produce (to evaluate diversity or the probability of failure). For example, does the model have a 1% chance to output a grossly wrong answer? If you only look at one sample, you might miss that, but evaluating multiple samples (or the probability mass of certain tokens) could catch low-probability bad outcomes.
Community and Crowdsourced Evaluation: Projects like Dynabench have users trying to stump models, and when they succeed, that example is added to the test set. This dynamic adversarial approach, done by humans, will likely continue. The future might involve crowdsourced eval competitions, where people are incentivized to find cases where a model fails, and those become part of evolving benchmarks. This keeps evaluation up-to-date and model developers on their toes.
Explainability in Evaluation Criteria: There is growing interest in not just whether a model got something right, but also if it can explain its answer or reasoning. Future evaluation might give models a score based on the quality of their explanations. For instance, if a model says "The answer is X because …", we might evaluate the 'because' part for correctness and coherence. This can encourage models to not only be right but also verifiably right (if they state reasoning steps that can be checked). Some current eval tasks (like the Chain-of-Thought Prompting evaluation) informally do this; but it could be formalized, like a metric for logical consistency between explanation and answer, or a human rating of explanation helpfulness.