AI agent evaluation: Metrics, strategies, and best practices



Table of contents

What are autonomous agent evaluations (and why do they matter)?

Validation of performance: We want to confirm that the agent actually accomplishes its goals (for example, answers questions correctly or navigates a robot to a target) at an acceptable success rate. Evaluation provides evidence of how well the agent works.
Identifying weaknesses: By testing an agent in a structured way, we can discover failure modes or areas where it struggles (for example, a chatbot that handles straightforward questions well but fails on tricky ones).
Improving through iteration: Reliable metrics allow developers to iterate and improve the agent. If we change the agent’s design or model, we need to measure if it got better or worse. Without proper evaluation, developers are left guessing whether their application is improving its accuracy, latency, cost, and user experience.
Comparing approaches: In the rapidly evolving AI field, many new models and agent frameworks are emerging. For instance, 2024 witnessed the launch of over a hundred new AI models. Consistent evaluation lets us compare different agents or techniques fairly. A standardized benchmark or score can tell us which agent is best suited for a task.
Resource and cost management: Evaluations often include measuring efficiency (like time and computational cost). This is increasingly important when deploying agents at scale or using expensive API calls. If two agents have similar accuracy but one is much more costly to run, that’s a crucial insight from evaluation.

Key metrics for evaluating AI agents

Latency: This measures how fast the agent responds or completes a task. Latency can be measured per action (for example, how many seconds an agent takes to decide or produce an output) or end-to-end for a full task. Lower latency means a more responsive agent, which is important for user experience and real-time applications.
Cost: In AI agents, cost often refers to monetary or computational expense. Many modern agents use API calls to large models (for instance, calling an OpenAI or Anthropic language model which charges per token) or consume significant compute resources. Cost can be measured in dollars per 1000 operations, GPU-hours, or any unit that reflects expense.
Token usage: For agents that rely on language models, token usage is a key metric. Tokens are chunks of text processed by the model. Higher token usage usually correlates with higher latency and cost (since most API pricing is per token). Monitoring token counts helps developers optimize prompts or the number of interactions.
Accuracy/success rate: This is a measure of effectiveness – how often does the agent achieve the correct or desired outcome? It might be defined as percentage accuracy or a success/failure rate, depending on the task.
Robustness: Robustness measures how well the agent maintains performance under varying conditions, including unexpected inputs or perturbations. A robust agent is not easily thrown off by edge cases or adversarial conditions.
Adaptability: Adaptability is the agent’s ability to handle new tasks or changing requirements without extensive reprogramming. Evaluations might include tests like transfer learning performance or online learning ability.
Reliability: Consistency of results across multiple runs is another important aspect. A reliable agent produces repeatable, stable outcomes even when faced with similar inputs repeatedly.

Evaluation strategies: Automated vs human-in-the-loop vs reproducible workflows

Automated benchmarks and testing: Evaluating agents in an automated way means setting up benchmarks or test suites where the agent is run through many tasks and metrics are recorded without human oversight. Automated evaluation is fast, scalable, and consistent, allowing for statistically significant comparisons across different agent versions or techniques.
Human-in-the-loop assessments: Not everything an agent does is easily evaluated by a computer. Human judgment is sometimes needed to assess aspects like tone, creativity, or user experience. Although human evaluations are qualitative and more time-consuming, they capture nuances that automated metrics might miss.
Reproducibility in evaluations: It is essential that evaluation setups be reproducible. This means controlling for variables, fixing random seeds, and documenting the configuration of the agent. Reproducible evaluations provide transparency and facilitate debugging as well as fair comparisons between different models or configurations.

Best practices for effective agent evaluation

Define clear success criteria: Be explicit about what constitutes success for your agent. Whether it is achieving a certain accuracy or meeting a specific response time threshold, clear goals help drive evaluation design.
Track multiple metrics and balance them: Avoid optimizing for a single metric in isolation by creating a dashboard that displays all key metrics side by side.
Use baselines and comparisons: Compare current agent performance against a baseline or a previous version. This contextual comparison can highlight improvements or regressions.
Automate evaluation in the development workflow: Integrate evaluation as a regular part of your CI/CD or research pipeline. Continuous evaluation helps catch regressions early.
Log detailed data for debugging: When an agent fails or performs suboptimally, detailed logs of the evaluation process (including the sequence of actions and intermediate outputs) help pinpoint the issue.
Include human feedback where appropriate: If your agent interacts directly with users, consider mechanisms to gather and log user feedback on the agent’s performance.
Consider robustness tests: Introduce stress tests or edge cases in your evaluation process to ensure the agent performs reliably under adverse conditions.
Document and version everything: Keep clear records of your evaluation setup, including any changes to test scenarios or success criteria.
Iterate and refine: Use evaluation results as guidance for improving the agent. As new challenges emerge, expand the set of metrics to capture them.

Tutorial: Tracking and visualizing agent evaluations with W&B Weave

Step 1: Setup W&B Weave

Install the Weave library via pip along with any necessary agent or LLM libraries:

pip install weave openai

Note: You should also create a free W&B account and obtain an API key from your account settings.
Initialize Weave in your code. For example, in a Python script or notebook, import and initialize a project:

import weave
import openai
import os
import asyncio # Needed for running evaluations
from weave import Evaluation

weave.init(project_name="agent_evaluation_demo")

Step two: Define the evaluation dataset and scoring functions

# Collect your examples
examples = [
    {"question": "What is the capital of France?", "expected_answer": "Paris"},
    {"question": "Who wrote 'To Kill a Mockingbird'?", "expected_answer": "Harper Lee"},
    {"question": "What is the square root of 64?", "expected_answer": "8"},
]

# Define a custom scoring function
@weave.op()
def match_score(expected_answer: str, output: dict) -> dict: # Changed model_output to output
    # output will be the dictionary returned by our agent function
    generated_text = output.get('generated_text', '')
    # Here is where you'd define the logic to score the model output
    return {'is_match': expected_answer.lower() == generated_text.lower()}

Step 3: Instrument the agent function for evaluation

openai_api_key = os.environ.get("OPENAI_API_KEY", "YOUR_ACTUAL_API_KEY") # Replace YOUR_ACTUAL_API_KEY if not using env var
client = openai.OpenAI(api_key=openai_api_key)

@weave.op()
def answer_question(question: str): # Accepts 'question' from the dataset
    start_prompt = {"role": "system", "content": "You are a helpful agent."}
    user_prompt = {"role": "user", "content": question}

    # Use the client object and dot notation for openai>=1.0.0
    response = client.chat.completions.create(
        model="gpt-3.5-turbo", # Using a common model for demo
        messages=[start_prompt, user_prompt]
    )

    # Access the content using dot notation: response.choices[index].message.content
    generated_text = response.choices[0].message.content

    # Return output as a dictionary, including 'generated_text' for scoring
    return {'generated_text': generated_text}

Step 4: Create and run the agent evaluation

# Create an Evaluation instance
evaluation = Evaluation(
    dataset=examples, # Provide the dataset
    scorers=[match_score] # Provide the list of scoring functions
)

# Run the evaluation using the agent function
# Use asyncio.run() because evaluation.evaluate is an async function
print("Running evaluation...")
await evaluation.evaluate(answer_question)
print("Evaluation complete. Check W&B UI.")

5: Visualize and analyze evaluation results in W&B Weave

Summary statistics: You'll see aggregated results from your scoring functions (e.g., the average is_match score across all examples).
Examples table: A table listing each example from your dataset. For each example, you can see:
The original input (question).
The agent's output (model_output, including the generated_text).
The calculated scores (is_match).
Details of the trace for that specific example run (the answer_question call and the nested OpenAI API call), including its latency, token usage, and cost.



Trace view: You can still click into individual runs to see the detailed step-by-step trace, just like in the simpler tracing example.



Compare Evaluations: You can create a view to quickly and easily compare important scores and values associated with your traces.



Conclusion

Resources

Weights & Biases Announces General Availability of W&B Weave for Enterprises to Deliver Generative AI Applications with Confidence
Agent Evaluation: Holistic Agent Leaderboard (HAL) | by Sulbha Jain | Medium
AI Agent Evaluation: Insights from LiveMathBench and G-Pass@k | by Anna Alexandra Grigoryan | Medium
Monitoring Amazon Bedrock Agents with W&B Weave | by Dave Davies | Medium
Track LLM inputs & outputs | W&B Weave
Weave Integration - CrewAI