Tracing your CrewAI application

Table of contents

Understanding CrewAI and its capabilities

How CrewAI enables monitoring and analysis

Figure: W&B Weave's trace interface capturing a CrewAI execution. The screenshot shows a hierarchical trace of a crew.kickoff() run, including two main tasks (Research Analyst and Report Writer). Under each task, we see sub-operations like LLM calls (e.g., llm.complete) and their parameters. On the right panel, the input and output of the selected operation are displayed (in this case, an excerpt of an investment report generated on the topic "AI in Material Science"). This detailed trace helps developers monitor agent interactions and analyze performance at each step.

Getting started with tracing a CrewAI application

Steps to get started with Weave

Install CrewAI and Weave: Ensure you have Python 3.10–3.13 installed, then install the required packages via pip. For example: pip install crewai weave. If you plan to use CrewAI’s extra tools (like web browsing or other integrations), you can install them with pip install 'crewai[tools]' as well. This will set up both the CrewAI framework and W&B Weave library in your environment.
If you haven’t already, sign up for a free Weights & Biases account: This is needed because Weave will log traces to the W&B cloud where you can view them. Once you have an account and are logged in (for example by running wandb login in your terminal or using an API key), you’ll be ready to record and visualize traces.
Initialize Weave in your application: In your Python script or Jupyter notebook, import weave and call weave.init(...) at the beginning of your code to initialize tracing. You should give a project name to weave.init (e.g., weave.init(project_name="crewai_demo")) – this name will be used to organize your trace logs. After initialization, the console will usually print a URL to the Weave dashboard for this project, which you can open in your browser.
Define your agents, tasks, and crew: Set up your CrewAI agents and tasks as usual using the CrewAI API. This involves creating an LLM instance (e.g., a GPT-4 based model), then creating Agent objects with specific roles/goals, and Task objects that describe what each agent should do. For example, you might have: a Researcher agent whose goal is to gather information, and a Writer agent whose goal is to produce a report. Each Task is assigned to an agent (e.g., a research task using the Researcher agent, and a writing task using the Writer agent). Finally, you create a Crew that includes those agents and tasks, and decide on a process (sequential or parallel) for how tasks should be executed. For instance, you might use a sequential process so that the research task completes before the writing task starts (as one would expect in this example).
Run the crew: With everything defined, kick off the multi-agent workflow by calling crew.kickoff(inputs={...}) with any required input parameters. In our running example, you might pass an input like {"topic": "AI in material science"} to tell the Researcher what to investigate. Once you call kickoff(), CrewAI will start the agents and they will begin executing their tasks. During this execution, Weave is automatically capturing every operation – from the top-level kickoff down to each LLM call made by the agents. You can print the final result or outputs as the code runs, but the real magic is that a trace of the entire process is being recorded behind the scenes.
View the trace in Weave: After the run completes (or even while it’s running, since Weave can stream data), open the Weave dashboard URL that was provided when you initialized the project. In the web interface, you’ll see your project and a list of trace runs (each run is typically identified by the function you called, like crewai.Crew.kickoff). Click on the latest run to open the trace visualization. Here you can explore the trace, which will show: all LLM calls and their metadata, the sequence of task executions and agent actions, performance metrics like execution time and token counts, and any errors or exceptions that occurred. Using the sidebar or timeline, you can drill into specific operations – for example, inspecting what prompt an agent sent to the LLM and what response it got. This trace view is your primary tool for debugging and understanding the behavior of your CrewAI application.

Tutorial: Tracing a customer support triage system with CrewAI and W&B Weave

Setup and installation

pip install crewai wandb weave datasets openai

import os
import wandb
import weave
from crewai import Agent, Task, Crew, LLM, Process

# Set your OpenAI API key (ensure this is configured; you can also use os.environ)
os.environ["OPENAI_API_KEY"] = "<YOUR_OPENAI_KEY>"

# Initialize Weave for tracing (creates a project for logs)
wandb.login()
weave.init(project_name="customer_support_triage")

Loading a sample dataset for support queries

from datasets import load_dataset

# Load the CLINC150 out-of-scope dataset (use the 'imbalanced' version for full label set)
data = load_dataset('DeepPavlov/clinc_oos', 'imbalanced', split='test')
# Choose an example customer query from the dataset
sample = data[42]
user_query = sample['text']
true_category = sample['label_text']
print(f"Sample user query: {user_query}")
print(f"True intent (for reference): {true_category}")



Defining CrewAI agents and tasks

Classifier Agent: Determines the query’s category (intent). For simplicity, it will use an LLM to classify the query into one of the known support intents (like direct_deposit, card_activation, fraud_report, etc.), or flag it as “out-of-scope” if it doesn’t match any category.
Responder Agent: Generates a helpful response based on the classified category. If the query is within a known category, this agent will attempt to answer it. If the category is out-of-scope (or if classification confidence is low), it will respond with an escalation notice (simulating handing off to a human agent).

# Create a deterministic LLM instance (using OpenAI's GPT-3.5 Turbo model)
llm = LLM(model="gpt-4o", temperature=0)

# Define the Classifier agent
classifier = Agent(
    role="Support Classifier",
    goal="Identify the category of the customer's query (e.g., billing, account, card issue, or out-of-scope)",
    backstory="A customer support AI specialized in triaging requests by intent.",
    llm=llm,
    verbose=True
)

# Define the Responder agent
responder = Agent(
    role="Support Agent",
    goal="Provide a helpful response to the customer based on the query category, or escalate if out-of-scope.",
    backstory="An AI assistant that answers customer questions or escalates them to a human if it cannot handle them.",
    llm=llm,
    verbose=True
)

# Task 1: Classification Task

classification_task = Task(
    description=(
        "Classify the customer query into a support category. Query: \"{query}\"\n"
        "Return ONLY the category name from this list: direct_deposit, card_issue, billing, fraud, atm_support, transfer, oos"
    ),
    expected_output="A single category name only (e.g., 'transfer' or 'billing' or 'oos').",
    agent=classifier
)

# Task 2: Response Generation Task
response_task = Task(
    description=(
        "You have the user's query and the classification result in your context. "
        "Based on that, write either a helpful answer or an escalation notice."
    ),
    expected_output="A support reply or escalation message.",
    agent=responder,
    context=[classification_task],
)

Adding a guardrail for fallback logic

from typing import Tuple, Any

@weave.op(name="guardrail_validate_classification")
def validate_classification(result: Any) -> Tuple[bool, Any]:
    """
    Guardrail check for classification task.
    Returns (True, cleaned_result) if category is valid, or (False, error_info) if invalid.
    """
    try:
        # Extract the raw output from the result
        if hasattr(result, 'raw'):
            category = str(result.raw).strip()
        else:
            # Fallback if structure is different
            category = str(result).strip()
            
        # List of known categories
        known_categories = ["direct_deposit", "card_issue", "billing", "fraud", "atm_support", "transfer", "oos"]
        if category not in known_categories:
            # If the classifier gave something unexpected, flag it
            return (False, {"error": f"Unknown category '{category}'", "code": "CLASSIFICATION_ERROR"})
        if category == "oos":
            # Out-of-scope category triggers fallback/escalation
            return (False, {"error": "Query is out-of-scope", "code": "ESCALATION_TRIGGER"})
        # If valid and in-scope, return cleaned category
        return (True, category)
    except Exception as e:
        return (False, {"error": "Exception in guardrail", "code": "SYSTEM_ERROR", "details": str(e)})

# Attach the guardrail to the classification task
classification_task.guardrail = validate_classification

Initializing the Crew and kicking off the workflow

# Create the crew with the two agents and their tasks in order
crew = Crew(
    agents=[classifier, responder],
    tasks=[classification_task, response_task],
    process=Process.sequential,  # tasks will run sequentially
    verbose=True
)

# Run the crew on the sample query
result = crew.kickoff(inputs={"query": user_query})
print("\nFinal output from Responder agent:\n", result)

Observing the Trace in Weave



I was able to use the failed status to assist in troubleshooting my prompts, and get the classifier running properly.

Classification Step: Verify the classifier’s prompt (it should contain the user’s query) and the category it predicted. In our example, we expect something like "direct_deposit" as the output.
Guardrail Check: The guardrail function validate_classification will appear as a step. If the category was valid and not "oos", it returns True with the category. If it had returned False (e.g., category was "oos"), you would see the error code (ESCALATION_TRIGGER) in the trace. This makes it easy to spot when the system decides to escalate.
Response Generation: The responder agent’s step will show the final answer. Check that it addressed the query appropriately. For instance, if the category was direct_deposit, the answer might list steps to set up direct deposit. If the category had been out-of-scope, the answer would be a polite escalation message (as per our agent’s goal/logic).

Building and orchestrating multi-agent workflows with CrewAI

Use cases for multi-agent workflows

Automated research and reporting – A team of agents can collaboratively gather information from numerous sources, analyze the data, and synthesize it into comprehensive reports with minimal human input. For instance, one agent could search literature and news, another agent interprets and summarizes findings, and a third agent compiles a final report. This is useful in domains like market research, academic research reviews, or competitive intelligence.
Content creation pipelines – Rather than a single AI trying to do everything, you can assign specialized agents to each phase of content production. For example, a Researcher agent collects facts, an Outline Creator structures the piece, a Writer drafts the content, and an Editor agent refines and checks the draft. This division of labor can produce higher-quality articles, marketing copy, or documentation much more efficiently than a single-agent approach.
Customer support automation – Multi-agent setups can handle complex customer service tasks. You might have a Triager agent that classifies incoming support queries, then delegates to an agent specialized in billing issues or technical troubleshooting, etc. Agents can escalate queries to more advanced agents or humans if needed. This kind of crew can operate 24/7, improving response times and consistency in support for industries like e-commerce or IT services.
Business intelligence and data analysis – In a business intelligence scenario, different agents can focus on different data sources or analysis methods and then combine their insights. For example, one agent analyzes market trend data, another parses customer feedback, and another looks at sales figures. A manager agent can then aggregate these findings into a coherent analysis for decision-makers. This multi-agent approach mirrors how an analytics team might divide tasks and can yield comprehensive intelligence reports faster.
Software development assistance – AI agents can collaborate to speed up software development tasks. You could have an agent that generates code given a specification, another that reviews or tests the code, and another that writes documentation for it. By orchestrating these agents, you create an automated workflow from requirement to tested code. This has applications in rapid prototyping or assisting developers by handling repetitive parts of coding and documentation.

Conclusion

Agent interactions (prompts, responses, token usage)
Guardrail checks (validations, fallbacks, and escalations)
Task orchestration (the flow of work across your crew)