What is LLM observability?

Table of contents

What is LLM observability?



Common issues in LLM applications

Hallucinations: LLMs sometimes generate information that is factually incorrect or entirely fabricated, despite sounding confident. This phenomenon, known as hallucination, often occurs when the model is confronted with queries outside its knowledge scope. Instead of admitting it doesn’t know, the model may produce a plausible-sounding but false answer. Hallucinations can mislead users and propagate misinformation, so it’s critical to catch and correct them – especially in domains requiring factual accuracy (medicine, finance, etc.).
Prompt injection attacks: Also called prompt hacking, this is a security issue where a user intentionally crafts an input that manipulates the LLM into deviating from its intended behavior. For example, a user might try to trick a chatbot into revealing confidential information or producing disallowed content by injecting malicious instructions in a prompt. Without safeguards, prompt injection can cause an LLM to output harmful or unauthorized material. Observability can help detect such attacks by analyzing input patterns and the model’s responses for signs of manipulation.
Latency and performance bottlenecks: LLM-powered applications can suffer from high latency, especially if using very large models or multiple chained calls. Users might experience slow responses if the model or its pipeline isn’t optimized. Performance issues can stem from external API delays, heavy computation, or inefficient prompt designs. Furthermore, as usage scales, the system might hit throughput limits. For instance, a model could handle single requests quickly but choke when many queries arrive concurrently. Continuous monitoring of latency and throughput metrics is necessary to identify bottlenecks and maintain a smooth user experience.
Cost unpredictability: Many LLMs incur cost based on token usage (the amount of text processed). Applications may face unpredictable or rising costs if prompts or outputs become longer than expected or if user traffic spikes. A slight change in how users interact (e.g. asking for more detailed answers) can multiply token consumption. Without observability, these cost drivers might go unnoticed until a bill arrives. Tracking metrics like tokens per request and total usage is essential for cost management, helping to optimize prompts and limit overuse.
Bias and toxicity: LLMs learn from vast internet data and can inadvertently produce biased, offensive, or otherwise inappropriate content. They might, for example, exhibit gender or cultural biases in their responses or use toxic language. These ethical issues can harm users or a company’s reputation. Monitoring for bias and toxicity is crucial – for instance, using content filters or evaluators to flag outputs that contain hate speech or reflect unfair stereotypes. Observability tooling often includes safety checks or bias detection metrics to ensure the AI’s behavior aligns with ethical and inclusive standards.
Security & privacy risks: In addition to prompt injection, LLM applications face other security concerns. The model might inadvertently leak sensitive data that was in its training set or provided context. Users might exploit the system to extract hidden prompts or confidential info. Also, if the application integrates with databases or external services, those components must be monitored for failures or breaches. Observability helps by logging all interactions and alerting on anomalies (e.g., a sudden spike in data volume or unusual access patterns). This ensures that any potential data leakage or unauthorized access can be quickly detected and addressed.

LLM observability vs. LLM monitoring



LLM Monitoring focuses on the what. It continuously tracks performance metrics of the LLM application in real time to ensure the system is behaving as expected. This includes metrics like response latency, request throughput, API error rates, and usage counts (e.g. number of prompts or tokens processed). Monitoring can also involve simple checks on inputs and outputs – for instance, confirming the model is responding and maybe counting how often certain errors or keywords occur. The goal of monitoring is to detect operational issues quickly (like the service going down or responses slowing beyond a threshold) and trigger alerts or dashboards for engineers. It answers questions like, “Is the LLM service up and meeting our performance SLAs?”.
LLM Observability goes beyond that to focus on the why. An observability solution provides full visibility into all the moving parts of the LLM system. It not only gathers the metrics that monitoring would, but also correlates them with logs, traces, and rich data about each request to enable debugging and in-depth analysis. With observability, engineers can reconstruct the path of a specific query through the system – see the prompt that was given, the steps the application took (e.g. calls to a vector database or multiple model invocations), and the output generated. This means when an issue is detected (say, a sudden spike in latency or a bizarre answer from the model), observability tools let you dig into the details and find the root cause: Was it a particular prompt that caused the model to loop? Was an external API call slow? Did a certain user input trigger a hidden bug? Tracing is a key component here: observability often involves capturing traces or timelines of each request across components, which is invaluable for diagnosing complex issues that involve multiple services or asynchronous steps. In short, monitoring might tell you something’s wrong, but observability helps you understand why it’s wrong and how to fix it.

Challenges in observing LLM systems

Non-deterministic outputs: Unlike a classical algorithm that will produce the same output for a given input, LLMs are stochastic. The same prompt can yield different completions from one run to another. Moreover, LLM behavior can drift or change over time – for example, if you use a third-party API, the provider might update the model or training data, subtly altering its responses. This non-determinism means you can’t rely on fixed expected outputs or unit tests in the usual way. Small prompt wording changes or differences in input context can produce significantly different results. Traditional testing (comparing to a single “ground truth” answer) fails because there may be many acceptable answers, or the model may be unpredictably wrong in new ways. As Confident AI’s guide notes, “real users will always surprise you with something unexpected… LLM observability enables you to automatically detect these unpredictable queries and address any issues they may cause.” Continuous observability is needed to statistically detect when the model’s outputs start straying outside acceptable parameters, since you can’t pre-define all correct outcomes.
Complex, multi-component pipelines: LLM applications often involve more than just a single model call. There may be retrieval components (e.g. searching a vector database for relevant context), various transformers or prompt templates, external API calls, and post-processing steps. Additionally, modern LLM systems might be composed of chained LLM calls or agents: for example, one LLM call’s output feeds into another, or an agent that iteratively decides to use tools or make more queries. This complexity makes it difficult to pinpoint where something went wrong. An error or slowdown could originate in the database lookup, the prompt formatting logic, the model inference step, or even in a downstream parsing of the model’s output. Debugging across these distributed components is a major challenge. Observability needs to tie together events from all parts of the pipeline. For instance, when a user question comes in, you want to trace it through the retrieval step (and see what documents were fetched from the vector DB), then through the LLM call (and capture the prompt and model response), and through any subsequent logic. Without this, you’re “left in the dark” about why a request failed or returned a bad answer. One common scenario: Did our knowledge base fail to return relevant data, or did the LLM mis-handle perfectly good data? Observability data (like logged retrieval results and model outputs) lets you do root cause analysis on such questions. In essence, an LLM observability solution must embrace distributed tracing and logging so that developers can see end-to-end flow across complex LLM pipelines.
Black-box model reasoning: Even if we capture the flow of operations, the core LLM itself is often a black box in terms of how it arrives at a particular output. We don’t have explicit internal state or explainability from the model’s millions of neurons. This opacity can be frustrating: for example, why did the model output a biased statement? Why did it latch onto an irrelevant part of the prompt? Traditional ML observability might use techniques like feature importance or SHAP values for simpler models, but those don’t translate well to the giant opaque embeddings of an LLM. Explainability in LLMs is still an open research area, but observability tools try to provide proxies for it – such as showing the intermediate steps in a chain of thought, visualizing attention weights or embeddings, or correlating output changes with prompt changes. The challenge is to surface interpretable insights from the model’s behavior. While we can’t fully open the black box, observability can at least log things like which knowledge source was used, or provide semantic traces of how the model’s output evolves with each prompt refinement. For instance, techniques like comparing the embeddings of the model’s answer to known references can hint if the model was referencing the provided context or just relying on its internal memory. The non-deterministic and opaque reasoning of LLMs means observability has to capture rich context (prompts, embeddings, etc.) to give any hope of diagnosing issues like “the model often confuses similar concepts” or “the model is veering off-topic for certain inputs.”
Infinite input possibilities: Users can ask literally anything from an LLM. This open-ended input space means you cannot anticipate every possible query or scenario in advance. There will always be edge cases and long-tail queries that reveal new failure modes. For example, a user might combine instructions or ask something in a format you never tested, causing the model to respond oddly. This is a challenge for observability because it must be proactive and adaptive. Rather than only focusing on known metrics or predefined test cases, LLM observability should help automatically surface when something unusual or problematic occurs in new inputs. One approach is to perform real-time evaluations on a sample of live traffic – essentially treating production as an ongoing test. If the system detects an anomaly (say, a sudden increase in user feedback thumbs-down on answers, or a spike in a “hallucination score” for outputs), that query can be flagged for analysis. Without observability, these rare but important cases might slip by until a user reports an issue. As one guide put it, you can test thoroughly before deployment, “but real users will always surprise you… the range of possible queries is endless, making it impossible to cover everything with complete confidence”. Thus, observability acts as a net to catch the unknown unknowns in production.
Performance and scaling trade-offs: Another challenge is that LLMs are resource-intensive. Monitoring performance metrics isn’t just about keeping users happy; it’s about controlling resource usage and costs. An observability solution for LLMs needs to account for metrics like GPU/CPU utilization, memory usage, and token usage per request. It should help answer questions like: Can our infrastructure handle more concurrent users? or Which part of the pipeline is the slowest and needs optimization? For example, maybe the vector database lookup is adding 200ms overhead – is that the bottleneck or is it the model inference on large prompts? Observability should visualize such breakdowns. Scalability is a challenge: as the application scales, the observability system itself must handle lots of telemetry data without causing too much overhead or latency. Furthermore, the cost dimension means we want to observe not just technical metrics but also business metrics like cost per query. Non-deterministic token usage can make cost hard to predict, so continuous observability of token counts and their dollar conversion helps in optimizing deployments.

Key features to look for in LLM observability tools

End-to-end tracing of LLM chains: Modern LLM applications often consist of LLM chains or agent sequences, where one step’s output feeds into the next. It’s essential that your observability tool can trace and debug these chains step by step. This means capturing each sub-call or operation in the workflow – for example, the retrieval from a vector database, the assembly of the final prompt, the call to the LLM API, and any post-processing. By visualizing the complete request flow (as a trace with spans), you can quickly identify where things might be going wrong. For instance, if a chain is slower than expected, the trace will show whether the delay is in the LLM call or perhaps in an upstream data fetch. If an LLM agent gets stuck in a loop, tracing reveals which step is repeating. LLM chain debugging capability is thus a must-have: it turns the complex pipeline into a transparent sequence that engineers can inspect and optimize.
Visibility into the full application stack: An LLM application doesn’t run in isolation – it sits on a stack that could include model servers (or API clients), databases (vector stores, caches), web services, and more. When a user sees a poor outcome, the root cause could be anywhere in this stack. Therefore, your observability solution should cover all layers of the LLM application, not just the model calls. This means correlating events from the infrastructure level (CPU/GPU usage, memory, network latency), the data layer (e.g. how long a database query took, did it return results), and the application logic (errors in code, slow functions in the chain). Full-stack visibility allows you to pinpoint, for example, that a response was slow not due to the model, but because a vector database was overloaded, or that outputs became inaccurate after a certain deployment of a prompt template or model version. Essentially, the observability tool should be able to tell you which component or layer is at fault when something breaks. This often involves integrating with existing APM (Application Performance Monitoring) data or cloud monitoring for components like databases, and linking those metrics to the LLM request traces.
Explainability and anomaly detection: Since LLMs are black boxes, your observability platform should provide features to explain and evaluate model decisions as much as possible, and automatically flag unusual behavior. Explainability can come in forms such as: logging the input prompts and output responses (so you can inspect what the model saw and produced), visualizing intermediate representations (like embeddings or the retrieved context documents used for a query), or even showing attention weights or token probabilities if the model provides them. These help in understanding why the model responded a certain way or why it might have erred. On top of that, anomaly detection is crucial. The tool should be able to monitor the streams of inputs and outputs and detect outliers – for example, a sudden surge in negative user feedback, or the model starting to output a string of very long responses (which might indicate it’s rambling or looping), or an unusual drop in some quality score. Advanced observability solutions include out-of-the-box detectors for things like bias or toxicity in outputs, or significant deviations from normal usage patterns. For instance, if the model usually answers within 2 sentences and suddenly gives a 10-paragraph response, that could be flagged as an anomaly. By catching such anomalies automatically, the system can alert engineers to investigate potential issues (like a prompt change that caused the model to go off-track or a user attempting a prompt injection). In summary, look for tools that not only collect data, but also intelligently analyze it to surface quality issues, biases, or errors in real time.
Integration and security features: An effective LLM observability solution must blend into your existing workflows and tech stack. This includes being able to integrate with the LLM providers or frameworks you use, as well as with your data stores and messaging systems. For example, if you use OpenAI’s API and a local vector database, the observability tool should have connectors or SDKs to instrument calls to both. Many tools support integration with popular libraries (like LangChain, Hugging Face, etc.) or open standards like OpenTelemetry for tracing. Scalability is also part of integration – the tool should handle your volume of data and allow you to aggregate or sample intelligently as needed. On the security side, because LLM applications often handle sensitive data, the observability platform should offer robust security and privacy capabilities. This includes PII redaction (automatically masking personally identifiable information in prompts or outputs that get logged) and protection against logging sensitive secrets inadvertently. It should also be secure in itself – with proper access controls, encryption, and compliance if needed (especially in enterprise settings). Some observability solutions even provide features to guard against prompt hacking by scanning inputs and outputs for known attack patterns. In essence, you want a tool that is enterprise-ready: it integrates smoothly with your environment (APIs, databases, devops tools) and maintains high security standards so that monitoring your model doesn’t become a vulnerability.
Evaluation and feedback loops: While the above features cover monitoring and debugging, a strong LLM observability solution also supports continuous evaluation of model quality. This means having mechanisms to measure the correctness or relevance of outputs on an ongoing basis, not just when things go wrong. Some tools incorporate an evaluation framework where you can define metrics or tests (for example, using reference datasets or regex checks for certain answer formats) and have the tool automatically grade each response or periodically run evaluation jobs. Automated LLM evaluations could include checking factuality (perhaps by cross-verifying answers against a knowledge base), computing embedding similarities to expected answers, or applying NLP metrics like BLEU/ROUGE for summarization tasks. In addition, human-in-the-loop feedback capabilities are valuable: the tool might allow easy capture of user ratings (thumbs up/down) or reviewer comments on responses. Integrating these feedback signals into the observability dashboard lets you correlate objective metrics with actual user satisfaction. Over time, having this evaluation layer helps answer the question “Are our LLM outputs getting better or worse?” and identifies specific areas for improvement (e.g., maybe the model is good at general questions but consistently fails on a certain category of queries – continuous eval will catch that). While not every observability tool has built-in eval, the best ones do or at least integrate with external evaluation pipelines. The ability to replay scenarios is another related feature: you should be able to take a recorded prompt/response and try it against a new model version or prompt template to compare outcomes, facilitating A/B testing and prompt optimization in a safe way.
User interface and filtering: Finally, an often underrated feature is a good UI/UX for analysis. Observability data is only as useful as your ability to derive insights from it. Look for solutions that provide interactive dashboards, search and filtering capabilities, and perhaps custom visualization builders. For example, if you have thousands of logged LLM interactions, you want to easily filter down to those that were flagged as failures or those from a specific user or those using a particular model version. Advanced filtering can allow slicing the data by metadata like conversation ID, prompt template version, or any custom tags you log. This way, if a problem is reported (“user X got a bad answer in session Y”), you can zero in on that session’s trace quickly. Or you can filter for all responses where your automated evaluation gave a low score, to see common patterns among those failures. A flexible querying interface (and the ability to export data for offline analysis) is highly useful for ML engineers to perform error analysis. Some observability platforms also let you define alerts on certain conditions – for instance, trigger an alert if hallucination rate goes above 5% or if average latency exceeds 2 seconds. This ties back into monitoring, but configured with the richer LLM-specific context. In summary, the observability solution should empower you to explore the data easily and get actionable insights, rather than drowning in logs.

Overview of the LLM observability tooling landscape

Open-source APM/tracing solutions with LLM integrations: Some open-source Application Performance Monitoring tools (for example, those built on OpenTelemetry) have begun adding support for LLM workloads. They excel at capturing traces, metrics, and logs in a unified platform. By instrumenting LLM calls (often via language-specific SDKs or middleware), they allow you to leverage standard distributed tracing for LLM chains. These solutions often require a bit of setup and expertise but can be very powerful and cost-effective, providing end-to-end visibility if you’re willing to integrate them. Common capabilities include custom dashboards for LLM metrics (like total LLM calls, token throughput, latency per model), and possibly pre-built instrumentation for popular LLM frameworks.
Cloud monitoring & AIOps platforms: Many cloud-based monitoring services now advertise LLM monitoring features as part of their offerings. These are typically extensions to their existing logging/metrics systems to handle prompt/response data and LLM-specific alerts. They might provide easy integration if you’re already using the platform (e.g., an agent that hooks into your application to automatically capture LLM requests). Typical features are real-time metrics, anomaly detection (leveraging AI to spot unusual model outputs or usage patterns), and out-of-the-box alerts for things like “hallucination detected” or “prompt hijack attempt”. Some of these platforms can be quite robust and scalable, though they may come at enterprise pricing. They emphasize reliability, security, and integration with broader observability (so you can see your LLM stats alongside your microservice metrics, for instance).
Specialized LLM observability & evaluation tools: A number of newer tools are purpose-built for ML/LLM observability and focus heavily on evaluation of model outputs and data quality. These tools often provide a suite of LLM-specific features such as: prompt versioning and experimentation, automated evaluation metrics (e.g., comparing model output to ground truth or checking for biases), dataset management for prompts and responses, and even model “comparison reports” when you try a new model or prompt variation. They may incorporate guardrails (business rules or AI-based filters to catch unwanted outputs) as part of the observability loop, effectively merging monitoring with active policy enforcement. The strength of specialized tools is that they deeply understand LLM use cases – for example, they might allow you to A/B test two models in production and observe which yields better responses, or trace not just the technical calls but the semantic changes in output quality. They often offer fine-grained filtering and search over your logged interactions, which is invaluable for analysis. The trade-off is that these might require you to adopt their framework for logging/evaluation, and there’s a learning curve to using the full feature set (especially if they introduce their own concepts for scoring or feedback).
Experiment tracking & LLMOps platforms: There are also tools that originate from ML experiment tracking (for training models) which have extended into the LLMOps and observability space. These are tools data scientists might already use to track model training runs, and now they can be used to log prompt-response pairs, track model versions, and visualize metrics for LLMs. They are great during development and can also be adapted for production monitoring. Typically, they offer strong support for versioning (of models, prompts, datasets) and integrating with Jupyter notebooks or scripts. They might not have all the real-time alerting capabilities of a dedicated monitoring tool, but they excel at providing a workspace to analyze model behavior and share reports. For example, after deploying an update, you could log a bunch of interactions and then use the tool’s UI to compare the new model’s responses to the old one side-by-side, using built-in metrics. These platforms often have some support for visualizing embeddings, computing similarity metrics, or hooking in custom evaluation code. Many of them are extensible, so you can plug in your own checks (for fairness, toxicity, etc.) and log those as additional metrics.

Tutorial: Tracking LLM outputs and anomalies with W&B Weave

Setup and instrumentation: Installing W&B Weave and annotating our LLM call function to enable tracing.
Logging LLM calls: Running the instrumented function to generate some example outputs and sending those logs to Weave.
Anomaly detection with guardrails: Using Weave’s built-in Guardrails scorers to automatically evaluate outputs for issues like hallucinations or toxicity.
Visualizing traces and metrics: Viewing the collected traces and metrics in the Weave UI to identify any anomalies.

1. Install and initialize W&B Weave

pip install weave wandb
wandb login  # to authenticate with your W&B API key

import weave
import asyncio
from typing import Dict, Any
# Initialize a Weave project (logs will go here)
weave.init("llm-observability-demo")

2. Instrument the LLM call with @weave.op

@weave.op()  # This decorator tells Weave to track this function's inputs/outputs
def answer_question(question: str) -> str:
    """Call the LLM to get an answer to the question."""
    # Simulate a response (for demo, we hard-code a couple of cases)
    if "capital of France" in question:
        return "The capital of France is Paris."
    elif "president of France" in question:
        # Let's simulate a hallucination here:
        return "The president of France is Napoleon Bonaparte."  # (Incorrect - hallucinated)
    else:
        # A default generic response
        return "I'm sorry, I don't have the information on that."

questions = [
    "What is the capital of France?",
    "Who is the president of France?",
    "How many legs does a spider have?"
]
for q in questions:
    answer = answer_question(q)  # calling the op (this gets logged by Weave)
    print(f"Q: {q}\nA: {answer}\n")

The answer_question function runs for each question.
Because of @weave.op, Weave intercepts each call. It records the function name, the input (q string), and the output (answer string), as well as timing information.
Weave streams this data to the W&B platform under the project "llm-observability-demo". After each call, Weave typically prints a link in your console to view the trace in the browser.



3. Adding guardrails for output evaluation

Guardrail mode: active prevention (e.g., refuse or modify an output if it’s flagged).
Monitor mode: passive evaluation (just log a metric, but don’t change the output).

# Create a custom scorer for hallucination detection
class HallucinationScorer(weave.Scorer):
    @weave.op
    def score(self, output: str) -> dict:
        """A simple hallucination detector that flags certain keywords."""
        # A dummy check: flag as hallucination if certain keyword is in output.
        # In practice, this could call a model or check against a knowledge base.
        hallucinated = "Napoleon" in output  # just for demo criteria
        return {
            "hallucination": 1 if hallucinated else 0,
            "confidence": 0.9 if hallucinated else 0.1
        }

# Create a dataset of our questions for evaluation
dataset = [
    {"question": "What is the capital of France?"},
    {"question": "Who is the president of France?"},
    {"question": "How many legs does a spider have?"}
]

# Create an evaluation
evaluation = weave.Evaluation(
    dataset=dataset,
    scorers=[HallucinationScorer()]
)

# Run the evaluation
results = evaluation.evaluate(answer_question)
print("Evaluation complete! Check the Weave UI for detailed results.")

We define a HallucinationScorer class with a score method. This method inspects the output string and returns a dictionary with a metric (here "hallucination": 1 or 0). In a real use case, you might implement a more sophisticated check, but for illustration we just flag if the output contains “Napoleon”.
We use answer_question.call(...) instead of a direct call. The .call() method provided by Weave executes the function and also gives us a Call object representing that execution. We need this object so that when we log the scorer’s result, it attaches to the correct function call record in Weave’s database.
Then we call the scorer’s score via .call() as well, passing in the output we got and the _call object of the original function call. This ensures the score (hallucination=1 or 0) is linked with that specific run.

# Additional scorer for answer relevancy
class RelevancyScorer(weave.Scorer):
    @weave.op
    def score(self, question: str, output: str) -> dict:
        """Check if the answer is relevant to the question."""
        if "don't have the information" in output:
            return {"relevancy": 0.2}  # Low relevancy for non-answers
        elif any(word in output.lower() for word in question.lower().split()):
            return {"relevancy": 0.8}  # High relevancy if answer contains question words
        else:
            return {"relevancy": 0.5}  # Medium relevancy otherwise

# Update our evaluation with multiple scorers
evaluation = weave.Evaluation(
    dataset=dataset,
    scorers=[HallucinationScorer(), RelevancyScorer()]
)

# Run the evaluation with multiple metrics
results = evaluation.evaluate(answer_question)



4. Visualizing traces and metrics in Weave

Traces Tab: Shows all your function calls with inputs, outputs, and timing information
Evaluation Results: When you run evaluations, you'll see aggregated results and can drill down into individual scored calls
Call Details: Click on any call to see full details including the evaluation scores attached to that specific call

The function that was called (answer_question)
The input (the question)
The output (the answer)
Timing information
Any evaluation scores that were computed

hallucination: 1 (flagged)
confidence: 0.9
relevancy: 0.8 (if using the relevancy scorer)

hallucination: 0 (not flagged)
confidence: 0.1

Filtering and Searching: Filter traces by evaluation scores, time ranges, or specific inputs
Comparison Views: Compare the performance of different models or prompt variations
Aggregated Metrics: See summary statistics across all your evaluations
Export Capabilities: Export your trace data for further analysis

Summary

Instrumentation: We decorated our LLM function with @weave.op() for automatic tracing
Logging: Every function call is automatically logged with inputs, outputs, and metadata
Evaluation: We created custom scorers to detect hallucinations and assess relevancy
Analysis: All data flows to the Weave UI where you can analyze patterns, set up alerts, and track model quality over time

Weave makes it easy to add observability to LLM applications with minimal code changes
The evaluation framework allows you to systematically assess output quality
All data is automatically linked and visualizable in the web UI
You can scale this approach to production systems with real LLM APIs

LLM observability metrics and evaluation

Quality and behavior metrics – how good are the model’s outputs? Are they relevant, correct, fair, etc.? These help monitor the content the LLM generates.
System and performance metrics – how well is the system running? How fast, how resource-intensive, how reliable? These ensure the operational health of the application.

Quality evaluation metrics for LLM outputs

Relevance: This measures how well the model’s response addresses the user’s query or the task at hand. A relevant answer is on-topic and useful to the user’s request. In tasks like question-answering, relevance overlaps with correctness (a relevant answer is usually a correct one to the question). In dialogue, relevance means the response stays on topic. We can measure relevance via information retrieval metrics (does the answer contain the information found in reference texts for the query?) or via semantic similarity (comparing the embedding of the answer to that of an expected answer). Human evaluation is often used too (raters judge if an answer is relevant). Maintaining high relevance is important because an irrelevant or off-topic answer indicates the model misunderstood the query or veered off – something observability should catch. For example, if users ask “What is the weather tomorrow in Paris?” and the model gives a generic definition of Paris, that’s low relevance. Many observability setups include a relevancy scorer to automatically compare the question and answer embeddings – a low score might flag a potential issue.
Perplexity: Perplexity is a classic metric from language modeling that measures how well a model predicts a sample of text. Technically, it is the exponentiated negative log-likelihood of the test data according to the model. In simple terms, perplexity gauges the model’s “surprise” or uncertainty when generating text – lower perplexity means the model finds the sequence more predictable (and thus it fits the model’s training distribution better). For LLM observability, perplexity is more often used as a model-level evaluation rather than per single output. It’s useful when assessing model versions or changes: if you have a holdout dataset, you can track perplexity to see if the model’s overall language understanding is improving. A lower perplexity on a relevant dataset generally correlates with a more fluent and potentially more accurate model. However, perplexity alone doesn’t guarantee task success – a model can have low perplexity but still hallucinate on factual questions, since perplexity doesn’t directly measure truthfulness. In monitoring, perplexity can be used to detect if a model starts performing oddly on the data distribution. For example, if suddenly your model’s perplexity on recent queries goes up, it might indicate distribution shift in inputs or that the model is struggling to generate likely text, often correlating with confusion or errors in output. Some advanced observability tools might calculate a form of prompt perplexity or use model-internal likelihoods to detect uncertain answers. In summary, perplexity contributes to assessing the model’s general confidence and can reveal drift or degradation in the model’s language performance.
Fairness and bias metrics: Fairness in LLM outputs means the model’s responses are free from inappropriate bias or discrimination toward particular groups. Measuring this can be complex, as bias is context-dependent. Common approaches include bias benchmarks (sets of queries designed to elicit responses about different demographic groups to see if the model treats them differently) and statistical parity metrics (e.g., comparing sentiment of model outputs when the prompt only varies the gender or ethnicity in a scenario). For LLM observability, one might use metrics like the toxicity score or bias score provided by external classifiers. For instance, Jigsaw’s Perspective API can assign a toxicity probability to text; if your LLM outputs cross a certain toxicity threshold, you log that as an event. Similarly, one could define a simple fairness metric: feed the model a template like “Two surgeons are talking. One is a man and one is a woman. Who is better at surgery?” – if the model picks one gender consistently, that’s a bias sign. In live monitoring, direct bias evaluation might involve detecting protected attribute mentions in outputs and ensuring they’re not used in a derogatory or stereotypical manner. Some tools have bias detectors that look for certain phrases or unfair language. The contribution of fairness metrics is to ensure the model maintains ethical standards and a positive user experience. If an observability dashboard shows, for example, that responses containing ethnic terms have a higher toxicity score on average, that’s a red flag to address. Fairness metrics impact trust: by monitoring them, organizations can catch potentially harmful outputs before they escalate into public incidents. As the saying goes, “If you’re not measuring it, you’re missing it” – so integrating fairness checks into LLM observability is becoming a best practice for responsible AI deployment.

System performance and resource metrics for LLMs

Latency: Latency is the time it takes for the system to return a response to the user after receiving a request. In the context of LLMs, we often break this down further: for example, the time spent in the model inference vs. time spent in retrieval or other preprocessing. High latency can severely degrade user experience, especially in interactive applications where users expect quick answers. It’s important to monitor not just average latency but also tail latency (95th percentile, etc.), as occasional very slow responses can be just as problematic. Observability should link latency spikes to their cause – e.g., was a slow response due to a long prompt that took the model longer, or due to an external call (like a database search) hanging? By tracking latency per component, you can find bottlenecks. For instance, you might discover that prompt serialization is adding 100ms you could trim, or that enabling streaming could reduce perceived latency. Many teams set latency budgets (like, 1 second overall, with 0.7 sec for the LLM and 0.3 for all else) – observability data helps verify these are met and triggers alerts if breached.
Throughput: Throughput measures how many requests (or tokens) the system can handle per unit time – essentially the capacity of the system. For LLM services, one might measure requests per second or, sometimes, tokens generated per second as a throughput measure. It’s particularly relevant for scaling: you want to know if your system can handle, say, 100 concurrent users or if it saturates at 20. Sometimes a model has great single-query latency but throughput suffers when parallel requests are high. Monitoring throughput alongside resource usage is key to planning scaling and load balancing. For example, if throughput plateaus and latency grows when you go beyond X qps (queries per second), that indicates you’ve hit a CPU/GPU or network bottleneck. Throughput impacts cost and user satisfaction – if your throughput is too low, requests will queue and users experience delays or timeouts. By observing throughput and maybe doing stress tests (and seeing those metrics in your observability dashboard), you can determine the safe limits of your system and set up autoscaling triggers appropriately. Throughput metrics also help in understanding efficiency: e.g., after an optimization, did your system handle more requests per second for the same hardware usage?
Token usage & cost: In LLM applications, tracking token counts is extremely informative. Each request can be characterized by how many input tokens it had and how many output tokens were generated. This affects both latency and cost. If you see an unexpected increase in output token length over time, it might mean the model is rambling more or not following instructions to be concise. Monitoring token usage per request can thus signal changes in model behavior (maybe after a model update, answers got 20% longer on average – which might be good or bad). From a cost perspective, if you’re using a paid API, cost = (input tokens + output tokens) * price per token. Observability should include dashboards for total tokens used per hour/day and cost estimates, so you can catch if a particular user or feature is causing a spike. For example, maybe a poorly designed prompt caused the model to spew a 10,000-token response somewhere – you’d want to know that to avoid a surprise bill. Cost metrics tie in with business KPIs: you can watch cost per user request and work on prompt optimization to reduce it if needed. Additionally, token usage monitoring helps with rate limiting and quota management: if you have an internal policy to not exceed N tokens per day, you need to observe the token consumption trend.
Resource utilization (CPU/GPU, memory): LLMs can be heavy on compute. If you are hosting the model yourself (on GPU or CPU), you need to monitor how utilized these resources are. High GPU utilization might mean you’re efficiently using hardware, but if it’s constantly at 100% and latency is suffering, you’re overloaded. Memory (RAM or GPU VRAM) is critical too – if memory usage is near capacity, you risk OOM errors. Observability should track these metrics ideally correlated with the LLM usage. For instance, it’s useful to see a timeline of GPU utilization alongside the number of requests in that interval, to identify saturation points. If you notice that memory usage grows over time (possible memory leak or accumulating cache), that’s something to fix. Disk I/O or network I/O could also be relevant if your LLM loads a lot of data or calls external services. The impact of resource metrics is mostly on system stability and scalability: they help ensure you’re not overcommitting your hardware, and they guide scaling decisions (e.g., add another GPU instance when CPU usage exceeds 70% consistently). They also help in performance tuning – e.g., if GPU isn’t fully utilized and CPU is idle, maybe you can increase batch size or concurrency to improve throughput. Conversely, if GPU is maxed out and CPU is low, the bottleneck is clearly the model computation.
Error rates and failure modes: In any application, it’s important to monitor errors. For LLM apps, errors might include things like model service errors (e.g., an API call to the LLM provider fails or times out), invalid inputs (the user prompt couldn’t be handled), or downstream errors (e.g., unable to retrieve context, or your own code exceptions while formatting outputs). Observability should capture when a request does not complete normally. For example, if 2% of requests are returning errors (perhaps due to hitting a token limit or an external API outage), you need to see that trend and ideally get alerted. Content filtering can also be considered here: if the LLM refuses to answer certain queries (triggering its built-in safety filter), those could be logged as “no-answer” or filtered responses – essentially a kind of error from user perspective. Monitoring the frequency and triggers of those is useful (e.g., “we’re seeing many refusals for medical questions, maybe our prompt needs adjusting to allow them”). Error rate directly affects reliability – a spike in errors means users are not getting answers at all. It can also hint at bugs or external issues (if vector DB queries fail 5% of the time, maybe that service is flaky under load). Observability tools often include alerting on error rate thresholds because that’s a basic indicator of system health.

Conclusion