Accelerating enterprise ROI from AI agents with the NVIDIA Enterprise AI Factory + Weights & Biases
Digging into how to build agents with the new NVIDIA Enterprise AI Factory
Created on May 14|Last edited on May 19
Comment
The concept of AI agents—software that autonomously perceives its environment, makes decisions, and takes actions to achieve specific goals using artificial intelligence—can transform the way enterprises operate, both internally and externally interacting with their customers. That being said, due to the non-deterministic nature of generative AI, achieving and maintaining high quality, while safeguarding and monitoring the applications is a major challenge that can prevent enterprises of all sizes from confidently deploying AI agents.
In this blog, we will discuss how the combination of the Weights & Biases platform and the NVIDIA Enterprise AI Factory validated design, which includes NVIDIA NIM and NVIDIA NeMo microservices,provides the building blocks and tools needed to build, iterate, monitor and safeguard AI agents, and how to continuously improve AI agents once they are in production.
What is the NVIDIA Enterprise AI Factory?
Announced at Computex 2025, the NVIDIA Enterprise AI Factory is a full-stack validated design that offers guidance for developing, deploying, and managing agentic AI, physical AI, and HPC workloads on theNVIDIA Blackwell platform in on-premises data centers. Designed for enterprise IT, the design integrates NVIDIA Blackwell accelerated computing, NVIDIA networking, NVIDIA AI Enterprise software and storage to help deliver faster time-to-value AI for enterprise factory deployments while mitigating deployment risks.
The AI flywheel
AI flywheels are central to driving intelligent AI reasoning agents. There are two main components in the AI Flywheel, the inference workflow and the training workflow. The inference workflow refers to leveraging AI—building, interacting, and monitoring an agent and capturing feedback of the agent, both from users and from a systems perspective. The training workflow is focused on the AI model including creating the training and evaluation dataset, training & fine-tuning the model, and analyzing the training run. To bring the whole loop together, a critical step of the flywheel is to capture all of the inputs, outputs, and cataloging the AI models leveraged in a central repository—especially critical for AI agents leveraged in regulated industries and customer facing use cases.
Training Workflow
The foundation of any AI agent is the GenAI model powering the agent. For many organizations with industry or enterprise specific data & use cases, an off the shelf model will not provide the required performance to power the AI agent. These organizations oftentimes will train their own GenAI model, or more commonly, fine-tune an open source model such as Llama or Mistral.
After deciding which model to fine-tune, the first step is to create the training dataset(s) that will be used to fine-tune the model. For this, NVIDIA NeMo Curator provides the building blocks to create a high-quality dataset. It streamlines data cleaning, deduplication, filtering, and formatting processes, ensuring that the training data is optimized for performance and aligned for specific use cases. NeMo Curator is especially useful in enterprise settings where large-scale, domain-specific data needs to be processed securely and reliably for AI model development. Once the training dataset has been created, it can be logged with W&B Artifacts. W&B Artifacts will keep a version history of the dataset and also track which training runs leverage the dataset, critical for reproducibility and debugging purposes.
Once the dataset has been created and the model has been selected, it’s time to fine-tune the model. NVIDIA NeMo Customizer enables enterprises to fine-tune large language models (LLMs) on their own proprietary data with minimal infrastructure overhead. It supports supervised fine-tuning (SFT) and parameter-efficient tuning methods like LoRA, allowing organizations to personalize foundation models for domain-specific tasks.
NVIDIA NeMo Customizer ensures efficient, scalable, and secure customization workflows across on-prem, cloud, or hybrid environments. NeMo Customizer integrates directly with Weights & Biases’ Experiment Tracking, logging all hyperparameters, training metrics, system metrics (e.g., GPU usage, memory), model artifacts (e.g., checkpoints, trained weights, logs), code versioning and environment details.
Now that the model has been fine-tuned, let’s see how it performs. For this, we use NVIDIA NeMo Evaluator, a microservice that assesses the performance and quality of large language models using standardized and custom evaluation benchmarks. It enables developers to measure accuracy, relevance, and robustness of model outputs across tasks like summarization, classification, and question answering, helping guide fine-tuning and deployment decisions. The evaluations can be analyzed within Weights & Biases Weave, and can be compared to evaluations of different models and different versions of a specific model.
Finally, once the model has been evaluated and approved, this model can be logged into the Weights & Biases Registry, which will capture the history of the model development, what stage the model is leveraged (prototype, production, etc.). This Registry can be shared across the enterprise and organizations can create a model leaderboard as they leverage and create new models.
We’re ready to move onto the Inference Workflow!
Inference Workflow
Now that the LLM is ready to go, we’re ready to build the AI agent. The first step is to deploy the LLM onto the inference engine. NVIDIA NIM microservices are prebuilt, optimized containers that deliver enterprise-ready generative AI models via standard APIs for fast and scalable deployment. Designed to run efficiently on NVIDIA accelerated computing, NIM simplifies the integration of AI into applications by abstracting complex model serving infrastructure. For use cases which will require the LLM to gather information/context from enterprise specific data, often the RAG technique is leveraged. NVIDIA NeMo Retriever is a collection of NIM microservices that enable fast and accurate retrieval of relevant documents or data chunks for RAG workflows. It includes NIM-powered models for embedding and reranking to enhance contextual understanding and response quality by grounding outputs in enterprise-specific knowledge.
While building out the AI agent, built on the NIM-deployed LLM, enterprises will often go through a bit of experimentation with prompts, analyzing and debugging the traces, and understanding the AI agent flow to ensure the agent is performing as expected. W&B Weave is purpose built for this workflow - systematically capturing all inputs, outputs, and relevant metadata and displaying the information in a developer friendly UI. AI agents break tasks up into sequential steps, and the W&B Weave Trace Tree allows developers to visualize these complex rollouts to speed up iteration. A deeper dive into W&B Weave for AI agents can be found here.
One of the most critical steps for improving the performance of an AI agent is leveraging feedback of the end users. W&B Weave provides feedback gathering capabilities such as human-annotation templates, which allows the end user of the application to provide live feedback of the AI agent. This information will then be summarized for the developer in the W&B Weave interface, providing the required information to target where to best improve the performance of the agent.
Once the application has been deployed, production trace monitoring becomes a critical workflow. Monitoring production traces for generative AI applications is essential to ensure model outputs remain accurate, safe, and aligned with user expectations over time. These systems often operate in dynamic environments with evolving data and usage patterns, making it critical to detect issues such as model drift, hallucinations, or performance degradation early.
With Weave, teams can continue to trace every step of an AI pipeline—from input prompts and model versions to intermediate computations and final outputs—within an interactive, end-to-end visualization. This enables developers and stakeholders to quickly diagnose anomalies, compare behaviors across deployments, and continuously improve model performance in real-world settings.
Finally, there may come a time where the performance of the AI agent will require re-tuning the underlying LLM. To close out the flywheel, or perhaps restart the flywheel, we will require a new training dataset. This dataset should be based on all of the feedback and interactions from end users interacting with the AI agent. Users can create a dataset directly within Weave by selecting calls from Weave Traces and adding them to a new dataset. They can then select the specific traces that they want to retrain the model on, simplifying the task of collecting and formatting. Details on leveraging this feature can be found here. We’re now ready to retrain or fine-tune the LLM powering the AI agent, restarting the flywheel.
Add a comment
Iterate on AI agents and models faster. Try Weights & Biases today.