Tutorial: The OpenPipe ART project

Key limitations of existing reinforcement learning frameworks

Inadequate support for multi-turn interactions: Many reinforcement learning frameworks were originally built for environments like games or simulations with single-step actions. They often struggle to handle multi-turn rollouts or long-horizon dialogue tasks that LLM-based agents engage in. For example, a conversation agent might need to remember context over many turns. Traditional frameworks don’t easily accommodate this type of sequential decision process with dynamic context. ART fills this gap by natively supporting multi-step trajectories (sequences of messages between user and assistant) and treating the entire interaction as one trajectory to learn from.
Integration challenges with existing codebases: Using reinforcement learning in a project usually requires significant refactoring or using a specialized environment (like OpenAI Gym). Existing libraries might demand that you wrap your task in a custom Env class or adhere to a training loop that is separate from your application logic. This is cumbersome for developers who have an existing agent codebase or workflow. OpenPipe ART avoids this by providing a lightweight client that you can plug into your existing code. You can run your agent as-is – making API calls for model decisions – and ART will seamlessly handle the background training. This means minimal code changes are needed to start training an agent within your own application or simulation.
Inefficient GPU utilization and training workflow: Another limitation is that typical reinforcement learning training loops can be inefficient with hardware resources. For instance, you might generate experiences (agent rollouts) on CPU and then periodically move data to a GPU for training, leaving the GPU idle much of the time. Or you might run one rollout at a time, underutilizing parallelism. OpenPipe ART’s architecture is explicitly designed to maximize GPU usage. It allows multiple rollouts to be executed in parallel, collecting lots of experience quickly, and then performs batch training updates on the GPU. By separating the inference and training phases and optimizing each, ART keeps the GPU busy and reduces wasteful downtime. The result is faster training and the ability to use smaller or fewer GPU resources to achieve the same results.
Lack of built-in support for LLM-specific features: Reinforcement learning with LLMs introduces challenges like handling lengthy text outputs, using pre-trained models efficiently, and applying techniques like LoRA (Low-Rank Adaptation) for fine-tuning. Many existing RL frameworks don’t have native support for these features. ART addresses this by being built for LLM agents – it uses adaptation techniques (like LoRA) to fine-tune large models efficiently, and it integrates with high-performance inference libraries for LLMs. It’s also compatible with the OpenAI API format by design, meaning it understands chat messages, tools, and other modern LLM interaction patterns. These capabilities make ART a better fit for language model agents compared to generic reinforcement learning libraries.

Architecture and integration

OpenPipe ART's workflow with enhanced GPU utilization and seamless codebase integration

How does the new architecture of ART improve GPU utilization and integration?

Frontend (Client) – lightweight and integrative: The ART client runs within your application’s process. It serves as a drop-in replacement for making model calls. In fact, it provides an OpenAI-compatible interface for your model. This means if your code is already using OpenAI’s API (for example, calling client.chat.completions.create), you can switch to ART’s client with minimal changes. The client sends requests to the ART backend and streams back the model’s responses. Because the client mimics the OpenAI API, you don’t have to change how you format prompts or parse outputs – integration is seamless. You essentially embed ART into your existing system without a heavy rewrite. The client also takes care of logging interactions and preparing data for training, invisible to the developer.
Backend (Server) – optimized for performance: The ART backend is where the heavy lifting happens, and it’s designed to maximize GPU usage. The backend can run on any machine with a GPU (local or remote) and is responsible for both generating model tokens (inference) and performing training updates on the model. One key component used here is vLLM, a high-performance inference engine for language models. vLLM allows the backend to handle many simultaneous generation requests efficiently, which is perfect for running multiple parallel agent rollouts. In ART’s training loop, the backend might serve, say, 8 parallel query streams of the agent interacting with its environment – keeping the GPU fully occupied generating text. Once those rollouts are done and rewards are assigned, the backend switches to training mode. It uses a reinforcement learning algorithm called GRPO (Group Relative Policy Optimization) to update the model’s weights (specifically, an attached LoRA adapter) based on the collected experiences. ART’s architecture smartly manages GPU memory as it switches between inference and training, ensuring that neither stage unnecessarily blocks the other more than needed.
GPU utilization improvements: By batching tasks and parallelizing rollouts, ART avoids the common scenario where a GPU sits idle waiting for one long episode to finish. Instead, multiple episodes (or conversations) can be run concurrently. After each training iteration, the updated model (or more precisely the updated LoRA weights) are loaded back into the inference engine so that subsequent rollouts use the latest model. This iterative process continues for a set number of training cycles. The net effect is a highly efficient use of the GPU – nearly all the time either generating text or applying gradient updates. This efficiency can dramatically reduce training time (and cost), especially compared to naive approaches where generation and training are not well-coordinated.
Seamless integration through OpenAI-compatible endpoint: As mentioned, integration is easy because ART’s client exposes an interface that mirrors the OpenAI API. For example, you can get an openai_client = model.openai_client() from ART, then use openai_client.chat.completions.create(...) just like you would with the official OpenAI library. The difference is, behind the scenes the request is routed to your fine-tuning backend rather than an external API. This compatibility means you can embed ART into various applications (chatbots, autonomous agents, tool-using agents, etc.) without having to reinvent how you call the model. Furthermore, it allows ART to work with popular agent frameworks or libraries – if those libraries support OpenAI API models, they can work with an ART-managed model as well. In summary, the new architecture improves GPU utilization by parallelizing and carefully managing the inference-training cycle, and it improves integration by cleanly separating concerns: your code talks to a familiar interface (client) and the backend focuses on optimized training.

Components and open-source availability

Open-source components: The core of ART – including the client library, the training backend, and the reinforcement learning algorithm implementations – is fully open source. The project is available on GitHub under an open license (Apache 2.0), meaning you can inspect the code, contribute improvements, or customize it for your needs. Many pieces that ART uses are themselves open source projects. For instance, ART’s training relies on Unsloth (an open-source library for finetuning language models with reinforcement learning techniques) and Hugging Face’s trl library for some underlying algorithms. The inference side uses vLLM, which is also open source. This openness is great for the community: you’re not locked into a proprietary system, and you can run everything on your own infrastructure if desired.
Frontend / Client: The ART client (Python SDK) is open source and intended to be embedded in your application. You install it via pip install openpipe-art and use it in your code – there are no hidden components here.
Backend options: For the backend (the actual training server), ART provides two main modes:
Local Backend: If you have your own machine with a suitable GPU, you can run the training backend locally. This might be a single server or even your personal computer if it has a strong GPU. The local backend is simply a process that will spin up the vLLM engine and the training loop on the local machine. This mode keeps everything self-contained and is fully open source – you have control over the environment.
Hosted/Ephemeral Backend: If you don’t have a GPU machine available or prefer not to use your own, ART offers an easy way to launch a backend on the cloud. It integrates with a tool called SkyPilot to provision an optional hosted backend. With a single command, ART can spin up a remote server (for example, on a cloud provider or a GPU rental service like RunPod) with the required GPU, set up all the necessary libraries, and deploy the training backend there. This cluster will then act as your ART backend, communicating with your local client. The advantage here is convenience – you get a powerful GPU when you need it, and you can tear it down when training is over (so you only pay for what you use). Even this hosted mode uses open-source tooling (SkyPilot and the ART package itself); it’s just orchestrating cloud resources on your behalf. The OpenPipe team runs a coordinator service to make this seamless, but you are not tied to a proprietary backend – you could also manually deploy the open-source server on your own cloud instance if you prefer.

How does ART handle training costs?

Agent learning from its own experiences via OpenPipe ART

Efficient use of resources: As discussed earlier, ART keeps the GPU busy with parallel rollouts and batched training. This efficiency means you get more learning done in less time. Faster training directly translates to lower cost when you’re paying hourly for cloud GPU instances. For example, instead of a GPU sitting idle waiting for data, ART’s backend maximizes throughput. In practical terms, if it takes 2 hours for an ART training run on a given task, that might be significantly shorter than a naive training loop which could take 4+ hours for the same amount of experience. By cutting wall-clock time, ART cuts your cloud bill.
Ephemeral GPU usage: With the SkyPilot-based backend, you can provision a powerful GPU machine only for the duration of training and shut it down immediately afterward. ART makes this process easy (even automatable). This means you only pay for the exact training time. There’s no need to keep a server running 24/7. If an experiment finishes early, the backend can be brought down to avoid extra charges. This granular control is a big cost saver, especially for ad-hoc training jobs or iterative experimentation.
LoRA fine-tuning for lower compute costs: OpenPipe ART uses LoRA (Low-Rank Adaptation) to fine-tune the model’s weights. Instead of updating all billions of parameters of a large model (which would be memory and compute intensive), LoRA trains a much smaller set of additional parameters (often just a few million). This drastically reduces the GPU memory required and speeds up each training update. The outcome is that training runs consume less GPU time and can even be done on smaller GPUs than would otherwise be needed. By minimizing the computational load of each training step, ART reduces the overall cost of training runs.
Optimized algorithm (GRPO): The reinforcement learning algorithm at the heart of ART, Group Relative Policy Optimization (GRPO), is designed to be stable and sample-efficient. In RL, “sample-efficient” means you need fewer interactions (and thus less computation) to achieve good results. GRPO leverages comparisons of group outcomes to derive rewards, which can squeeze more learning out of each batch of rollouts. In effect, the agent can converge to a high-performing policy with fewer training iterations compared to standard algorithms. Fewer iterations = fewer GPU cycles = lower cost.
Cost estimates and monitoring: ART encourages good practices by making it easy to monitor your training. By integrating with observability tools like Weights & Biases, you can track metrics such as reward trends, losses, and even track how long training is taking. This visibility allows you to gauge if an experiment is promising or if it’s converging slowly. If something isn’t working, you can early-stop to save time and money. In the OpenPipe ART examples, they often give an idea of the resources needed – for instance, a tutorial might mention “Training time: 2 hours on a T4 GPU (cost: ~$0, if using free Colab)”. Indeed, some simpler ART training runs can be done on free tiers (like Google Colab’s free GPU), effectively costing nothing but time. For more intensive runs, you might use a rented GPU, but knowing the ballpark (say, a few hours on an affordable GPU instance) upfront helps plan the budget.
Scaling down or up as needed: ART doesn’t force you to always use the largest models. You can choose a smaller base model that’s cheaper to train if it suffices for your task. For example, if you’re training an agent to play a simple game or follow basic instructions, a 3 billion parameter model might be enough (which is much cheaper to train than a 70B model). ART’s ability to work with various model sizes gives you the option to trade off cost and performance. Start small to validate the approach, which keeps initial costs low; only scale up if necessary.

Tutorial: Implementing OpenPipe ART with Weights & Biases

pip install openpipe-art wandb

import wandb

wandb.login()  # you'll be prompted to enter your W&B API key (from your W&B account page)
wandb.init(project="my-agentic-task", name="openpipe-art-demo")

import art  # import the OpenPipe ART library

# Create a trainable model instance
model = art.TrainableModel(
    name="agent-001",                # an arbitrary name for your model (used for logging)
    project="my-agentic-task",       # W&B project name for grouping runs
    base_model="Qwen/Qwen2.5-3B",    # base model to fine-tune (needs to be a model identifier or path)
)

# Set up the training backend
backend = art.LocalBackend()  # use local GPU; ensure your machine has a GPU and drivers if using this

# If no local GPU, you can use SkyPilotBackend as follows:
# from art.skypilot import SkyPilotBackend
# backend = await SkyPilotBackend.initialize_cluster(cluster_name="art-demo", gpu="A10")  
# (The above would asynchronously provision a remote GPU machine, e.g., with an Nvidia A10 card.)

# Register the model with the backend (prepare the backend server)
await model.register(backend)

TrainableModel sets up the model. We provided base_model="Qwen/Qwen2.5-3B" as an example – this should correspond to a model checkpoint accessible to the ART system (if it’s a public model on Hugging Face Hub, ART will attempt to download it). You could replace this with another model you want to fine-tune.
We create a LocalBackend, which will launch a local server process for training. When we call await model.register(backend), ART starts up the backend (loading the model in the backend’s vLLM server and getting ready to train). This step may take a little time initially, as it needs to load the model weights into memory.
Note: The await indicates that register is an asynchronous operation. In a notebook environment that supports top-level await, this is fine. In a regular Python script, you’d want to run this inside an asyncio event loop. For simplicity, assume this code is in a notebook or an async context. (If using a script, wrap these calls in an async def main() and use asyncio.run(main()).)

import random
import math

# Define one scenario rollout for the multiplication task
async def rollout(model: art.TrainableModel) -> art.Trajectory:
    # Step 1: Prepare a random multiplication problem
    a = random.randint(1, 20)
    b = random.randint(1, 20)
    question = f"What is {a} * {b}?"
    # We use the model's OpenAI-compatible client for inference
    openai_client = model.openai_client()
    # Create the conversation message (system prompt can be empty or an instruction)
    messages = [
        {"role": "user", "content": question}
    ]
    # Step 2: Get the model's answer
    completion = await openai_client.chat.completions.create(
        model=model.name,
        messages=messages,
        max_tokens=10
    )
    answer = completion.choices[0].message.content.strip()

    # Step 3: Calculate reward based on correctness
    correct_answer = str(a * b)
    if answer == correct_answer:
        reward_value = 1.0   # correct answer, positive reward
    else:
        reward_value = -1.0  # incorrect answer, negative reward

    # Step 4: Package into a Trajectory and assign reward
    trajectory = art.Trajectory(messages=messages + [{"role": "assistant", "content": answer}])
    trajectory.reward = reward_value
    return trajectory

We randomly generate two integers a and b for the multiplication problem.
We form a user message asking the product of these numbers. (In a more complex agent, there could be a system message with instructions too, but here it’s straightforward.)
We get an openai_client from our model, and use its chat.completions.create method to get the model’s response. We specify model=model.name which ensures the request goes to our fine-tuning instance (the one we registered on the backend).
The model’s answer is captured. We strip it to avoid any formatting issues.
We then determine the reward: if the answer matches the true product, reward = +1, otherwise -1. (We’re using a simple reward scheme; more complex tasks might have more nuanced scoring.)
We create an art.Trajectory object to record the interaction. We include both the user question and the assistant’s answer in the messages. Then we assign the reward to the trajectory.
The trajectory is returned, encapsulating one rollout of our agent in the environment.

# Train the model for a certain number of steps
training_steps = 50
rollouts_per_step = 8  # how many problems to attempt per step (parallel rollouts)

for step in range(training_steps):
    # Collect trajectories from multiple parallel rollouts
    trajectories = [rollout(model) for _ in range(rollouts_per_step)]
    # Wait for all the asynchronous rollouts to finish
    trajectories = await asyncio.gather(*trajectories)

    # (ART will automatically use these trajectories to train the model via GRPO)
    # After this point, the model's parameters (LoRA weights) are updated on the backend.
    
    # Log the mean reward of this batch for monitoring
    avg_reward = sum(t.reward for t in trajectories) / len(trajectories)
    wandb.log({"step": step, "average_reward": avg_reward})

# After training is done, save the LoRA weights 
final_lora_dir = "./.art/models"  # hypothetical path where ART stored LoRA checkpoints
# Log the fine-tuned model as a W&B artifact for versioning
artifact = wandb.Artifact(name="multiplication-agent-lora", type="model")
artifact.add_dir(final_lora_dir)
wandb.log_artifact(artifact)

test_questions = ["What is 7  8?", "What is 15  14?", "What is 3 * 19?"]
for q in test_questions:
    completion = await openai_client.chat.completions.create(
        model=model.name,
        messages=[{"role": "user", "content": q}]
    )
    answer = completion.choices[0].message.content.strip()
    print(f"Q: {q} -> A: {answer}")

Conclusion

Overcoming reinforcement learning challenges with OpenPipe ART

Enhanced reward modeling: Currently, you define rewards for each trajectory manually or via simple heuristics. In the future, ART might integrate more advanced techniques like learned reward models or leverage human feedback more directly (similar to RLHF) so that agents can be trained on qualitative criteria. In fact, ART already introduced a feature called RULER that uses an LLM as a judge to score trajectories, eliminating the need for hand-crafted reward functions in many cases. We can expect more features like this to lower the barrier for defining “success” for an agent.
Broader model support and scalability: As larger and more specialized models emerge, ART will likely extend support for them. The separation of frontend and backend means it’s well-positioned to handle distributed training or very large models sharded across hardware, which could be on the roadmap as the project matures beyond the alpha stage. We might also see optimizations for new hardware or efficiency improvements in the GRPO algorithm.
Production integration: Today, ART is great for research and prototyping, but one can imagine tools to help use ART in production systems. For example, automatically training on real user interactions (with proper guardrails) to continuously improve a deployed agent is a compelling application. Future releases of ART might focus on making such continual learning safe and easy – so your agent in production gets smarter with each day of use, retraining on feedback overnight.
Community contributions and ecosystem: Being open source, ART’s direction will also be influenced by its user community. We anticipate integrations with other frameworks (for instance, plugging ART into LangChain or similar agent orchestration libraries), community-created scenario libraries, and more pre-built examples. Weights & Biases’ ecosystem around ART will also grow – with public dashboards, reports (like this one), and even community contributions to best practices in using Weave for agent analysis.