Skip to main content

Getting started with embodied chain-of-thought (ECoT)

Learn how to get started with Embodied Chain-of-Thought, a robotics reasoning approach that helps vision-language-action models plan, explain, and execute grounded actions.
Last updated: June 17, 2026
Embodied Chain-of-Thought Reasoning (ECoT) is a method to train robots to think through a task in natural language before acting. Instead of mapping an image and a language instruction directly to motor commands, the robot first produces a written reasoning chain that covers the plan, the current sub-task, the motion it needs to perform, where objects are in the scene, and where its gripper is, and then predicts the action from that chain. This article explains what ECoT is, why it works better than simpler reasoning approaches, how the synthetic data pipeline behind it scales to large robot datasets, and how to start experimenting with it using Weights & Biases.
By the end of this article, you will understand the core principles of embodied chain-of-thought, the data-generation approach, the 28% performance gain reported in the paper, and how W&B Weave makes every step of the pipeline observable at data generation time, during training, and in production.

What we're covering



What embodied chain-of-thought reasoning is

Embodied chain-of-thought (ECoT) reasoning trains a robot policy to generate grounded, step-by-step reasoning about a task, its visual scene, and its own physical state before it predicts an action. Rather than mapping a camera image and instruction straight to a motor command, the model first "thinks" in structured text and then acts on that reasoning. It's a technique for vision-language-action models, not text chatbots.
The chain covers several levels at once'
  • a high-level task restatement,
  • a multi-step plan,
  • the current sub-task,
  • a motion description,
  • the pixel coordinates of visible objects,
  • the 2D position of the end-effector in the image, and
  • the gripper state.
A VLA is a pre-trained vision-language model fine-tuned to also output robot actions: it takes a camera image and a natural language instruction as input and predicts the next motor command as output tokens. OpenVLA is the main open-source VLA used in the paper Robotic Control via Embodied Chain-of-Thought Reasoning (Zawalski et al., 2025). It combines a SigLIP visual encoder, a DinoV2 visual encoder, and a Llama 2 7B backbone, and serves as the base model throughout.



From Robotic Control via Embodied Chain-of-Thought Reasoning (Zawalski et al., 2025)
The value of ECoT is threefold:
  1. the policy generalizes better to novel scenes because it learns why an action fits the situation,
  2. it is more robust because the reasoning anchors attention to specific visual evidence, and
  3. it is interpretable because the chain is readable text that can be inspected and corrected before the robot moves.

How ECoT differs from naïve CoT and why grounding matters

Standard chain-of-thought prompting adds reasoning text before the answer. With language models, this works well because training data is rich enough for the model to reason meaningfully. In robotics, the problem is that robot demonstration data is far simpler, as most steps are short motions like "move left" or "close gripper," and there is nothing in that data to ground verbal reasoning to what the robot actually sees.
A naive CoT chain for "pick up the mug" might say: "I need to find the mug and pick it up. I will move my arm toward it and close the gripper." This sounds reasonable but references nothing specific - no object location, no arm pose, no spatial context.
An ECoT chain for the same instruction names the mug's bounding box, states the gripper's pixel position, describes the spatial relationship between the arm and the mug, and then commits to a specific motion primitive. That grounding forces the policy to look carefully at the scene before acting, not just think carefully about the task in the abstract. The paper shows this distinction is responsible for most of the performance gain over naive CoT.

How ECoT fits into VLA models such as OpenVLA

ECoT inserts a reasoning phase between the instruction and the action tokens. The model generates the full chain - TASK, PLAN, SUBTASK REASONING, SUBTASK, MOVE REASONING, MOVE, GRIPPER POSITION, VISIBLE OBJECTS, autoregressively, and then predicts the action tokens from that grounded intermediate representation. Language becomes the control signal rather than a preprocessing wrapper.
This is not a separate module bolted onto the policy. It is the same model predicting the same sequence, just with reasoning tokens before action tokens. The model learns to attend to its visual inputs during chain generation, so the action prediction happens in the context of a written summary of what the robot just "looked at."

How the synthetic data pipeline scales ECoT training

The most expensive part of ECoT is supervision. You need a reasoning chain for every step of every demonstration in your training dataset. Handwriting chains for millions of transitions is not feasible. The paper solves this with a five-step automated pipeline that generates synthetic chains from existing robot data, with no human labeling required.
  • Step 1 generates a scene description from the image and task instruction using a vision-language model.
  • Step 2 extracts bounding boxes for all visible objects using Grounding DINO, with the concatenated task instruction and scene description as the text prompt.
  • Step 3 computes motion primitives deterministically from the robot's proprioceptive state using a rule-based classifier.
  • Step 4 projects the 3D end-effector position to 2D pixel coordinates.
  • Step 5 feeds all of the above into a large language model to generate the final structured chain.
The paper ran this on 60,000 Bridge V2 demonstrations (2.5 million transitions) over seven days.

How ECoT improves generalization, robustness, and real-world performance

The headline result is a 28% point absolute improvement in success rate on challenging generalization tasks compared to a plain OpenVLA baseline trained on the same data. The ECoT model even outperforms RT-2-X, a 55-billion-parameter closed model trained on 10 additional robot datasets, despite being a 7B-parameter model trained only on Bridge V2.
The evaluation covers novel objects, novel instruction phrasings, novel spatial relations, and out-of-distribution camera viewpoints. ECoT handles these conditions better because the chain forces the model to identify what it actually sees in the novel scene before committing to an action. The comparison against naive CoT confirms that the grounding - bounding boxes, gripper position, motion primitives - is what drives most of the improvement, not the chain-of-thought structure alone.

How embodied chain-of-thought enables interpretable human-robot interaction

Embodied chain-of-thought makes policies interpretable because the chain is readable text before the robot moves. If the chain says SUBTASK REASONING: The screwdriver is in the top-left corner but the screwdriver is on the right, a human can read that mistake and correct it in one sentence. The paper uses a correction LLM to reformulate the chain in light of the feedback, then feeds the corrected chain back into the policy for the next few steps.
On the hardest evaluation tasks, where the uncorrected policy succeeds 32% of the time, a single human intervention lifts the ECoT success rate by 48 % points.
Vanilla OpenVLA, given the same natural language correction, shows almost no improvement because it has no mechanism for incorporating reasoning-level feedback. This interpretability matters for trust and safe deployment: when a policy fails, the chain tells you whether the failure is a perception problem, a planning problem, or an execution problem, which is far harder to diagnose from raw action sequences.

Tutorial: Getting started with Embodied Chain-of-Thought

The notebook we built implements the paper's full five-step pipeline against Bridge V2. What follows explains each step: the idea behind it, why it was built that way, and what the implementation does.

Step 1: Install dependencies

The idea here is to keep the environment minimal. The notebook uses -
  • Groq for free hosted LLM inference,
  • huggingface_hub for downloading Bridge V2 data,
  • av for decoding MP4 video frames,
  • weave and wandb for tracing and logging, and
  • standard scientific Python.
No GPU required. This is a deliberate constraint -> the paper uses Grounding DINO and OWLv2+SAM for Steps 2 and 4, both of which require a GPU and 700+ MB of weights. For a tutorial notebook, keeping everything API-based means that any reader can run it on a free Colab CPU instance with just two API keys.

Reference to the Grounding DINO, Owl+SAM from the paper - Robotic Control via Embodied Chain-of-Thought Reasoning" (Zawalski et al., 2025)
!pip install -q groq huggingface_hub weave wandb Pillow numpy pandas av matplotlib

Step 2: Setup

The idea is to establish the two inference clients and initialize Weave before any data loads. We use -
  • meta-llama/llama-4-scout-17b-16e-instruct for vision calls (scene description and bounding boxes) and
  • llama-3.3-70b-versatile for text calls (chain generation and human correction).
Both are on Groq's free tier with a 30 req/min rate limit.
weave.init("ecot-pipeline-demo") creates the project in W&B where every subsequent @weave.op() trace will appear. Initializing Weave here means the trace coverage starts before the data pipeline runs, not after.
import os, json, base64, io, math, textwrap, time
import numpy as np
import pandas as pd
import av
from PIL import Image, ImageDraw
from collections import defaultdict, Counter
from groq import Groq
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import weave
import wandb
from google.colab import userdata
from IPython.display import display

os.environ["GROQ_API_KEY"] = userdata.get("GROQ_API_KEY")
os.environ["WANDB_API_KEY"] = userdata.get("WANDB_API_KEY")
os.environ["HF_TOKEN"] = userdata.get("HF_TOKEN")

groq_client = Groq(api_key=os.environ["GROQ_API_KEY"])

VISION_MODEL = "meta-llama/llama-4-scout-17b-16e-instruct"
TEXT_MODEL = "llama-3.3-70b-versatile"

weave.init("ecot-pipeline-demo")
print(f"Vision : {VISION_MODEL}")
print(f"Text : {TEXT_MODEL}")
print("Weave : initialized")





Step 3: Load Bridge V2

Now we want to access Bridge V2 without the lerobot library, which had version incompatibilities that broke loading in earlier iterations. The paper's training set is IPEC-COMMUNITY/bridge_orig_lerobot(link to dataset). 19,974 task labels and 60,000 episodes. We stream only the parquet files (action vectors, proprioceptive state, task index) and the MP4 video files for 100 episodes, avoiding a full corpus download.
Why this matters: Bridge V2 stores images as MP4 video files rather than individual frames, and task labels in a separate meta/tasks.jsonl file.
💡
The raw datasets library does not decode either of these correctly. Pulling parquet files directly with hf_hub_download and decoding video with pyav gives us clean access to the exact data the paper trained on, without any library version dependency.
from huggingface_hub import hf_hub_download

REPO_ID = "IPEC-COMMUNITY/bridge_orig_lerobot"
EPISODES_TO_LOAD = list(range(100))

# Task labels
tasks_file = hf_hub_download(repo_id=REPO_ID, filename="meta/tasks.jsonl", repo_type="dataset")
tasks = {}
with open(tasks_file) as f:
for line in f:
t = json.loads(line)
tasks[t["task_index"]] = t["task"]
print(f"Loaded {len(tasks)} task labels.")

# Parquet data per episode
episodes = defaultdict(list)
for ep_idx in EPISODES_TO_LOAD:
chunk = ep_idx // 1000
try:
p = hf_hub_download(
repo_id=REPO_ID,
filename=f"data/chunk-{chunk:03d}/episode_{ep_idx:06d}.parquet",
repo_type="dataset",
)
df = pd.read_parquet(p)
for _, row in df.iterrows():
episodes[ep_idx].append(row.to_dict())
except Exception as e:
print(f" [skip] ep {ep_idx}: {e}")

print(f"Loaded {len(episodes)} episodes.")
for ep_idx, frames in list(episodes.items())[:5]:
task = tasks.get(frames[0].get("task_index", 0), "unknown")
print(f" Ep {ep_idx}: {len(frames)} frames — '{task}'")

Output :



Step 4: Episode and video quality filtering

The idea here is to apply the same data quality criteria the paper uses before spending any API credits. Four filters run in sequence:
  1. Task label word count (paper Appendix C requires at least one space in the instruction, discarding labels like "", "bmbfbbfgjjg", or single-word entries),
  2. Minimum episode length (fewer than 5 frames discarded),
  3. Video file size (stub MP4 files of ~2 KB are discarded; real episodes are 90–400 KB), and
  4. Black frame brightness check (mean pixel value below 12 is discarded).
Why this matters: Without these filters, roughly 70% of the 100 loaded episodes are unusable. Bridge V2 contains numerous corrupted stub video files and noisy task labels in the early episodes. Running every episode through the VLM pipeline without filtering would waste Groq API credits on data that cannot produce useful chains. Out of 100 episodes loaded, around 30 pass all four filters that are consistent with the distribution of valid data in the early portion of the dataset.
MIN_VIDEO_BYTES = 10_000
MIN_FRAME_MEAN = 12.0
MIN_EPISODE_FRAMES = 5
MIN_TASK_WORDS = 2

GARBAGE_TASK_PATTERNS = {
"end effector reaching",
"end effector transition",
"the robotic arm did not",
"bmbf", "asdf", "test",
}


def is_valid_task(task: str) -> bool:
if len(task.strip().split()) < MIN_TASK_WORDS:
return False
if any(task.lower().startswith(p) for p in GARBAGE_TASK_PATTERNS):
return False
return True


def load_valid_frame(video_path: str):
if os.path.getsize(video_path) < MIN_VIDEO_BYTES:
return None
try:
container = av.open(video_path)
for frame in container.decode(video=0):
img = frame.to_image()
container.close()
if np.array(img).mean() < MIN_FRAME_MEAN:
return None
return img
except Exception:
return None


def load_all_valid_frames(video_path: str) -> list:
if os.path.getsize(video_path) < MIN_VIDEO_BYTES:
return []
try:
container = av.open(video_path)
frames = [f.to_image() for f in container.decode(video=0)]
container.close()
if not frames or np.array(frames[0]).mean() < MIN_FRAME_MEAN:
return []
return frames
except Exception:
return []


def get_video_path(ep_idx: int):
chunk = ep_idx // 1000
for vkey in ["observation.images.image_2", "observation.images.image_3"]:
try:
vp = hf_hub_download(
repo_id=REPO_ID,
filename=f"videos/chunk-{chunk:03d}/{vkey}/episode_{ep_idx:06d}.mp4",
repo_type="dataset",
)
if os.path.getsize(vp) >= MIN_VIDEO_BYTES:
return vp
except Exception:
continue
return None


def get_keyframe(ep_idx: int):
vp = get_video_path(ep_idx)
if vp is None:
return None
return load_valid_frame(vp)


def get_frame_at_index(ep_idx: int, frame_idx: int):
"""Return a specific frame from the episode video by index."""
vp = get_video_path(ep_idx)
if vp is None:
return None
try:
container = av.open(vp)
frames = [f.to_image() for f in container.decode(video=0)]
container.close()
if frame_idx < len(frames):
img = frames[frame_idx]
if np.array(img).mean() >= MIN_FRAME_MEAN:
return img
except Exception:
pass
return None


def get_all_frames(ep_idx: int) -> list:
vp = get_video_path(ep_idx)
if vp is None:
return []
return load_all_valid_frames(vp)


# Pre-filter sweep
FILTER_REASONS = {"bad_task": [], "short_ep": [], "bad_video": [], "black_frame": []}
valid_episodes = {}

print(f"Pre-filtering {len(episodes)} episodes...\n")
for ep_idx, frames in episodes.items():
task = tasks.get(frames[0].get("task_index", 0), "")
if not is_valid_task(task):
FILTER_REASONS["bad_task"].append(ep_idx)
continue
if len(frames) < MIN_EPISODE_FRAMES:
FILTER_REASONS["short_ep"].append(ep_idx)
continue
vp = get_video_path(ep_idx)
if vp is None:
FILTER_REASONS["bad_video"].append(ep_idx)
continue
img = load_valid_frame(vp)
if img is None:
FILTER_REASONS["black_frame"].append(ep_idx)
continue
valid_episodes[ep_idx] = frames

print(f"{'─'*52}")
print(f" Total loaded : {len(episodes)}")
print(f" ✓ Kept : {len(valid_episodes)}")
print(f" ✗ Bad task label : {len(FILTER_REASONS['bad_task'])} ep {FILTER_REASONS['bad_task']}")
print(f" ✗ Too short : {len(FILTER_REASONS['short_ep'])} ep {FILTER_REASONS['short_ep']}")
print(f" ✗ No/corrupt video : {len(FILTER_REASONS['bad_video'])} ep {FILTER_REASONS['bad_video']}")
print(f" ✗ Black frame : {len(FILTER_REASONS['black_frame'])} ep {FILTER_REASONS['black_frame']}")
print(f"{'─'*52}")
print("\nValid episodes:")
for ep_idx, frames in valid_episodes.items():
task = tasks.get(frames[0].get("task_index", 0), "")
print(f" Ep {ep_idx:02d}: {len(frames):3d} frames — {task[:60]}")

Output :


Step 5: Motion primitive computation

Here we're reproducing the paper's Appendix B algorithm - the most faithful cell in the notebook, requiring no model or API call.
The algorithm takes the difference between the robot's absolute end-effector pose (observation.state) at the current step and four steps ahead, applies a 0.03 threshold per axis, and composes movement labels from a 729-template taxonomy. The labels cover forward, backward, left, right, up, down, tilt up (from droll), tilt down, rotate clockwise, rotate counterclockwise, open gripper, and close gripper. Multi-axis motions produce compound labels like move right down tilt up rotate counterclockwise.
Why this matters: Using observation.state (absolute pose) rather than the action vector (velocity delta) is a critical distinction that previous versions of the notebook got wrong. Taking the difference between two velocity commands at two time points is not the same as measuring the actual displacement over four steps. The corrected version produces rich compound primitives averaging over five words per label, compared to mostly stop and rotate when using the action vector which gives the chain generation LLM much more informative motion context to reason from.
find_representative_frame is an improvement over always using frame 0. Let's break it down what this is -
  • Think of a BridgeV2 episode like a short video clip of a robot completing a task. The clip always starts with the robot stationary before it has touched anything - arm at rest, no motion, just a static kitchen scene. If you took a screenshot at that exact moment and asked a vision model to describe what is happening and which objects matter, you would get a generic response. The scene looks like nothing is happening yet, so the model has no useful context to ground its reasoning in.
  • That is exactly the problem with always using frame 0. The motion primitive at the start of most episodes is stop, which tells the chain generation model very little about what the robot is trying to do or which object it needs to interact with. The chains produced from these frames tend to be vague — "the robot should assess the scene before beginning" rather than anything grounded in specific objects or positions.
  • find_representative_frame solves this by scanning every step in the episode, computing the motion primitive at each one, and picking the frame where the most is happening, which is measured by how many components the primitive has. This set - move right down tilt up rotate counterclockwise scores higher than stop. At that step, the robot arm is mid-manipulation, i.e -approaching an object, rotating toward a target, or actively grasping. The vision model sees a far more informative scene and the resulting chain is noticeably richer with SUBTASK REASONING that references specific visible objects, MOVE REASONING that explains the spatial relationship between the arm and the target, rather than generic text that could apply to any episode.
Worth noting: this is a simplification the paper does not need to make. The paper generates one chain per step across the full episode, so every frame gets a chain including the static ones at the start. We generate one chain per episode as a tutorial constraint, so picking the most informative frame is our way of getting as much signal as possible from that single call.
💡
THRESHOLD = 0.03
def _to_list(x) -> list:
if isinstance(x, str):
try:
return json.loads(x)
except Exception:
return [0] * 7
if hasattr(x, "tolist"):
return x.tolist()
try:
return list(x)
except Exception:
return [0] * 7

def compute_motion_primitive(frames: list, step_idx: int, lookahead: int = 4) -> str:
def get_state(frame):
state = frame.get("observation.state")
if state is not None:
s = _to_list(state)
if len(s) >= 6:
return s
return _to_list(frame.get("action", [0] * 7))

current = get_state(frames[step_idx])
future = get_state(frames[min(step_idx + lookahead, len(frames) - 1)])

dx = future[0] - current[0]
dy = future[1] - current[1]
dz = future[2] - current[2]
droll = future[3] - current[3]
dyaw = future[5] - current[5]

grip = future[6] if len(future) > 6 else 1.0
current_grip = current[6] if len(current) > 6 else 1.0

parts, move_parts = [], []
if abs(dx) > THRESHOLD: move_parts.append("forward" if dx > 0 else "backward")
if abs(dy) > THRESHOLD: move_parts.append("right" if dy > 0 else "left")
if abs(dz) > THRESHOLD: move_parts.append("up" if dz > 0 else "down")
if move_parts: parts.append("move " + " ".join(move_parts))
if abs(droll) > THRESHOLD: parts.append("tilt " + ("up" if droll > 0 else "down"))
if abs(dyaw) > THRESHOLD: parts.append("rotate " + ("clockwise" if dyaw > 0 else "counterclockwise"))
if grip > 0.5 and current_grip <= 0.5: parts.append("open gripper")
elif grip <= 0.5 and current_grip > 0.5: parts.append("close gripper")

return " ".join(parts) if parts else "stop"

def find_representative_frame(episode_frames: list) -> int:
best_idx, best_score = 0, 0
for i in range(len(episode_frames)):
score = len(compute_motion_primitive(episode_frames, i).split())
if score > best_score:
best_score, best_idx = score, i
return best_idx

# Verify
ep0 = episodes[0]
rep_idx = find_representative_frame(ep0)
print(f"Episode 0 - representative frame: {rep_idx}")
print(f" Primitive at frame 0 : {compute_motion_primitive(ep0, 0)}")
print(f" Primitive at frame {rep_idx:2d} : {compute_motion_primitive(ep0, rep_idx)}")
print()
print("First 10 steps:")
for i in range(min(10, len(ep0))):
print(f" step {i:02d}: {compute_motion_primitive(ep0, i)}")

Output :


Step 6: API helpers and pipeline steps 1, 2, 4, 5

Implement the four VLM-dependent steps of Figure 4 in one cell, with Weave tracing on every function that makes an external call.
  • call_vision and call_text both include time.sleep(1.5) at the top. This is not optional. Groq's free tier allows 30 requests per minute per model, and each episode makes 3 calls. Without sleeping, the pipeline hits rate limits around episode 10 and starts failing silently.
    • def encode_data_uri(pil_image: Image.Image) -> str:
      buf = io.BytesIO()
      pil_image.convert("RGB").save(buf, format="JPEG", quality=85)
      b64 = base64.standard_b64encode(buf.getvalue()).decode("utf-8")
      return f"data:image/jpeg;base64,{b64}"
      
      def call_vision(image_uri: str, prompt: str, max_tokens: int = 1024) -> str:
      # max_tokens raised from 512 to 1024: bbox JSON for 6-8 objects easily hits 600+ tokens
      time.sleep(1.5)
      resp = groq_client.chat.completions.create(
      model=VISION_MODEL,
      messages=[{"role": "user", "content": [
      {"type": "image_url", "image_url": {"url": image_uri}},
      {"type": "text", "text": prompt},
      ]}],
      max_tokens=max_tokens,
      )
      return resp.choices[0].message.content.strip()
      
      def call_text(prompt: str, max_tokens: int = 1024) -> str:
      time.sleep(1.5)
      resp = groq_client.chat.completions.create(
      model=TEXT_MODEL,
      messages=[{"role": "user", "content": prompt}],
      max_tokens=max_tokens,
      )
      return resp.choices[0].message.content.strip()
      
  • describe_scene uses the paper's exact Appendix C prompt: "The robot task is: {task}. Briefly describe the things in this scene and their spatial relations to each other."
    • SCENE_DESCRIPTION_PROMPT = (
      "The robot task is: {task}. "
      "Briefly describe the things in this scene and their spatial relations to each other."
      )
      @weave.op()
      def describe_scene(task: str, image_uri: str) -> str:
      """Step 1 (Figure 4): scene description via VLM. Weave traces input + output."""
      return call_vision(image_uri, SCENE_DESCRIPTION_PROMPT.format(task=task))
  • extract_bboxes concatenates the task instruction and scene description as one string before sending to the vision model. This matches how Grounding DINO receives input in the paper: task plus description as a combined text prompt. Separating them as two distinct inputs reduces the model's ability to focus on task-relevant objects.
  • max_tokens is set to 1024 rather than 512 because JSON for 6-8 objects with names and coordinates easily reaches 600+ tokens, and truncated JSON fails to parse. The function returns both the parsed bounding box list and the raw response string so that Weave captures exactly what the VLM said when detection fails, not just an empty list with no diagnosis.
  • validate_bboxes applies geometric quality filters equivalent to Grounding DINO's confidence thresholds: positive area, 0.2–97% image coverage, in-bounds coordinates. This is pure computation — no API call — so it is intentionally not decorated with @weave.op(). Only functions that can fail or produce variable output benefit from tracing.
# matching how Grounding DINO receives input in the paper (Appendix C).
BBOX_PROMPT = """{task}. {scene_description}

List all visible objects and their bounding box coordinates.
Respond ONLY with a JSON array (no markdown):
[{{"name": "object name", "bbox": [x1, y1, x2, y2]}}, ...]
Use normalised 0-1 coordinates relative to image width/height.
"""

@weave.op()
def extract_bboxes(task: str, scene_description: str, image_uri: str) -> dict:
"""
Step 2 (Figure 4): bounding box extraction via VLM.
Returns dict with bboxes + raw_response so Weave captures failures too —
when bboxes=[] you can read the raw VLM output directly in the trace.
"""
raw = call_vision(image_uri, BBOX_PROMPT.format(task=task, scene_description=scene_description))
cleaned = raw.replace("```json", "").replace("```", "").strip()
try:
result = json.loads(cleaned)
bboxes = result if isinstance(result, list) else []
except json.JSONDecodeError:
bboxes = []
# Weave improvement: log raw_response so failures are diagnosable in the trace
return {"bboxes": bboxes, "raw_response": raw, "parse_success": len(bboxes) > 0}

def validate_bboxes(bboxes: list, img_w: int = 256, img_h: int = 256) -> list:
"""
Geometric quality filter equivalent to Grounding DINO conf thresholds.
NOT a Weave op — pure computation, no external calls.
"""
valid = []
for obj in bboxes:
if "parse_error" in obj or "bbox" not in obj or "name" not in obj:
continue
try:
b = [float(v) for v in obj["bbox"]]
except (ValueError, TypeError):
continue
if len(b) != 4:
continue
x1, y1, x2, y2 = b
if any(v > 1.5 for v in b):
x1, y1, x2, y2 = x1/img_w, y1/img_h, x2/img_w, y2/img_h
if x2 <= x1 or y2 <= y1:
continue
area = (x2 - x1) * (y2 - y1)
if area < 0.002 or area > 0.97:
continue
if not (0 <= x1 < x2 <= 1.15 and 0 <= y1 < y2 <= 1.15):
continue
obj["bbox"] = [round(v, 4) for v in [x1, y1, x2, y2]]
valid.append(obj)
return valid
  • project_to_2d approximates the paper's OWLv2+SAM+RANSAC gripper projection using a fixed linear mapping calibrated to the Bridge V2 WidowX workspace. The paper outputs 2D pixel coordinates like [156, 55]. Previous versions output 3D metres like [0.297, 0.087, 0.091], which mismatched the paper's chain format and gave the chain generation LLM misleading spatial context.
  • format_objects_inline converts normalised 0–1 bounding box coordinates to pixel integers before formatting. The paper's chains show Watermelon [126, 146, 141, 125], not watermelon [0.49, 0.57, 0.55, 0.49]. This matters because the LLM generating the chain reasons about object positions in image space — normalised floats are harder to reason about spatially than pixel integers.
  • The CHAIN_GENERATION_PROMPT is adapted from Figure 11 and Appendix C of the paper with one meaningful addition: the SUBTASK REASONING field now includes an example from Figure 3 showing that the reasoning must reference specific visible objects and current gripper state, not just restate the plan.
Without this example, the LLM tends to write generic text like "we are currently at step 2" rather than "the mug is to the right of the arm and the gripper is open, so the arm needs to move right before grasping."
# Step 4: gripper state + 2D projection
def project_to_2d(pos_3d: list, img_w: int = 256, img_h: int = 256) -> list:
"""
Approximate 2D pixel projection matching paper Figure 3 format: [px, py].
Paper uses OWLv2 + SAM + RANSAC per trajectory. We use a fixed linear
approximation for the Bridge V2 WidowX workspace.
"""
if len(pos_3d) < 2:
return [img_w // 2, img_h // 2]
x, y = pos_3d[0], pos_3d[1]
px = int(np.clip((x - 0.1) / 0.4 * img_w, 0, img_w - 1))
py = int(np.clip((0.2 - y) / 0.4 * img_h, 0, img_h - 1))
return [px, py]

def extract_gripper_state_from_row(row: dict) -> dict:
def safe_list(x):
if isinstance(x, str):
try: return json.loads(x)
except: return [0] * 7
if not isinstance(x, list):
try: return list(x)
except: return [0] * 7
return x

action = safe_list(row.get("action", [0] * 7))
state = safe_list(row.get("observation.state", action))
try:
pos_3d = [round(float(state[i]), 3) for i in range(min(3, len(state)))]
gripper = "open" if len(action) > 6 and float(action[6]) > 0.5 else "closed"
pos_2d = project_to_2d(pos_3d)
return {"position_3d": pos_3d, "position_2d": pos_2d, "gripper": gripper}
except (ValueError, TypeError, IndexError):
return {"position_3d": [0.0, 0.0, 0.0], "position_2d": [128, 128], "gripper": "unknown"}

# Step 5: ECoT chain
# Paper Figure 3 + Appendix C format exactly:
# GRIPPER POSITION: 2D pixel [x, y] (paper: [156, 55])
# VISIBLE OBJECTS: pixel integers (paper: Watermelon [126, 146, 141, 125])

CHAIN_GENERATION_PROMPT = """You are an expert reinforcement learning researcher. A robotic arm has completed a trajectory.

Task instruction: "{task}"
Scene description: {scene_description}
Current motion primitive (what the robot is about to do): {motion_primitive}
Gripper state: {gripper_state}
Visible objects with bounding boxes: {objects}

Generate a single-step embodied chain-of-thought reasoning.
Respond ONLY in this exact format (no markdown, no extra text):

TASK: <restate the task goal>
PLAN: <numbered list of high-level steps>
SUBTASK REASONING: <explain which step should execute NOW and why, referencing specific visible objects and current gripper state — not just restating the plan. Example: "The watermelon is the first object the robot needs to interact with. The robot is not yet close to the watermelon, so it needs to move closer.">
SUBTASK: <current high-level step>
MOVE REASONING: <explain why this specific motion is needed, referencing object positions from VISIBLE OBJECTS>
MOVE: {motion_primitive}
GRIPPER POSITION: {gripper_position}
VISIBLE OBJECTS: {objects_inline}
"""

def format_objects_inline(bboxes: list, img_w: int = 256, img_h: int = 256) -> str:
"""Paper format: 'name [x1, y1, x2, y2]' with pixel integer coordinates."""
if not bboxes or (len(bboxes) == 1 and bboxes[0].get("parse_error")):
return "unknown"
parts = []
for o in bboxes:
if "name" not in o or "bbox" not in o:
continue
try:
b = [float(v) for v in o["bbox"]]
if all(v <= 1.15 for v in b):
px = [int(b[0]*img_w), int(b[1]*img_h), int(b[2]*img_w), int(b[3]*img_h)]
else:
px = [int(v) for v in b]
parts.append(f"{o['name']} {px}")
except (ValueError, TypeError):
continue
return ", ".join(parts) if parts else "unknown"

@weave.op()
def generate_ecot_chain(task, scene_description, motion_primitive, gripper_state, bboxes) -> str:
"""Step 5 (Figure 4): generate ECoT chain. Weave traces full prompt + output."""
return call_text(CHAIN_GENERATION_PROMPT.format(
task=task,
scene_description=scene_description,
motion_primitive=motion_primitive,
gripper_state=json.dumps(gripper_state),
objects=json.dumps(bboxes),
gripper_position=str(gripper_state.get("position_2d", [128, 128])),
objects_inline=format_objects_inline(bboxes),
))

print("All helpers defined.")

Output :


Weave Trace for extract_bboxes() :



Step 7: Run full pipeline on all valid episodes

The idea is to compose the five steps into a single @weave.op() function and run it across all valid episodes. The full Weave trace for each episode captures the inputs and outputs of every sub-call :
  • scene description,
  • bounding boxes,
  • chain generation, as nested spans under one parent trace.
This makes it straightforward to open any episode in the Weave dashboard and trace a chain-quality problem back to the specific sub-step that produced the bad input.
run_ecot_pipeline uses find_representative_frame to select which frame to send to the VLM, fetches that specific frame from the video file, computes the motion primitive and gripper state at that frame index, and runs the five-step pipeline.
The result includes :
  • rep_frame_idx (which frame was used),
  • bbox_parse_ok (whether the JSON parsed successfully), and
  • bbox_raw_response (the raw VLM output) alongside the standard pipeline outputs.
These fields make the W&B results table filterable by detection success, which is the most useful first cut when debugging output quality.
One important distinction from the paper: The paper generates one reasoning chain per step for every transition across the full episode. So 2.5 million chains total for Bridge V2. This notebook generates one chain per episode using a single representative frame. This is a deliberate simplification for the tutorial as running the full per-step pipeline on 30 episodes at 3 Groq API calls per step would require thousands of calls and is not feasible on the free tier.

The structure of what we generate is identical to the paper's format but the difference is scale. To produce a training-grade dataset you would loop over every step_idx in the episode frames, reuse the scene description and bounding boxes computed once per episode (which is what the paper does), and call generate_ecot_chain with the per-step motion primitive and gripper state for each step.
💡
@weave.op()
def run_ecot_pipeline(episode_id: int, episode_frames: list) -> dict:
task = tasks.get(episode_frames[0].get("task_index", 0), "unknown task")

# Guard: skip if video unavailable
keyframe_img = get_keyframe(episode_id)
if keyframe_img is None:
raise ValueError(f"No valid keyframe for episode {episode_id}")

# Find the most informative frame rather than always using frame 0
rep_idx = find_representative_frame(episode_frames)
rep_img = get_frame_at_index(episode_id, rep_idx) or keyframe_img
image_uri = encode_data_uri(rep_img)

motion_primitive = compute_motion_primitive(episode_frames, step_idx=rep_idx)
gripper_state = extract_gripper_state_from_row(episode_frames[rep_idx])
scene_description = describe_scene(task=task, image_uri=image_uri)

bbox_result = extract_bboxes(
task=task, scene_description=scene_description, image_uri=image_uri
)
# extract_bboxes now returns a dict with raw_response for Weave traceability
raw_bboxes = bbox_result.get("bboxes", bbox_result) if isinstance(bbox_result, dict) else bbox_result
bboxes = validate_bboxes(raw_bboxes)
bbox_raw = bbox_result.get("raw_response", "") if isinstance(bbox_result, dict) else ""
parse_ok = bbox_result.get("parse_success", len(bboxes) > 0) if isinstance(bbox_result, dict) else len(bboxes) > 0

ecot_chain = generate_ecot_chain(
task=task,
scene_description=scene_description,
motion_primitive=motion_primitive,
gripper_state=gripper_state,
bboxes=bboxes,
)
return {
"episode_id": episode_id,
"task": task,
"n_frames": len(episode_frames),
"rep_frame_idx": rep_idx, # which frame was used
"motion_primitive": motion_primitive,
"gripper_state": gripper_state,
"scene_description": scene_description,
"bboxes": bboxes,
"bbox_raw_response": bbox_raw, # for failure diagnosis in Weave
"bbox_parse_ok": parse_ok,
"ecot_chain": ecot_chain,
}


results = []
for ep_id, ep_frames in valid_episodes.items():
current_task = tasks.get(ep_frames[0].get("task_index", 0), "unknown")
print(f"\nEp {ep_id}: {current_task[:60]}")
try:
r = run_ecot_pipeline(episode_id=ep_id, episode_frames=ep_frames)
results.append(r)
print(f" frame={r['rep_frame_idx']:2d} | primitive={r['motion_primitive']} | "
f"gripper={r['gripper_state']['gripper']} | objects={len(r['bboxes'])} | "
f"bbox_ok={r['bbox_parse_ok']}")
except Exception as e:
print(f" ERROR: {e}")

print(f"\n✓ {len(results)} episodes processed.")
print(f" With objects > 0 : {sum(1 for r in results if len(r['bboxes']) > 0)}")
print(f" Bbox parse ok : {sum(1 for r in results if r['bbox_parse_ok'])}")

Outputs :


Inside describe_scene() function in W&B Weave Trace

Trace for generate_ecot_chain() on W&B Weave

Step 8: Bounding box drawing helper

In this step, we create a reusable rendering function that takes a PIL image and a list of bounding boxes and returns an annotated frame. Each object gets a distinct color, a labeled rectangle, and a text box. The motion primitive appears as a colored bar at the bottom of the image -
  • Red for move,
  • Green for gripper,
  • Orange for rotate
  • Blue for tilt
  • Grey for stop.
Why this matters: The visual output is what makes the pipeline tangible. Seeing bounding boxes drawn on real kitchen scenes with correct object labels and a motion label at the bottom is the clearest way to communicate what the pipeline produces. The color system for primitives also makes the trajectory strip and overview grid immediately readable at a glance. You can see at which point in an episode the gripper closes, or where the rotation happens, without reading the text labels.
BBOX_COLORS = [
"#FF4444", "#44BB44", "#4488FF", "#FFD700",
"#FF69B4", "#00CED1", "#FF8C00", "#9370DB",
]

def prim_color(prim: str) -> str:
if "move" in prim: return "#d32f2f"
if "gripper" in prim: return "#0f9d58"
if "rotate" in prim: return "#e65100"
if "tilt" in prim: return "#1565c0"
return "#37474f"

def draw_bboxes_on_image(pil_image, bboxes, motion_primitive=""):
if pil_image is None:
return Image.new("RGB", (256, 256), (60, 60, 60))
img = pil_image.copy().convert("RGB")
W, H = img.size
draw = ImageDraw.Draw(img)
for i, obj in enumerate(bboxes):
if "bbox" not in obj or "name" not in obj:
continue
color = BBOX_COLORS[i % len(BBOX_COLORS)]
try:
b = [float(v) for v in obj["bbox"]]
except (ValueError, TypeError):
continue
if all(v <= 1.15 for v in b):
x1, y1, x2, y2 = b[0]*W, b[1]*H, b[2]*W, b[3]*H
else:
x1, y1, x2, y2 = b
if x2 <= x1 or y2 <= y1:
continue
draw.rectangle([x1, y1, x2, y2], outline=color, width=2)
label = obj["name"][:22]
try:
tb = draw.textbbox((x1, max(0, y1 - 14)), label)
draw.rectangle(tb, fill=color)
draw.text((x1, max(0, y1 - 14)), label, fill="white")
except Exception:
pass
if motion_primitive:
bar_color = prim_color(motion_primitive)
draw.rectangle([0, H - 22, W, H], fill=bar_color)
draw.text((4, H - 20), f"▶ {motion_primitive}", fill="white")
return img

print("draw_bboxes_on_image defined.")




Step 9: Per-episode card

The idea is a three-panel figure showing the representative frame, the annotated frame with bounding boxes, and the generated ECoT chain with each field color-coded by its semantic role.
  • TASK in blue,
  • PLAN in green,
  • SUBTASK REASONING in purple,
  • SUBTASK in violet,
  • MOVE REASONING in orange,
  • MOVE in red,
  • GRIPPER POSITION in grey,
  • VISIBLE OBJECTS in dark grey.
Why this matters: A single episode card is the most concentrated visualization of what the pipeline produces. It lets you check scene description quality, bbox accuracy, primitive labeling, gripper position, and chain coherence all at once. Episodes are sorted by object count, so the best-detected examples appear first. These are the most useful for evaluating chain quality because the LLM has the most grounded input to work with.
subplots_adjust replaces tight_layout here to avoid the UserWarning that fires when matplotlib axes with differing aspect ratios are present in the same figure.
FIELD_COLORS = {
"TASK": "#1a73e8",
"PLAN": "#0f9d58",
"SUBTASK REASONING": "#7b1fa2",
"SUBTASK": "#9c27b0",
"MOVE REASONING": "#e65100",
"MOVE": "#d32f2f",
"GRIPPER POSITION": "#555555",
"VISIBLE OBJECTS": "#37474f",
}

def show_episode_card(r: dict):
# Use the representative frame image, not always frame 0
rep_img = get_frame_at_index(r["episode_id"], r["rep_frame_idx"])
if rep_img is None:
rep_img = get_keyframe(r["episode_id"])
if rep_img is None:
print(f" [skip] Ep {r['episode_id']}: no image available")
return

annotated = draw_bboxes_on_image(rep_img, r["bboxes"],
motion_primitive=r["motion_primitive"])

fig = plt.figure(figsize=(18, 5))
gs = gridspec.GridSpec(1, 3, figure=fig, wspace=0.35)

ax0 = fig.add_subplot(gs[0])
ax0.imshow(rep_img)
ax0.axis("off")
ax0.set_title(f"Frame {r['rep_frame_idx']} (representative)", fontsize=10,
fontweight="bold", pad=6)

ax1 = fig.add_subplot(gs[1])
ax1.imshow(annotated)
ax1.axis("off")
ax1.set_title("VLM detections + motion primitive", fontsize=10,
fontweight="bold", pad=6)

ax2 = fig.add_subplot(gs[2])
ax2.axis("off")
ax2.set_facecolor("#f8f9fa")
ax2.set_title("Generated ECoT chain", fontsize=10, fontweight="bold", pad=6)

y = 0.97
for line in r["ecot_chain"].split("\n"):
color = "#222222"
for field, fc in FIELD_COLORS.items():
if line.startswith(field + ":"):
color = fc
break
for wl in textwrap.fill(line, width=44).split("\n"):
ax2.text(0.01, y, wl, transform=ax2.transAxes,
fontsize=7.5, color=color, verticalalignment="top",
fontfamily="monospace")
y -= 0.072
if y < 0:
break

fig.suptitle(
f"Episode {r['episode_id']} — {r['task']}",
fontsize=12, fontweight="bold", y=1.02
)
fig.subplots_adjust(left=0.02, right=0.98, top=0.88, bottom=0.02, wspace=0.35)
plt.show()


best = sorted(results, key=lambda r: len(r["bboxes"]), reverse=True)
print("Top 5 episodes by object count:")
for r in best[:5]:
print(f" Ep {r['episode_id']:02d}: {len(r['bboxes'])} objects | frame {r['rep_frame_idx']} | {r['task'][:50]}")
show_episode_card(r)

Output:









Step 10: Episode trajectory strip

Here we're trying to show 10 frames sampled evenly across the full episode timeline with the motion primitive label under each frame. The representative frame used for chain generation is marked with a star in the title.
Why this matters: The per-episode card shows one frame. The trajectory strip shows how the robot actually moved, which is often more informative for understanding whether the pipeline produced a meaningful chain. You can see where in the episode the gripper closed, where the arm rotated, and whether the representative frame was chosen at a moment of actual manipulation or at a static holding pose. This visualization is not in the paper but is the most useful for debugging representative frame selection and for showing the temporal structure of Bridge V2 episodes to readers.
def show_trajectory_strip(ep_id: int, n_frames: int = 10):
all_frames = get_all_frames(ep_id)
if not all_frames:
print(f"[skip] Ep {ep_id}: no valid video frames")
return

ep_frames = valid_episodes[ep_id]
task = tasks.get(ep_frames[0].get("task_index", 0), "")
indices = np.linspace(0, len(ep_frames) - 1, n_frames, dtype=int)
frame_imgs = [all_frames[i] for i in indices if i < len(all_frames)]
rep_idx = find_representative_frame(ep_frames)

n = len(frame_imgs)
fig, axes = plt.subplots(1, n, figsize=(n * 2.4, 3.2))
if n == 1:
axes = [axes]

for ax, img, idx in zip(axes, frame_imgs, indices):
prim = compute_motion_primitive(ep_frames, int(idx))
color = prim_color(prim)
ax.imshow(img)
ax.axis("off")
# Mark the representative frame with a star
title = f"t={idx} ★" if int(idx) == rep_idx else f"t={idx}"
ax.set_title(title, fontsize=7, pad=2,
fontweight="bold" if int(idx) == rep_idx else "normal")
ax.set_xlabel(prim, fontsize=6.5, color=color, labelpad=3)

fig.suptitle(f"Ep {ep_id} — {task} (★ = representative frame used for chain)",
fontsize=9, fontweight="bold", y=1.04)
plt.tight_layout()
plt.show()

for ep_id in list(valid_episodes.keys())[:5]:
show_trajectory_strip(ep_id, n_frames=10)

Output :








Step 11: Motion primitive distribution

Next, we need a bar chart of motion primitive frequency across all processed episodes, with the paper's Appendix B numbers in the title for comparison. The chart validates that the rule-based classifier is producing realistic output - the distribution should be dominated by stop, rotation variants, and gripper operations, not by exotic multi-axis moves.
Appendix B of the referenced paper
Why this matters: The paper reports stop=26.9%, close gripper=10.8%, open gripper=7.2% across 2.5 million transitions. Our distribution from 30 episodes differs because the early Bridge V2 episodes skew toward cloth folding and appliance tasks with high rotation content, and because we use observation.state which produces richer compound primitives than the action vector would.
def show_primitive_distribution(results: list):
all_prims = []
for r in results:
ep_frames = valid_episodes[r["episode_id"]]
for i in range(len(ep_frames)):
all_prims.append(compute_motion_primitive(ep_frames, i))

counter = Counter(all_prims)
top = counter.most_common(15)
labels, counts = zip(*top)
total = sum(counts)
pcts = [c / total * 100 for c in counts]
colors = [prim_color(l) for l in labels]

fig, ax = plt.subplots(figsize=(13, 5))
bars = ax.barh(labels[::-1], pcts[::-1], color=colors[::-1], edgecolor="white")
ax.bar_label(bars, [f"{p:.1f}%" for p in pcts[::-1]], padding=4, fontsize=8)
ax.set_xlabel("% of total steps", fontsize=10)
ax.set_title(
f"Motion primitive distribution — {len(results)} episodes, {total} steps\n"
"(paper Appendix B: stop=26.9%, close gripper=10.8%, open gripper=7.2%)",
fontsize=10, fontweight="bold"
)
ax.set_facecolor("#f8f9fa")
plt.tight_layout()
plt.show()

complex_prims = sum(1 for p in all_prims if len(p.split()) >= 4)
print(f"Total steps : {total}")
print(f"Complex (4+ words) : {complex_prims} ({complex_prims/total*100:.1f}%) — confirms observation.state in use")
print(f"With tilt : {sum(1 for p in all_prims if 'tilt' in p)} ({sum('tilt' in p for p in all_prims)/total*100:.1f}%)")


show_primitive_distribution(results)

The chart also reports the percentage of complex (4+ word) primitives and the tilt frequency as confirmation that the observation.state fix is working correctly.

Step 12: Human correction diff

The idea is to reproduce the paper's Figure 6 human correction loop directly in the notebook, with a side-by-side diff showing the original chain on the left and the corrected chain on the right. Changed lines are highlighted red in the original and green in the corrected version.
Figure 6 from the paper addressing human correction loop
Why this matters in terms of paper alignment: The correction prompt is adapted directly from Figure 12 and Appendix C of the paper. It gives the LLM the original chain, the task, and a single sentence of human feedback, and asks it to produce a corrected chain in the same format. The paper shows this intervention lifts the hardest-task success rate by 48% points. The diff view makes the correction mechanism transparent, you can see exactly which fields changed and whether the model incorporated the feedback into the right part of the chain.
One important clarification about human correction: This mechanism is inference-time only, not a training step. A human watches the deployed robot, sees the reasoning chain go wrong mid-episode. For example:
1. The chain says the screwdriver is on the left when it is on the right and types a correction in natural language while the robot is running.
2. The corrected chain is fed back into the policy for the next few steps to steer it away from the failure.
3. It is not a fine-tuning operation and does not update any model weights.

The notebook hardcodes three example feedback strings purely to demonstrate the mechanism in a reproducible way. In a real deployment, the feedback would come from a human operator watching a live feed. The hardcoded strings are not meant to represent real corrections, they are placeholders that show what the input format looks like so you can understand the mechanism. A reader new to ECoT should think of this cell as a simulation of the correction loop, not the loop itself.
💡

Why this matters for W&B monitoring: apply_human_correction is decorated with @weave.op(). Every correction call logs the original chain, the feedback, and the corrected chain as one Weave trace.
While the correction itself is inference-time, the stored traces have longer-term value: a library of real operator interventions and their corrections is exactly the kind of signal you would use to improve the chain generation prompt, evaluate whether certain task types systematically produce bad chains, or build a fine-tuning dataset that makes future chains less likely to fail in the same ways.
The Weave logging is what makes those interventions usable rather than momentary.
# Faithfully adapted from Figure 12 / Appendix C of the paper
CORRECTION_PROMPT = """You are an expert reinforcement learning researcher. You've trained a policy for controlling a robotic arm.
The policy computes the correct action based on a reasoning that leads to it.

The reasoning must have the following elements:
TASK: the task that remains to be done.
PLAN: a list of high-level steps that need to be executed.
SUBTASK REASONING: reasoning that determines the current subtask.
SUBTASK: the current subtask that should be executed.
MOVE REASONING: reasoning that determines the current move.
MOVE: the current move that should be executed.
GRIPPER POSITION: [x, y]
VISIBLE OBJECTS: name [x1, y1, x2, y2], ...

Write the answer as a plain string in the exact format above. No markdown.

The policy generated the following reasoning:
{original_chain}

Given the task "{task}" and human feedback "{feedback}", what should be the corrected reasoning?
"""

# Weave logs feedback + original + corrected chain as one trace.
# In production this builds a library of real failure modes and fixes.
@weave.op()
def apply_human_correction(task: str, original_chain: str, feedback: str) -> str:
return call_text(CORRECTION_PROMPT.format(
original_chain=original_chain, task=task, feedback=feedback
))

def show_correction_diff(r: dict, feedback: str):
rep_img = get_frame_at_index(r["episode_id"], r["rep_frame_idx"])
if rep_img is None:
rep_img = get_keyframe(r["episode_id"])
if rep_img is None:
print(f" [skip] Ep {r['episode_id']}: no image available")
return

original = r["ecot_chain"]
corrected = apply_human_correction(
task=r["task"], original_chain=original, feedback=feedback
)

orig_lines = original.split("\n")
corr_lines = corrected.split("\n")
max_lines = max(len(orig_lines), len(corr_lines))
orig_lines += [""] * (max_lines - len(orig_lines))
corr_lines += [""] * (max_lines - len(corr_lines))

annotated = draw_bboxes_on_image(rep_img, r["bboxes"],
motion_primitive=r["motion_primitive"])

fig = plt.figure(figsize=(20, max(6, max_lines * 0.33 + 2)))
gs = gridspec.GridSpec(1, 4, figure=fig, wspace=0.3)

ax_img = fig.add_subplot(gs[0])
ax_img.imshow(annotated)
ax_img.axis("off")
ax_img.set_title("Scene", fontsize=9, fontweight="bold")

ax_orig = fig.add_subplot(gs[1:3])
ax_orig.axis("off")
ax_orig.set_facecolor("#fff8f8")
ax_orig.set_title("Original chain", fontsize=9, fontweight="bold", color="#cc0000")

ax_corr = fig.add_subplot(gs[3])
ax_corr.axis("off")
ax_corr.set_facecolor("#f8fff8")
ax_corr.set_title("Corrected chain", fontsize=9, fontweight="bold", color="#006600")

y = 0.97
step = min(0.065, 0.95 / max(max_lines, 1))

for ol, cl in zip(orig_lines, corr_lines):
changed = ol.strip() != cl.strip()
fc_o = "#cc0000" if changed else "#333333"
fc_c = "#006600" if changed else "#333333"
bg_o = dict(boxstyle="round,pad=0.1", fc="#ffe0e0", ec="none") if changed else None
bg_c = dict(boxstyle="round,pad=0.1", fc="#e0ffe0", ec="none") if changed else None

kw_o = dict(transform=ax_orig.transAxes, fontsize=7, color=fc_o,
verticalalignment="top", fontfamily="monospace")
if bg_o: kw_o["bbox"] = bg_o
ax_orig.text(0.01, y, textwrap.fill(ol, 56)[:56], **kw_o)

kw_c = dict(transform=ax_corr.transAxes, fontsize=7, color=fc_c,
verticalalignment="top", fontfamily="monospace")
if bg_c: kw_c["bbox"] = bg_c
ax_corr.text(0.01, y, textwrap.fill(cl, 28)[:28], **kw_c)
y -= step

fig.suptitle(
f"Human correction — Ep {r['episode_id']}: {r['task']}\n"
f'Feedback: "{feedback}"',
fontsize=10, fontweight="bold", y=1.01
)
fig.subplots_adjust(left=0.02, right=0.98, top=0.88, bottom=0.02, wspace=0.3)
plt.show()

feedbacks = [
"the target object is on the right side, not the left — adjust the plan",
"release the object now, the target container is directly below the gripper",
"you are too high, move the arm down before attempting to grasp",
]
best_3 = sorted(results, key=lambda r: len(r["bboxes"]), reverse=True)[:3]
for i, r in enumerate(best_3):
print(f"\nEp {r['episode_id']}: {r['task']}")
show_correction_diff(r, feedbacks[i % len(feedbacks)])
time.sleep(2)

Output :





Weave Output :


Step 13: Overview grid

In this step, we want a complete before/after grid showing every valid episode, sorted by object count so the best-detected examples appear first. Each episode occupies two cells:
  • The representative frame on the left and
  • The annotated frame on the right, with the motion primitive and object count in the title.
Why this matters: The grid is the deliverable. It shows 30 episodes at once and makes the full output of the pipeline reviewable in one figure. Sorting by object count means the top rows show the clearest detections and richest annotations, while the bottom rows show the episodes where the VLM returned zero bounding boxes, which is immediately obvious and useful for diagnosing detection failures at scale. The grid is also saved as a PNG and logged to W&B so it is linked to the run and available for sharing.
def show_overview_grid(results: list, cols: int = 5):
sorted_results = sorted(results, key=lambda r: len(r["bboxes"]), reverse=True)
n = len(sorted_results)
rows = math.ceil(n / cols)

fig, axes = plt.subplots(rows, cols * 2, figsize=(cols * 4.8, rows * 3.4))
if rows == 1:
axes = [axes]

fig.suptitle(
f"Bridge V2 — ECoT pipeline · {n} valid episodes\n"
"Left: representative frame | Right: VLM bounding boxes + motion primitive",
fontsize=11, fontweight="bold", y=1.01
)

for idx, r in enumerate(sorted_results):
row, col = divmod(idx, cols)
ax_l = axes[row][col * 2]
ax_r = axes[row][col * 2 + 1]

raw = get_frame_at_index(r["episode_id"], r["rep_frame_idx"]) or get_keyframe(r["episode_id"])
if raw is None:
ax_l.axis("off")
ax_r.axis("off")
continue

annotated = draw_bboxes_on_image(raw, r["bboxes"],
motion_primitive=r["motion_primitive"])
ax_l.imshow(raw)
ax_l.axis("off")
ax_l.set_title(f"Ep {r['episode_id']} f{r['rep_frame_idx']}", fontsize=6.5, pad=2)
ax_l.set_xlabel(r["task"][:28], fontsize=5.5, color="#333", labelpad=2)

ax_r.imshow(annotated)
ax_r.axis("off")
ax_r.set_title(
f"{r['motion_primitive'][:30]} ({len(r['bboxes'])} obj)",
fontsize=5.8, color=prim_color(r["motion_primitive"]), pad=2
)

for idx in range(n, rows * cols):
row, col = divmod(idx, cols)
axes[row][col * 2].axis("off")
axes[row][col * 2 + 1].axis("off")

plt.tight_layout()
plt.savefig("ecot_grid.png", dpi=150, bbox_inches="tight")
plt.show()
print(f"Saved ecot_grid.png ({n} episodes)")

show_overview_grid(results)





Step 14: Export training dataset and log to W&B

Next, we want to serialize the pipeline output as a JSONL file and log everything -> the training data, the overview grid, and a results table with raw and annotated images to W&B as a versioned artifact.
Each JSONL record is an (observation, reasoning chain, action) tuple: the training format used to fine-tune OpenVLA in the paper. The paper produced 2.5 million such records from the full Bridge V2 corpus over seven days. Our 30-episode run produces around 30 records, which is sufficient to demonstrate the format and validate the chain structure.
The W&B Artifact versions the JSONL file with metadata including the vision model, text model, total episodes, and number of episodes with successful bbox detection. This means any future training run can be linked back to the exact dataset version that produced it. The results table includes bbox_parse_ok and rep_frame_idx columns so you can filter by detection success in the W&B interface, the most useful first cut when comparing chain quality across episodes.
W&B Weave connects this export back to the traces from Steps 7 and 12. If a fine-tuned model underperforms on a specific task category, you can filter the Weave trace table by task, identify the chains that were used to train those tasks, and read the raw VLM responses that fed into them. The failure is diagnosable without re-running the pipeline.
output_path = "ecot_training_data.jsonl"

with open(output_path, "w") as f:
for r in results:
f.write(json.dumps({
"episode_id": r["episode_id"],
"instruction": r["task"],
"rep_frame_idx": r["rep_frame_idx"],
"scene_description": r["scene_description"],
"motion_primitive": r["motion_primitive"],
"gripper_state": r["gripper_state"],
"bboxes": r["bboxes"],
"ecot_chain": r["ecot_chain"],
}) + "\n")
print(f"Saved {len(results)} records → {output_path}")

run = wandb.init(project="ecot-pipeline-demo", job_type="pipeline-output")

artifact = wandb.Artifact(
name="ecot-training-chains",
type="dataset",
description="Synthetic ECoT chains — Bridge V2, Zawalski et al. 2025",
metadata={
"total_episodes": len(results),
"episodes_with_objects": sum(1 for r in results if len(r["bboxes"]) > 0),
"bbox_parse_ok": sum(1 for r in results if r["bbox_parse_ok"]),
"vision_model": VISION_MODEL,
"text_model": TEXT_MODEL,
}
)
artifact.add_file(output_path)
run.log_artifact(artifact)

run.log({"ecot_grid": wandb.Image("ecot_grid.png",
caption=f"Before/after · {len(results)} episodes")})

table = wandb.Table(columns=[
"episode", "task", "rep_frame", "raw_frame", "annotated_frame",
"motion_primitive", "n_objects", "bbox_parse_ok", "gripper_2d", "ecot_chain"
])
for r in results:
raw = get_frame_at_index(r["episode_id"], r["rep_frame_idx"]) or get_keyframe(r["episode_id"])
if raw is None:
continue
annotated = draw_bboxes_on_image(raw, r["bboxes"], motion_primitive=r["motion_primitive"])
table.add_data(
r["episode_id"],
r["task"],
r["rep_frame_idx"],
wandb.Image(raw),
wandb.Image(annotated),
r["motion_primitive"],
len(r["bboxes"]),
r["bbox_parse_ok"],
str(r["gripper_state"].get("position_2d", "unknown")),
r["ecot_chain"],
)
run.log({"ecot_results": table})
run.finish()
print("Everything logged to W&B.")


Weave Traces Log :

Part 1

Part 2
ECoT's core contribution is grounded, step-by-step reasoning that makes VLA models more capable in novel situations and more transparent when they fail. The reasoning chain connects visual observations and robot state to language in a way that standard chain-of-thought prompting does not, and that connection drives the 28% performance gain and the benefits of human correction.
The most important threads to follow from here are :
  1. OpenVLA as the base model architecture,
  2. Bridge V2 as the large-scale demonstration dataset,
  3. the synthetic data pipeline as the mechanism that makes the approach scale without hand-labeling, and
  4. The human correction loop is what sets ECoT apart from standard chain-of-thought reasoning in language models.
W&B Weave ties all four together, making every API call in the pipeline traceable, every detection failure diagnosable, every training run linked to the data version that produced it, and every production correction logged and usable.

Iterate on AI agents and models faster. Try Weights & Biases today.