Agent Trace Quality: 4 Common Failure Modes

Coding agents fail in places that look fine on a single-step benchmark but break in multi-step real work. A model that solves 80 percent of SWE-bench Verified can still flounder on a real engineering task that requires reading documentation, running tests, hypothesizing, and recovering from an error. The reason isn't pretraining scale. It's that the model has rarely seen high-quality examples of what doing real multi-step engineering work actually looks like.

That gap is what agent trajectory data is supposed to close. But most of the trajectory data on the market - and a significant fraction of what we see vendors deliver - doesn't close it, because the data has structural problems that aren't visible until the model trained on it underperforms in production. This essay walks through what a high-quality trace contains, why most public trajectory data is low-signal, and the four specific failure modes we reject before any batch leaves our agent trajectory data service.

What we mean by a "trace"

A trace, in our usage, is a multi-step record of an engineer completing a real task in a real environment. It contains six components, each layered together so the resulting record can be replayed, verified, and used to train a model that needs to operate in similar contexts.

The components are: a screen recording with synchronized click and keystroke events; a terminal session log with every command, its output, and exit code; browser activity with URLs, page content, and form interactions captured; IDE state including file diffs, test runs, and lint output at each step; reasoning verbalization captured live as first-person audio or text describing what the engineer is trying, why, and how they decide what to try next; and an outcome tag covering task completion status, whether success criteria were met, time-to-completion, and any blockers encountered.

A trace with all six components in good shape is the highest-signal training data for agent capabilities you can produce. The signal isn't in any one component - it's in the alignment between them. A model learns from watching reasoning lead to action, action produce environment state, and environment state inform the next reasoning step. Pull any one layer out and the signal collapses.

Most public datasets pull out at least two.

Why most public agent-trace data is low-signal

The bulk of trajectory data in open datasets and on vendor marketplaces is some version of: a final-state record (the diff that solved the task), reconstructed from GitHub PRs or similar artifacts, with a synthetic verbalization layer generated after the fact. These datasets look like trajectories on the surface, but they are missing the parts that actually carry training signal.

Specifically: real engineering work is full of dead ends, hypothesis revisions, error recovery, and intermediate states that don't make it into the final PR. A model trained on PR-derived "trajectories" sees engineering as a sequence of correct moves, because that's what survived into the PR. It does not learn what to do when it's stuck, which is the part that matters in production.

The fix is to capture trajectories live, with the dead ends preserved. That's expensive and operationally heavy. It's also what separates training data that improves agent capability from training data that just inflates dataset volume.

The next sections describe the four most common ways traces fail our QA process before delivery, and why each one matters.

Failure mode 1 - Verbalization gaps

A verbalization gap is a stretch of trace time - typically 30 seconds or longer - where the engineer is doing active work but not producing any reasoning trace. The terminal is running, the code is changing, the browser is loading, but there's no record of what the engineer is thinking.

This is the single most common failure mode in trajectory data. It happens because verbalization is cognitively expensive - talking through your reasoning while you work is harder than just working - and engineers slip out of it under pressure or fatigue. The result is a trace where the action layer is rich and the reasoning layer is sparse, and the parts where the reasoning was most valuable (when the engineer hit something unexpected and figured out how to handle it) are exactly the parts most likely to be silent.

We flag any trace where verbalization coverage drops below a threshold and either send it back to the engineer for re-capture or, in some cases, reject it entirely. The threshold varies by task family; for debugging tasks where reasoning is the central deliverable, the bar is higher than for routine implementation tasks where most decisions are pattern-matched.

The annotation rubric we use specifies what counts as a meaningful decision (one that an alternative-seeking engineer might have made differently), and we require verbalization on each one. This is one of the places where the 5-stage vetting process for engineers matters most - the engineers who pass our vetting are selected partly on their ability to verbalize under load.

Failure mode 2 - Synthetic-looking environments

A trace can be technically complete - all six layers present, no verbalization gaps - and still produce a model that fails on real work, because the environment the trace was captured in doesn't look like the environment the model will be deployed in.

The most common version of this is the "toy codebase": a single-file repository with one test file, no historical commits, no dependencies on real internal libraries, no actual config or CI infrastructure. Tasks captured in toy codebases produce traces that teach the model how to solve simplified engineering puzzles. They do not teach the model what to do in a 50,000-line codebase with five years of history, three custom internal frameworks, and an opinionated lint configuration.

Real engineering involves a lot of orientation work - reading existing code to understand conventions, tracing dependencies through unfamiliar internal libraries, figuring out what the team's testing posture is before adding a test. None of that exists in a toy codebase. A model trained only on toy-codebase traces will skip orientation entirely in real work, and its outputs will look out of place.

Our QA flags traces where the environment lacks one or more of: realistic codebase scale, dependency depth, internal-framework usage, and CI/test infrastructure. The fix isn't always to reject the trace - sometimes the task genuinely was a small one - but for any client training a coding agent intended to work in real codebases, the dataset composition needs to skew heavily toward realistic environments. We track this composition explicitly per engagement.

Failure mode 3 - Outcome falsification

This one is the worst of the four, because it corrupts the most valuable part of the trace.

Outcome falsification happens when a trace is marked "success," but the success criteria weren't actually met. The most common version: the engineer's tests pass at the end of the trace, but the tests don't actually validate the task. They might be testing the wrong condition, or testing trivial behavior, or have been modified during the trace in ways that made them pass without solving the underlying problem. The trace looks like a successful task completion. It is not.

A second version: the engineer reaches a state that subjectively feels like task completion, but doesn't meet the original task spec. The model trained on this data learns to declare victory in the wrong places.

Our outcome verification process replays every "success"-tagged trace in a fresh environment and re-runs the validation criteria. If the replay doesn't reproduce the success - because the trace depended on environment state, hidden file modifications, or tests that don't actually validate the task - the trace is flagged. We either re-tag it with the correct outcome, send it back for correction, or remove it from the batch.

This is operationally heavy. It also catches roughly 5 to 10 percent of "success"-tagged traces across most engagements, and the catch rate is consistently higher for vendors who don't do this verification. If the model your competitor trained reaches a state of false confidence faster than it should, this is one of the places that failure was introduced.

Failure mode 4 - Single-tool tunnels

A single-tool tunnel is a trace where the engineer used one tool - say, the IDE - for the entire task, despite the task realistically requiring multiple tools. A real bug-fix trace often involves reading the IDE, checking logs in the terminal, looking up library docs in the browser, running tests, and possibly checking external systems. A trace where the engineer never leaves the IDE is either a contrived task or an over-constrained capture.

This matters because tool-using behavior is one of the capabilities labs are most actively training, and the training signal for tool use is exactly the parts of a trace where the engineer switches tools. A dataset of single-tool tunnels teaches the model to stay in one tool, which is the opposite of what you want from an agent.

Our QA flags traces with low tool diversity for the task family. Some task families legitimately are single-tool (writing a single isolated function, perhaps). Most are not. For multi-step engineering work, traces with no terminal output, no browser activity, or no test runs are usually evidence of either a contrived task or an engineer working under capture friction that pushed them to stay in one tool. Either way, the data is lower-signal than the headline numbers suggest, and we surface that to the client rather than ship the batch unflagged.

Our QA process for traces

Pulling these together: every trace passes through four QA gates before delivery.

The first gate is completeness: every layer present, every step accounted for, timestamps aligned across layers, no missing artifacts. Traces that fail completeness either get rebuilt or removed from the batch.

The second gate is verbalization coverage: spoken or written reasoning present at every meaningful decision point, with no extended gaps on cognitively dense sections. Threshold varies by task family.

The third gate is outcome verification: every "success" trace independently replayed against the success criteria. False positives are corrected or removed.

The fourth gate is senior calibrator sign-off: a senior engineer reviews a sample of the batch, examines the environment for realism and tool diversity, and signs off before delivery to the client.

Traces that pass all four gates get delivered. The remainder are returned to the engineer with specific feedback, recaptured, or removed from the batch. We report flag rates per batch to the client so they have visibility into where the quality bar is landing in real time. Some clients want tighter gates over time; some are comfortable with the standard configuration. We adjust per engagement.

How we structure trace batches

A standard trace engagement runs 50 to 200 traces per week per pod, scoped to specific task families and environment profiles defined with the client during the calibration session. We deliver in structured JSON with bundled artifacts (screen recording, terminal log, browser captures, and so on) using a schema that's version-controlled and adapted to the client's internal trace format where required.

Where trace work fits in a broader engagement: it pairs well with code RLHF task types - particularly bug-fix demonstrations and refactoring trajectories - because the trace captures the multi-step structure that single-step tasks can't. For agent training programs, trace work is usually the highest-margin, highest-friction part of the data pipeline, and it justifies its cost when the model needs to operate autonomously in real environments.

For pricing and pod sizing on trace engagements, the indicative tiers are on our pricing and pod structures page. Briefly: trace work sits at the top tier because of capture overhead, verification cost, and the senior calibrator time required per batch. A typical first engagement is a six to eight week pilot pod producing 200 to 500 traces against a defined task family, with full QA reporting and a delivery schema co-designed with the client.

See a sample trace

The fastest way to evaluate whether our traces would improve your training pipeline is to look at one. If you tell us the task family you're training on, we'll capture and deliver a single fully-annotated trace at no cost within seven business days. From there, scoping a pilot becomes a much more concrete conversation.

You can request a sample on the agent traces service page, or send a task spec directly, and we'll respond within 48 hours.