IAA Is the Wrong Quality Metric for Code

Inter-Annotator Agreement Is the Wrong Quality Metric for Code Data

If you're measuring the quality of your code RLHF dataset using inter-annotator agreement, your dashboard is lying to you. Not because IAA is a bad metric in general - it isn't - but because the assumptions it was built on don't hold for code. We've seen batches with Cohen's kappa above 0.85 where, on careful expert review, more than thirty percent of the items were wrong in ways that meaningfully degraded the downstream model.

This essay walks through why that happens, what we measure instead at our code RLHF and annotation service, and when IAA still belongs in your QA stack. Written for the data scientist or quality lead who is responsible for whether a training batch is good enough to ship to a model.

What IAA was designed to measure

Inter-annotator agreement is a family of metrics - Cohen's kappa, Krippendorff's alpha, Fleiss' kappa, and a few others - that quantify how often independent annotators produce the same label for the same item, adjusted for chance agreement.

Cohen introduced his coefficient in 1960 for nominal classification tasks: discrete categories, mutually exclusive labels, a single correct answer per item. The math is elegant, and the metric has held up well in the categories it was built for. Text classification (positive vs negative sentiment), medical coding (which ICD-10 category applies), and structured information extraction all map onto Cohen's model cleanly.

Two annotators look at the same email. They label it as "spam" or "not spam." Either they agree, or they don't. IAA tells you how reliably your annotators are reproducing the same judgment.

For these tasks, IAA is the right metric. It is principled, statistically defensible, and well-understood. We use it ourselves on classification work.

Why it breaks on code

Code is not a nominal classification task. Three structural properties of code annotation work make IAA misleading rather than useful.

Multiple correct answers exist for most tasks. Ask two senior engineers to fix a memory leak in a piece of C++. One refactors the ownership model; the other adds a smart pointer. Both fix the leak. Both are correct. They disagree completely. IAA registers as low agreement. The reality is that both annotations are valid training signals for a code model that needs to know either approach is acceptable.

This is not a marginal case. For code generation tasks, Artstein and Poesio (2008) noted in their authoritative review that IAA presupposes a single ground truth and is inappropriate for tasks where multiple outputs can be correct. Code RLHF is exactly such a task.

Partial correctness is the norm, not the exception. A code review annotation that catches five out of seven defects in a piece of code is not equivalent to one that catches zero. IAA treats them as a mismatch - same as if one reviewer scored everything as correct and the other found nothing. This collapses a graded signal into a binary one and discards the most useful information in the dataset.

Reasoning quality is what the model actually learns from, and IAA does not measure it. For RLHF and especially for DPO-style training, the rationale attached to a label matters as much as the label itself. Two annotators can mark the same code as "high quality" - agreement perfect, IAA looks excellent - but one explains why with three sentences of substantive technical reasoning and the other writes "looks good." The first is training data. The second is noise. IAA cannot tell them apart.

What this looks like in practice

A recent engagement we calibrated against had the following metrics on its first batch under their existing vendor:

The numerator and the denominator were measuring different things. IAA was high because the reviewers had converged on a shared but flawed understanding of the rubric. The gold set, constructed independently by the client's senior engineers, exposed the gap. The dataset went into training, and the model showed exactly the failure modes the gold set predicted: confident-sounding but subtly incorrect code, especially in edge cases.

This is the failure mode IAA does not detect. The metric is measuring reviewer-to-reviewer consistency. It is not measuring reviewer-to-truth alignment. When the reviewers have all been calibrated the same wrong way, IAA looks excellent right up until the model trained on the data underperforms.

What we measure instead

For every code data engagement, we run four quality signals in parallel. IAA is one of them - but the smallest weighted of the four.

Gold-set acceptance is the primary signal. We co-construct a gold set with the client at the start of every engagement - typically 50 to 200 reference items with senior-engineer-verified labels and rationales. Five to ten percent of every batch is blind-sampled against the gold set. Acceptance is binary at the item level (does this annotation match the reference, including the rationale?), and we report acceptance rate per batch, per reviewer, and per task type.

Gold-set acceptance is the only metric that actually measures reviewer-to-truth alignment. It is also the most expensive metric to maintain, because the gold set has to be refreshed and recalibrated every quarter as the rubric evolves. Vendors who don't invest in gold-set maintenance default to IAA because it's cheap to compute, and that's where the silent quality erosion starts.

Reasoning quality score is the second signal. For every annotation that includes a rationale (which is most of them in modern code RLHF), a senior calibrator scores the rationale on a four-point scale: comprehensive, adequate, thin, or absent. We track the distribution across batches and across reviewers. Adequate is the floor; anything below adequate is sent back for re-annotation. Comprehensive is the goal for items in preference pairs and reasoning verbalizations, where the rationale is half the deliverable.

Debug-replay verification is the third signal, and it's specific to code. For bug-fix demonstrations and refactoring trajectories, we don't just check the output - we replay the work in a fresh environment to verify the fix actually works. About 5 to 10 percent of "complete" annotations don't survive this check, usually because they depend on environment state the reviewer didn't realize they were carrying. Catching this before delivery is the difference between training data and noise.

Inter-annotator agreement is the fourth signal, and we use it for one purpose only: detecting drift over time within a reviewer cohort. If an individual reviewer's agreement with the cohort starts trending downward across batches, that's a signal to recalibrate them. The absolute value of IAA tells us very little. The trajectory of IAA tells us a lot.

How this maps to your QA stack

If you're sitting on the buyer side and evaluating vendors, here is how to read a vendor's quality dashboard.

A vendor that reports only IAA is measuring reviewer-to-reviewer consistency and calling it quality. They will look excellent on the dashboard right up until your model trained on their data underperforms.

A vendor that reports gold-set acceptance, reasoning quality, and IAA together is measuring what actually matters. Ask them how the gold set was constructed, how often it's refreshed, and how they handle the cases where their batch acceptance disagrees with your gold set.

A vendor that can show you item-level audit trails - the labels, the rationales, the gold-set decisions, and the calibrator notes - is operating at the level of quality you need for frontier model training.

This is also the reason vendor vetting depth matters. The whole quality system depends on reviewers who can produce signal-rich rationales and senior calibrators who can construct meaningful gold sets. Without those, no QA metric stack saves you. With them, you can use simpler metrics and still ship data that lifts the model. Our 5-stage vetting process is the operational foundation this quality picture rests on.

When IAA still belongs

I don't want to overstate the case. IAA is a perfectly good metric for a specific class of work. We use it ourselves for classification-style code tasks where the labels are genuinely categorical (for example, "does this code contain a SQL injection vulnerability: yes/no") - IAA handles these cleanly. We use it for annotation drift detection within a stable reviewer cohort, where the trajectory matters even when the absolute value doesn't. And we use it for onboarding new reviewers, where their agreement with the established cohort is a useful early signal before they're producing meaningful gold-set acceptance numbers.

What we don't do is use IAA as the primary quality signal for any task with multiple correct answers, partial correctness, or rationale-dependent training signal. For modern code RLHF, that's almost everything.

The cost of running it right

Multi-signal quality measurement is expensive. Gold-set construction takes a senior engineer four to ten hours per 100 items. Quarterly recalibration sessions take another two to four hours each. Reasoning quality scoring adds 15 to 25 percent to the time per item on every batch. Debug-replay verification adds 5 to 10 percent more.

Add it up, and the per-task cost of a properly measured engagement runs 30 to 50 percent higher than an IAA-only engagement of the same volume. The data that comes out is also 20 to 40 percent more likely to actually move your model's eval numbers, in our experience and from the comparative studies clients have shared with us.

If you're scoping a vendor and the per-task rate looks great, but the quality stack is thin, you are paying for the cheaper engagement twice - once in the contract, once in the training run that fails to move the metric you care about.

Where to go from here

If you're evaluating vendors and want to see how a multi-signal QA stack works on a real task spec, we offer a no-cost sample on every prospective engagement. Tell us the task type and your current quality definition, and we'll deliver 10 to 20 annotated items along with the full QA breakdown - gold-set acceptance, reasoning quality, IAA, and any drift signals. You can see directly whether our quality picture matches your downstream model lift.

For more on how we structure the engineering side of this, the taxonomy of code RLHF task types and how we structure pods of different sizes are the natural follow-on reading.