Most of your eval isn't a judgment call
The default way to evaluate an LLM system is to ask another LLM whether the answer was good. It's fast to build, it produces a number, and the number moves when the system changes. It is also, for most of what you actually want to measure, the wrong tool.
Take an assistant that answers questions about orders. Did it call the right tools? That's not an opinion — the call trace is in the logs, and expected tools are known, so it's precision, recall, and F1 over a set. Did it report the correct payment method and fulfillment status? Also not an opinion: normalize the types and compare exactly. Did it invent an order number or a dollar amount that appears nowhere in the retrieved payload? Containment check. Three dimensions, zero judgment, and every result reproducible on a machine with no model access at all.
What's left is the free-text explanation, where "is this a good answer" genuinely requires reading. That one gets a judge. One dimension out of six.
The reason this matters isn't cost, though it is cheaper. It's that a deterministic scorer fails for exactly one reason, and you can read the diff. When a judge-scored suite drops four points, you're now debugging two systems and you can't tell which one moved. Every dimension you can make mechanical is a dimension that will never lie to you about why it changed.
The corollary is that the judge you do keep needs its own evidence. I calibrate it against the deterministic scorers on items where ground truth is known and report its agreement rate next to its scores. A judge that agrees with mechanical truth 93% of the time is a tool. A judge nobody has ever checked is a vibe with a decimal point.
Benchmarks rot, and it looks exactly like model drift
Hand-labeled golden answers have an obvious cost — someone has to write them — and a much worse hidden one: they were true when they were written. If your benchmark asks about real orders in real systems, the world underneath keeps moving. Fulfillment status changes. A shipment completes. The expected answer quietly becomes wrong, the eval starts failing, and the failure is indistinguishable from a model regression.
Two things fixed this for us. First, derive golden answers instead of labeling them. If the seed order was created with known parameters, the correct answer is a function of those parameters, not an annotation. Ground truth becomes reproducible rather than an ongoing headcount cost, and regenerating the dataset stops being a project.
Second, classify every field as immutable or volatile. Immutable fields — order number, storefront, payment method as recorded — should never change; if one does, that's a data defect and I want it raised as a defect, loudly. Volatile fields — status, timestamps, anything the pipeline advances — get re-derived from the services before each run, so the expected value is current by construction.
The distinction is the whole trick. Without it you have one bucket labeled "the expected value doesn't match" and no way to tell a broken model from a moved world. With it, one case updates itself silently and the other files a bug.
The general lesson: an eval dataset is a system with its own failure modes, not a fixture you write once. If nobody owns its staleness, it will quietly start reporting on itself instead of on the model.
A gate everyone re-runs isn't a gate
We started with the strict version: any dimension below threshold fails the build. It read as rigorous. What it actually taught the team was that a red run means hit re-run, because LLM systems are non-deterministic and a single sample of a borderline item flips often enough to make failure feel like weather.
Two changes fixed the credibility problem. The first was scoring honestly under non-determinism: five samples per item, pass-rate instead of pass/fail, and a consistency metric tracked per item. An item that passes three times out of five isn't a failure, it's an unstable item, and it should be reported as instability rather than laundered into a regression number. Isolating those meant eval flakiness stopped being attributed to the model.
The second was tiering the gate to match consequence. Hallucination rate, tool-selection F1, and accuracy fail the build outright — those are the ways this system hurts someone who trusts it. Completeness and consistency warn. Deltas for every dimension get posted to the pull request whether the run passed or failed, which turned out to matter more than the gate itself: people started reading the numbers on green runs, which is the only way anyone notices a slow slide.
The payoff showed up as a regression that a single blended score would have hidden completely. A model version landed where overall accuracy went up while tool-selection F1 fell on multi-tool questions. The assistant was skipping a call and answering confidently from stale data — worse in the way that matters, better on the headline metric. Per-dimension reporting caught it; the fix was clarifying tool staleness semantics in the tool definitions.
A quality gate is a social artifact as much as a technical one. If it fires for reasons people learn to dismiss, it stops being a gate no matter what the pipeline config says.