AI evals need fail-case budgets

The fastest way to make an AI product look ready is to put one large quality score on a dashboard and color it green. The fastest way to make an AI product actually safer to ship is different: budget for the failures that would break user trust, test those failures explicitly, and block release until the workflow survives them.

That is the thesis: AI product evals need fail-case budgets, not score dashboards.

A score can be useful after the team knows what it is measuring. Before that, it is often release theater. It compresses many product risks into one reassuring number and hides the question that matters most before launch: have we seen enough real failures to know what this system does under pressure?

Eugene Yan’s practical write-up on product evals makes the operating pattern concrete: label production-like samples, align one evaluator per quality dimension, and run the eval harness after each prompt, model, retrieval, or configuration change. He also argues for focusing on fail cases because those are the trust-busting defects that determine whether a product is safe to expose to users. Product Evals in Three Simple Steps is useful because it treats evals as product infrastructure, not as a slide for stakeholder confidence.

The score is not the release gate

A launch review that says “quality is 91 percent” sounds mature. It may still be empty. Ninety-one percent of what? From which users? Under which task mix? With how many examples of the failures that would make a customer stop trusting the feature?

Aggregate scores hide the denominator. A team can evaluate hundreds of easy samples, find only a handful of serious defects, and conclude that the product is strong. But if the sample contains only five meaningful failures, the team has not learned much about how the system behaves when it is wrong. It has learned that the eval set is comfortable.

This matters most for LLM features because the expensive failures are rarely average failures. They are the cases where a summary invents a fact, a support bot gives confident but inapplicable policy advice, a retrieval system cites the wrong document, or an agent performs the right action on the wrong account. A green score does not tell you whether those classes were present in the test set.

The release gate should therefore move from “is the dashboard green?” to “does every trust-busting failure class have enough real coverage, a clear threshold, and an owner who can explain the remaining risk?” That shift is close to the governance logic in AI product milestones need demo-to-product gates: the artifact that works in a demo is not the artifact that deserves production traffic.

What belongs in a fail-case budget?

A fail-case budget is a minimum commitment to learn from the failures that matter before release. It is not a quota for making the model look bad. It is a product control that forces the team to test the cases that would cause support tickets, compliance reviews, customer churn, or internal loss of confidence.

Start with the top five trust-busting failure classes for the workflow. For a RAG answer feature, they might be unsupported claims, wrong source attribution, missed refusal, stale information, and answer drift outside the allowed domain. For a sales-assistant workflow, they might be fabricated account details, unsafe personalization, bad prioritization, CRM write errors, and tone violations. The list should come from the product surface, not from a generic benchmark.

Then define the budget for each class. The budget should include four parts: the minimum number of real or production-like fail cases, the label definition, the evaluator responsible for detecting it, and the release threshold. “Faithfulness: at least 60 known fail cases, binary pass or fail label, faithfulness evaluator, ship only if fail recall stays above the agreed threshold” is far more useful than “quality above 90 percent.”

The most important word is “real.” Synthetic defects can help bootstrap a first eval set, but they can also be too theatrical or too subtle compared with the messy failures users actually encounter. Yan recommends adding organic samples from production as a priority when the dataset starts with synthetic examples. That advice should become a standing backlog item: every incident, escalation, confusing user query, and manual review miss should feed the eval set.

This is also where model selection becomes practical. You cannot choose a model responsibly with a single leaderboard number. You need a task portfolio, as argued in Model selection needs task portfolios, and each task needs enough failure coverage to expose the trade-offs. A cheaper model that passes generic tasks but fails long-context edge cases may be fine for one workflow and dangerous for another.

Where does evaluator alignment break?

Evaluator alignment breaks when the team asks one evaluator to judge everything. A single “overall quality” judge is tempting because it creates one number. It also makes debugging almost impossible. If the score falls, did the model become less faithful, less relevant, less concise, less safe, or simply less aligned with the judge prompt?

A stronger pattern is one evaluator per dimension. Faithfulness gets one evaluator. Relevance gets another. Refusal behavior gets another. Tone gets another only if tone is actually a product requirement. The team can then decide which dimensions are ship blockers and which are improvement metrics. A faithfulness failure in a medical or financial support workflow may block release. A minor concision miss may not.

Alignment also needs a held-out test set. If the team iterates on evaluator prompts using all labeled samples, the evaluator can overfit to the examples. It may look accurate in development and then miss new failures in production. Splitting labeled data into an alignment set and a test set is not academic ceremony. It protects the release gate from self-deception.

The human process matters too. If annotators disagree on labels, the evaluator will inherit that ambiguity. Binary labels help because they force a clear product decision: acceptable or not acceptable, win or lose, allowed or blocked. Numeric scales look precise, but teams often struggle to explain the difference between a 3 and a 4. When launch risk is high, ambiguity should be resolved in the rubric, not hidden inside a decimal.

Rerun gates after every product change

The moment an LLM workflow changes, yesterday’s green dashboard becomes historical evidence. A prompt edit can change tone and refusal behavior. A retrieval parameter can improve recall while worsening source quality. A model migration can reduce cost while creating new edge cases. A tool permission change can turn a harmless answer defect into an operational incident.

The eval harness has to run after each meaningful configuration change. That includes prompt templates, model choice, model parameters, retrieval settings, chunking strategy, ranking logic, tool schemas, and policy text. If the team cannot afford to rerun the evals, the eval process is too heavy for the product cadence.

This is why fail-case budgets should be small enough to run often and serious enough to block release. They are not a research benchmark. They are a product control loop. A good harness lets the team compare the current configuration with the candidate configuration, inspect regressions by failure class, and decide whether the change is safe. The question is not “did the score improve?” The question is “which user trust risks got better, which got worse, and are any of them above our release tolerance?”

This same discipline applies beyond evals. AI coding needs acceptance criteria first makes the parallel point for generated code: the system is only governable when the acceptance condition is explicit before output arrives. AI product evals need the same inversion. Decide what must not fail before the model produces a charming demo.

Audit one workflow before the next launch

Pick one AI workflow that is close to launch or already in production. Do not start by asking whether the overall score is high enough. Start with five questions.

Which five failure classes would most damage user trust? How many real examples of each class are in the eval set? Who labeled them, and against which rubric? Which evaluator detects each class, and how was that evaluator tested against held-out labels? What changes force a rerun before release?

If the team cannot answer those questions, the product may still ship, but it is not being released through an eval gate. It is being released through hope with analytics attached.

A fail-case budget slows the conversation at exactly the right moment. It makes the team name the risks, collect the evidence, align the judges, and rerun the gate when the system changes. The dashboard can still exist, but it should be downstream of the failure budget. Green is not the goal. Knowing which failures you can and cannot tolerate is the goal.