Evals Before Demos: How We Scope an Agent Build

Why we refuse to write agent code before writing the eval set — and the exact template we use with clients in week one.

---

There's a moment in almost every agent project where the demo works, everyone claps, and then production traffic arrives and the whole thing quietly falls apart. In 2026 we finally have the numbers to prove this isn't bad luck — it's a predictable failure mode with a predictable cause.

Forrester's root-cause analysis of agent deployments that reported negative ROI at the 12-month mark found that 41% of failures traced back to unclear success criteria, and another 26% to drift in evaluation coverage. Read that again: two-thirds of failed agent projects failed because nobody wrote down, in a testable form, what "working" meant. Gartner now projects that over 40% of agentic AI projects will be cancelled by 2027, and the pattern behind most of those cancellations is the same — teams shipped a demo, not a system.

So here's our rule, and we hold the line on it even when clients push back: we do not write agent code before we write the eval set.

Why demos lie

A demo is a single trajectory through a system that has millions of them. When you demo an agent, you're showing the happy path with a cooperative user, clean data, and a prompt someone rehearsed. Production is the opposite: adversarial phrasing, ambiguous requests, stale data, and tool APIs that return errors at 2am.

The industry has caught up to this. The 2026 consensus across evaluation platforms — LangSmith, DeepEval, Phoenix, Confident AI — is that evaluation is no longer a QA step at the end. It's woven into development from day one, with trace-based evals, multi-turn simulations, and regression datasets that grow every time a new failure mode appears in production. DeepEval's v3 release made component-level evaluation mainstream: you can now score every step of an agent's execution — tool calls, retrievals, reasoning steps, final outputs — not just the end result.

That tooling maturity changed what "week one" should look like. If the ecosystem gives you trace-level scoring out of the box, there's no excuse for scoping an agent build around vibes.

The week-one template

Here's the exact structure we use with clients before a single line of agent code exists.

1. The task inventory (day 1–2)

We sit with the people who currently do the job the agent will do, and we collect 50–100 real examples of the task. Not hypotheticals — actual tickets, actual emails, actual queries pulled from logs. Each example gets three fields: the input as it actually arrived, the output a competent human produced, and a note on what made it easy or hard.

This is unglamorous work and it's the single highest-leverage activity in the entire project. If the client can't produce 50 real examples, that's a signal the task isn't well-defined enough to automate yet.

2. The grading rubric (day 2–3)

For each task category, we define what a passing answer looks like in terms a script or an LLM judge can check. We force every criterion into one of three buckets:

• Deterministic checks — did the agent call the right tool, with valid arguments, in a sane number of steps? Did the output parse? Did it cite a real source? These are cheap, fast, and unambiguous, and they should be the majority of your eval suite.
• Model-graded checks — tone, completeness, faithfulness to retrieved context. LLM-as-judge works in 2026, but only with a tightly written rubric and spot-checked calibration against human labels.
• Human review — reserved for the 5–10% of cases where judgment genuinely can't be automated. If more than 10% of your rubric needs a human, your success criteria are still too vague. Go back to step one.

3. The failure taxonomy (day 3–4)

Before building, we write down how the agent is allowed to fail. Every agent will fail; the design question is whether it fails loudly and safely or quietly and expensively. We define, per task category: what the agent should do when it's unsure (escalate, ask, refuse), what the maximum blast radius of a wrong answer is, and which failures are annoying versus which are unacceptable. A refund agent that occasionally escalates too eagerly is fine. One that occasionally approves a refund it shouldn't is not — and the eval set needs to test for the second case specifically, with adversarial examples.

4. The baseline run (day 4–5)

Then we run the eval set against the dumbest thing that could possibly work — usually a single prompt to a frontier model with the relevant tools attached, no orchestration, no memory, no cleverness. This baseline does two jobs. First, it calibrates ambition: with the model quality available in mid-2026, we routinely see naive baselines pass 60–70% of a well-built eval set, which tells you the engineering effort belongs in the last 30%, not in re-solving what the model already does. Second, it gives every future architecture decision a number to beat. "Should we add a planning step?" stops being a debate and becomes an experiment.

What this changes about scoping

The eval-first week produces something most agent proposals never have: a definition of done. The statement of work stops saying "build a support agent" and starts saying "reach 92% pass rate on the 120-case eval set, with zero failures in the unacceptable category, at under $0.15 per resolved conversation."

That sentence does a remarkable amount of work. It tells the client exactly what they're buying. It tells the engineers exactly when to stop polishing. It creates the regression suite that will guard every future prompt change, model upgrade, and tool swap. And when a new model ships — and in 2026, one ships roughly every six weeks — you rerun the suite and know within an hour whether upgrading is safe, instead of finding out from angry users.

It also kills bad projects early, which is a feature. Roughly one in five scoping engagements we run ends with a recommendation not to build the agent yet — because the examples revealed the task is really three tasks, or the data isn't accessible, or the failure tolerance is so low that a human-in-the-loop workflow beats full autonomy. Finding that out in week one costs a week. Finding it out in month six costs the relationship.

Objections we hear, and the answers

"We don't have time — the demo is due Friday." Then ship the demo, but call it what it is: a sales artifact, not a milestone. The eval-first week still happens before production code. In our experience it doesn't even slow the schedule, because the baseline run in day five is a working prototype — one you can measure.

"Our task is too subjective to eval." This usually means the rubric hasn't been written yet, not that it can't be. "Professional tone" feels subjective until you collect ten examples the client rejected and ask why; the answers become criteria. Tasks that survive this exercise and remain truly unscoreable are tasks where the client themselves can't tell good from bad — and that's a project you want to discover you shouldn't build.

"The model is changing so fast, won't the evals go stale?" Backwards. The faster models change, the more valuable the evals are — they're the only stable measuring stick you have. What goes stale without evals is your confidence.

The eval set is the product

Here's the reframe we push hardest: the eval set isn't the test for the product. In a real sense, it is the product. Models will be swapped. Prompts will be rewritten. Frameworks will be abandoned — most agent code written in 2024 has already been deleted. The artifact that survives every one of those changes is the eval set, because it encodes the one thing that doesn't churn: what your business means by a correct answer.

Teams that treated evals as an afterthought are now doing archaeology on their own systems, reverse-engineering success criteria from production incidents. Teams that wrote evals first upgrade models the week they ship and compound the advantage.

Demos are for showing what's possible. Evals are for knowing what's true. We start with what's true.

---

Building an agent and not sure what your eval set should look like? That's literally the first week of our engagement — book a call.