How to Evaluate AI Agents Before You Trust Them With Real Work

An executive asked us last month: "We built an agent that drafts our client reports. The demos look great. How do we actually know it's good?" That question, in various forms, comes up in nearly every AI conversation we have with mid-market leadership teams. Everyone can build an agent now. Almost nobody can measure one.

So this post is a practical answer to how to evaluate AI agents when you are a $1M to $50M company, not a research lab. No PhD required, no bespoke tooling. A golden set of real tasks, a grading rubric, a regression habit, and a loop for getting better. If you can run a UAT suite, you can run agent evals.

Why "It Seemed Fine in the Demo" Fails

Demos fail as evidence because they sample the agent's best behaviour on its easiest inputs. Whoever built the agent chose the demo task, and they chose one it handles well. That is not dishonesty; it is human nature. But an agent that nails a clean, well-formed request tells you nothing about what happens when the input is a rambling email thread, a spreadsheet with merged cells, or a client name that happens to match a different client in your CRM.

Agents are also non-deterministic. Run the same task five times and you can get four good outputs and one confidently wrong one. A demo shows you a single draw from that distribution. Production shows you thousands of draws, and your clients see the bad ones.

The public leaderboards do not rescue you either. Right now GPT-5.6 Sol sits 2.8 points above Claude Fable 5 on the Artificial Analysis Coding Agent Index, while Claude Mythos 5 beats Sol on SWE-Bench Pro by 80.3% to 64.6%, per the index-reported figures. Same fortnight, same models, opposite conclusions depending on which benchmark you read. If the people who build benchmarks for a living cannot agree on a ranking, a leaderboard position tells you almost nothing about your invoice-reconciliation agent. Never buy on leaderboard position. Build a small internal eval on your own tasks and let that decide.

What Is an Agent Harness?

An agent harness is the scaffolding around the model: the tools it can call, the permissions it holds, the memory it carries between steps, the retry logic when a step fails, and the verification loops that check work before it ships. The model is the engine; the harness is everything else that makes it a vehicle rather than an engine on a bench.

This matters for evaluation because most agent failures are harness failures, not model failures. The agent could not find the file. The agent had read access but not write access. The agent lost context halfway through a long task. The agent got a malformed API response and gave up instead of retrying. When teams tell us "the AI is unreliable", the model is innocent more than half the time; the scaffolding around it was never designed.

It also means your evals must test the whole system, harness included, not the raw model. Swapping GPT for Claude inside a bad harness changes little. The vendors know this too: Anthropic's July updates to Claude Code added dynamic workflow sizing, which scales the number of agents to the size of the job, with OpenTelemetry attributes so you can watch what the harness actually did. Observability is becoming a first-class feature because serious buyers now ask for it.

How Do You Know an Agent Is Actually Good?

You know an agent is good when it scores well on a golden set: a fixed suite of tasks drawn from your real work, each with a known-good output, graded the same way every time. That is the whole trick, and it is worth stating plainly because so many teams skip it.

Building one takes a day, not a quarter:

1. 1Collect 30 to 50 real work samples for the task the agent performs. Real emails, real reports, real invoices, including the ugly ones. Past work is ideal because a human already produced the correct answer.
2. 2For each sample, write down what a good output looks like. Sometimes that is an exact answer (the invoice total is $14,320.50). Sometimes it is criteria (the summary names all three risks and recommends an action).
3. 3Freeze the set. This is now your exam paper. The agent never trains on it, and you never quietly swap out the questions it fails.
4. 4Run the agent across all of it and score the results.

The first run is usually humbling. We have watched agents that looked polished in demos score in the 60s on a golden set built from a client's actual documents. That number is not a disaster; it is the first honest information anyone in the room has had about the system. You cannot improve what you have never measured, and now you have a baseline.

Weight your golden set towards the work that matters. Anthropic's own usage data across 1.2 million Cowork sessions shows business-process operations at 33.4% of usage (reporting, checklists, spreadsheet reconciliation) while software development is only 8.7%. For most mid-market firms, the agents worth evaluating are the ones touching reports, reconciliations, and proposals, not code.

Pass/Fail or Graded Rubrics?

Graded rubrics, almost always. Pass/fail hides exactly the failures that matter most in business work: the report that is 90% right with one wrong number, the email that is accurate but reads like a robot wrote it, the summary that is correct but omits the one risk the client cares about. Binary scoring calls all of those "fail" and a blank page "fail" too, which tells you nothing about where to aim your fixes.

A workable rubric for most business agents has five dimensions, each scored 1 to 5:

• Accuracy. Are the facts, figures and names correct?
• Completeness. Did it cover everything the task required?
• Format. Does the output match the template, structure and length expected?
• Tone. Would you send this to a client without rewriting it?
• Safety. Did it stay inside its permissions, avoid leaking anything, and flag uncertainty instead of guessing?

Set a threshold per dimension rather than a single average. An agent averaging 4.2 with occasional 1s on accuracy is more dangerous than one averaging 3.8 with a floor of 3, because the first one fails rarely enough that people stop checking.

Reserve pass/fail for the things that genuinely are binary: did it write to the correct system, did it stay under budget, did it stop when it hit an approval gate.

How Do You Build UAT for AI Features?

You extend the UAT suite you already have with an eval layer, rather than building a separate benchmark project. This is the question we hear from every team shipping software with AI inside it: our UAT covers the deterministic features, so what covers the model?

The structure that works looks like this:

1. 1Keep deterministic UAT deterministic. Login flows, permissions, CRUD, integrations: normal assertions, nothing changes.
2. 2Add the golden set as an eval stage. Same pipeline, different assertion style: instead of "output equals X", it is "rubric score meets threshold across the suite".
3. 3Score statistically, not absolutely. One flaky output should not block a release, but a mean accuracy drop from 4.4 to 3.9 across 40 tasks absolutely should. Set thresholds on the aggregate and on the worst case.
4. 4Run it on every trigger that can change behaviour. Prompt edits, harness changes, tool additions, and above all model swaps. Model versions now change under you several times a year, and a swap that improves nine tasks can quietly break the tenth. Regression evals are how you find out on Tuesday in CI rather than Friday from a client.

For grading at scale, use an LLM as judge with human sampling on top. A second model scores every output against your rubric, which makes grading 40 tasks per run affordable. Then a human reviews a 10% sample each week to check the judge itself has not drifted. Judges are good at format, completeness and obvious factual errors; they are weaker on domain nuance, which is exactly what your sampled human catches. Neither alone is sufficient; together they are cheap and reliable.

If your evals live in CI next to UAT, "can we upgrade to the new model?" stops being a meeting and becomes a pipeline run. That is also what makes a multi-model posture practical rather than aspirational; we covered why you want that option in our multi-model strategy piece, and this month's export-control drama made the argument for us.

How Do You Improve an Agent Once You Can Measure It?

You improve an agent by classifying its failures before fixing any of them. The instinct after a bad eval run is to rewrite the prompt and hope. Resist it. Instead, build an error taxonomy: go through every failed task and label the cause. In practice most failures cluster into a handful of buckets:

• Retrieval failures: the agent never found the right document or record.
• Instruction failures: the prompt was ambiguous and the agent chose the wrong reading.
• Tool failures: an API call errored and the harness had no retry or fallback.
• Reasoning failures: everything was available and the agent still got it wrong.
• Formatting failures: right answer, wrong shape.

Then fix the biggest bucket first. This is where the taxonomy earns its keep, because each bucket has a different remedy. Retrieval failures are fixed in your data layer, not your prompt. Tool failures are harness engineering. Formatting failures are one line of instruction. Only reasoning failures might justify a stronger (and pricier) model, and in our experience they are the smallest bucket. Teams that skip the taxonomy routinely pay double the token cost for a model upgrade that fixes a problem better prompts would have solved.

Then re-run the golden set, log the new score, and go again. Improvement becomes a weekly number moving up a chart instead of a vibe.

What Does the Frontier of the AI SDLC Look Like?

The front edge of the AI development lifecycle is supervised delegation: agents do long stretches of work autonomously, and humans specify, monitor and approve at checkpoints. OpenAI's Codex Remote now lets developers kick off work from a phone and approve actions remotely; Claude Code sizes its own agent teams to the job and emits telemetry while they run. Kick off, monitor, approve is becoming the default shape of the work.

Which moves the bottleneck. When an agent can do eight hours of work overnight, the constraint is no longer doing; it is specifying the task well and reviewing the output fast. Evals are what make that review fast, because a golden-set score tells you where to look before you read a word. We wrote about compressing that review in eight hours of agent work, an eight-minute review, and the eval suite described here is the machinery underneath it.

The teams pulling ahead right now are not the ones with the cleverest prompts. They are the ones who can answer "how good is it, exactly?" with a number, and "is it getting better?" with a trend line.

Where to Start This Month

Start with one agent and one golden set. Pick the agent closest to revenue or client-facing risk, pull 30 real work samples, write the rubric, and get your baseline score this week. The whole exercise costs a day and changes every conversation about AI in your business from opinion to evidence.

If you would rather not build the muscle alone, our Deep Dive does this with you: we map your highest-value agent use cases, build the first golden set from your own work samples, and leave you with an eval habit your team runs without us. Either way, stop trusting demos. Start counting.