# Give the agent a definition of done

> Evaluation-driven development turns an ambiguous AI project into a sequence of testable improvements. The hard work is deciding what the tests should mean.

Author: HARTRO
Published: 2026-10-08
Updated: 2026-10-20
Topics: Engineering
Canonical URL: https://hartro.com/blog/evaluation-driven-development/

![A set of written test cases passes through an evaluation gate to checked and unresolved outcomes.](https://hartro.com/assets/blog/covers/evaluation-illustrated-v3.svg)

An invoice-processing workflow can extract every field correctly and still prepare an entry that finance must reject.

The supplier may have changed its bank details, or the same invoice may have arrived through two channels. Field accuracy alone will miss both problems.

I would give the agent those cases before asking it to improve the implementation. They define the decisions it must get right and the situations it must refer for review. That is what I mean by evaluation-driven development.

The opportunity is larger than prompt testing. Coding agents can modify parsers, retrieval, validation, interfaces, and orchestration. A good evaluation lets them investigate the whole path to a result. A poor one lets them optimize the wrong thing faster.


## Write down the behavior you expect

Ordinary software tests remain essential. They can establish that an amount parser handles a comma decimal separator or that a permission check rejects an unauthorized call.

Acceptance and integration tests already ask whether an assembled system behaves appropriately. Agent evaluation extends that practice with representative task sets, repeated trials for variable outcomes, and judgments that may need explicit rubrics. It can inspect the final answer, tool calls, state changes, and the path taken to reach them. The boundary is practical rather than absolute: many good agent evaluations contain ordinary software tests.

[Anthropic's evaluation guidance](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) distinguishes deterministic, model-based, and human graders. The distinction is useful because different claims require different judges. Code can check a ledger balance. A calibrated rubric can help assess whether an explanation is supported. A domain expert may still need to resolve an ambiguous policy interpretation.

The cases describe the behavior we currently understand. They will leave gaps, which need to be recorded as the evaluation grows.

## Start with a real disagreement

Consider a hypothetical invoice-intake assistant. Its job is to prepare draft accounting entries for review. It has no authority to initiate payments.

Rather than beginning with “extract invoices accurately,” ask finance and engineering to classify a handful of actual workflow situations. Which should produce a draft? Which should be rejected? Which need review? What information must accompany the decision?

The useful cases are often where people hesitate. An invoice is internally consistent but duplicates an existing record. The total includes a discount whose treatment is unclear. A scan is legible except for the currency. The supplier identifier matches, but its payment details have changed.

Resolving those cases creates an implementation target. Keeping disagreement explicit prevents a convenient label from hiding an unresolved policy decision.

For example, one acceptance case might look like this:

```yaml
case: changed-supplier-bank-details
fixture:
  supplier: known-supplier
  bank_details: different-from-approved-record
expected:
  disposition: human-review
  reason: payment-details-change
  draft_entry: allowed-if-other-checks-pass
  payment_calls: 0
  approved_supplier_record: unchanged
```

The contract makes consequential behavior inspectable. Extracting the new bank details correctly is only one part of passing.

## Judge outcomes before style

A natural first metric is field accuracy. It is easy to count and easy to improve. It can also conceal the failure the business cares about.

A more useful evaluation separates dimensions rather than averaging them immediately:

| Dimension | Evidence to inspect | Suitable first check |
| --- | --- | --- |
| Extraction | Amounts, currencies, dates, supplier fields | Deterministic comparison with accepted references |
| Decision | Draft, reject, or request review | Case-specific policy assertions |
| Authority | Tool calls and resulting state | No disallowed writes or payment actions |
| Explanation | Reasons tied to document evidence | Explicit rubric, calibrated with reviewers |
| Operational cost | Runtime, retries, tokens, human corrections | Measurements on the full workflow |

Some failures should be release blockers, not small deductions from an average score. An unauthorized write should not disappear behind excellent formatting. Other dimensions involve tradeoffs: a system that escalates every invoice can avoid unsafe decisions while providing little useful automation.

Track those tradeoffs directly. Show the rate of appropriate completion alongside review load and failure types. If the system abstains, assess whether abstention was warranted and whether the reviewer receives enough context to act.

## Make iteration informative

With those cases in place, the development loop is straightforward. Run a baseline on development cases. Inspect failures. Group them by cause. Change the relevant component and compare it against those cases. Reserve fresh acceptance cases for agreed release decisions, rather than inspecting them after every edit.

A failure involving retrieval may need a different document index. A currency error may need deterministic parsing. A duplicate-entry failure may belong in the write path rather than the prompt. An unsupported explanation may require evidence to be carried through the workflow.

The agent needs access to those distinctions. A single score hides them. Per-case traces and visible state changes make them investigable.

A minimal runner might follow this pseudocode:

```text
for case in development_cases:
    environment = reset_to_fixture(case.fixture)
    result, tool_calls = run_workflow(case.input, environment)
    final_state = inspect_state(environment)
    failures = check_assertions(case.expected, result, tool_calls, final_state)
    save(case.id, run_configuration, result, tool_calls, final_state, failures)
```

Resetting the fixture keeps one case's writes from contaminating the next. Inspecting state catches a workflow that reports success while leaving the wrong record behind. Reviewers can group the saved failures by cause, then give the implementation agent a specific class to investigate.

[OpenAI recommends continuous evaluation](https://developers.openai.com/api/docs/guides/evaluation-best-practices): run evaluations as the system changes and use observed failures to expand the case set. This turns production feedback into development material instead of treating launch as the end of quality work.

Reproducibility matters here. Record the model, instructions, tool definitions, data snapshot, and execution conditions. Repeat variable cases. Anthropic's [infrastructure-noise experiments](https://www.anthropic.com/engineering/infrastructure-noise) found that environment configuration alone could materially change agent benchmark outcomes. A changed score is a reason to investigate, rather than immediate proof that the latest prompt was better.

## Let the agent change the implementation, not its pass mark

An agent improving the implementation should not quietly change what counts as success. Give it development cases and useful diagnostics, while keeping acceptance criteria and protected tests under separate control.

This does not mean freezing an incorrect test forever. Sometimes the agent exposes an ambiguity or a grader bug. Resolve that through an explicit review: what requirement was wrong, what changes, and which earlier results are no longer comparable?

The same applies to model-based judges. Check that reviewers agree with the rubric, compare their judgments with the grader, and inspect disagreement. A judge that rewards confident prose can push the implementation toward confident errors.

Hold back genuinely fresh cases. A team that repeatedly studies every acceptance example has turned that set into development data, even if nobody copied its labels into a training file. Keep a record of when evaluation sets change so a higher score cannot be confused with an easier exam.

## Who owns the expected answer

For a small project, this can start with a short case file and a simple runner. There is no universal minimum dataset size that turns uncertainty into confidence. The first task is to make important behavior visible; broader coverage and stronger evidence follow as the system's responsibilities grow.

Give a domain owner responsibility for the expected decisions. Give engineering responsibility for making outcomes observable. Let the agent propose and test implementation changes within that structure. Review what the cases omit as carefully as what they cover.

This is the request I want to be able to make: “Fix this class of failure without losing these behaviors, and show the evidence.”

*Research checked on October 15, 2026. The invoice workflow and runner are illustrative.*