When an agent says it's done

A signup form accepts an email address and displays a success message. The address appears on the page. Reload the browser and it disappears.

Nick Nisi and Zack Proser use this failure in their WorkOS workshop on agent verification. The interface changed, but the signup was never persisted. A check that looked only for the address on the page would have passed.

The agent could truthfully report that it added a form and saw the confirmation. The customer would still be missing from the list.

Before reporting completion, the agent needs to check the result the customer depends on and keep evidence for review. If the signup disappears after a reload, that failure should send it back to work.

Check the result the user will depend on#

The signup example is small enough to make the problem clear. A visible confirmation establishes that the page displayed something. It does not establish that the application saved the right record or will find it again.

The required check depends on the promise. For browser-local storage, submitting and reloading may be the relevant test. For a shared waitlist service, the team also needs to check the authoritative stored record. A successful local-storage test would not establish that a server received the signup.

The same distinction applies beyond forms. A generated report can exist while containing the wrong period's data. A settings control can change appearance without changing the saved preference. A deployment command can succeed while traffic still reaches an older version.

Before implementation, identify the observation that would distinguish success from the nearest plausible failure. For the waitlist, the failure already supplies a useful question: after reloading, can the application recover the submitted entry from the place it is supposed to be stored?

Our article on evaluation-driven development covers how to define expected behavior. Here, the additional requirement is operational: the agent must be able to run the relevant check and bring back its result.

Give the agent a way to see the failure#

In the workshop, Nisi and Proser expose application actions and observations through a CLI: submit an entry, reload, inspect storage, capture the page, and run checks. The value is that the agent can act, inspect the result, and use a failure to choose its next fix. Workshop, 18:18–25:27

A particular command-line interface is optional. The important part is making the application observable through tools the agent can use. If it can edit a persistence function but cannot exercise the form or inspect the resulting state, somebody else inherits that verification work.

The checking path needs to be repeatable. Start from a known state, perform the action, and inspect the outcome. Use distinguishable test data so the check cannot accidentally find a record left by an earlier run. If the setup fails, report a setup failure rather than treating the absence of a useful result as a pass.

Anthropic's account of harnesses for long-running agents describes agents declaring features complete without sufficient testing, and the role of browser-based checks in catching problems beyond code inspection. It is an account of a particular harness, but the practical requirement carries across: the environment must let the agent test the behavior it is changing.

This does not require running every available test after every edit. A focused check can guide the repair. The final validation should cover the behavior and dependencies affected by the change.

Different claims need different evidence#

A screenshot is useful evidence for appearance. It is weak evidence for a database write. A green unit test can establish a function's behavior for its tested inputs while leaving the browser integration unexamined.

For a signup change, a compact evidence set might look like this:

These are example claims, not a mandatory checklist for every task. The point is to connect each observation to the conclusion it can support.

Use deterministic checks for things with a precise answer: a record exists, a value matches, a request failed as expected. Model-based review can help assess qualities such as clarity, using criteria checked against human judgments. A second model agreeing with the first does not establish that either inspected the right system.

Proser makes a related point in his account of experiments that passed his own checks. Measures that looked successful did not establish that the underlying product or model did its intended job. A verifier is another implementation to examine, not a source of automatic certainty.

Evidence belongs to a particular change#

Suppose the signup check passes. The agent then changes the storage key while cleaning up the implementation. The earlier result still exists, but it no longer establishes that the latest code works.

Completion evidence should identify what it checked. For a code change, record which version was tested, where it ran, and what test data and settings were used. Include any uncommitted changes. For a deployed application, record which release was exercised.

A useful handover can be brief:

Change: retain waitlist entries across page reloads
Revision: link to the checked revision
Environment: local test instance with a fresh browser profile
Evidence: submission, stored entry, reload, recovered entry
Checks: links to results and relevant screenshots
Remaining scope: production deployment has not been checked

This is an illustrative record. Its last line matters as much as its passing checks. It prevents a local result from quietly becoming a claim about production.

Evidence should remain inspectable outside the agent's summary. A reviewer should be able to open the result, see the conditions, and rerun the important check. The agent can explain what it means, but the explanation should point to the observation.

Let a failed check drive the next attempt#

Once the agent can observe a failure, the development loop becomes more useful. Run the check, inspect what failed, change the relevant implementation, and check again. A failed reload test tells the agent more than another instruction to be careful.

The loop still needs a stopping condition. Repeated failures may mean the implementation is wrong, the environment is unavailable, or the expected behavior has not been agreed. After a bounded investigation, the agent should return the blocker and the evidence rather than keep editing until something turns green.

If the check itself appears wrong, the agent should explain why and ask for the expectation to be reviewed. Weakening it merely to get a pass destroys the meaning of the result.

For the signup form, the handover should let someone trace the address from submission to storage to the reloaded page. If that path breaks, the work is still open, however convincing the completion message sounds.