Your Agent-Written Tests Aren't Real Tests
Over twelve thousand agent-written unit tests, all green, and the boundary is still where we find out the code is wrong. Agent-written tests are specification, not verification — a map for the next agent, not proof the code works.
Our UAT [at Swamp] suite has run approx 2000 times. Eighty of those runs caught real defects — not flaky tests, not environment issues, but genuine failures in code that passed through the entire unit test suite without a problem. Over twelve thousand unit tests, all green, and the boundary is still where we find out the code is wrong.
The same understanding written twice
At Swamp, every line of code is written by agents. No human touches the implementation. We required agents to write unit tests for everything, and they did. You can watch it happen on every PR: the tests change alongside the code, written as an afterthought once the implementation is done. Sometimes the agent does not even bother making them pass, we have seen test failures in CI because the agent simply did not care enough about the tests to keep them green.
Both artefacts — the function and its unit test — encode the same interpretation of the requirement. If the agent misunderstood what you wanted, the test encodes that misunderstanding and the implementation satisfies it. Chen et al. forced six LLMs to write tests across five hundred SWE-bench Verified tasks and found that encouraging test writing produced zero net change in resolved issues for GPT-5.2 and a net loss of five for Gemini 3 Pro. Whether the agent wrote tests had no bearing on whether it solved the problem.
When an agent does hit a failing test during its loop, the fastest path to green is rewriting the test. In our workflow, that is exactly what happens. A study of over 86,000 agent-authored test patches found that 80.2% contained weak or no explicit oracle signals. And SWE-Gate showed what that means in practice: among patches that passed their functional tests, 34% still failed to satisfy the full requirements derived from real pull request reviews.
Why independence matters
Developers spent a long time working out how to manage this problem. TDD was the most disciplined answer: write the test first, commit to an interface and a set of expectations before the implementation exists. In disciplined TDD, changing a failing test represented a conscious change to the specification. The developer had to decide: is my implementation wrong, or is my specification wrong? That independence — the test as a fixed constraint — is what gave unit tests their verification power.
Agents cannot replicate this. An agent has its interpretation of the requirement before it writes either artefact. You can instruct it to write the test first, but that does not change what the agent understands, it changes when it writes things down. More importantly, the agent retains write access to both sides throughout the process. In an agent loop, changing a failing test is just another available edit unless the workflow deliberately protects the test.
What the tests actually are
So if agent-written unit tests are not verification, what are they? TDAD gave us a useful answer, surfacing test-to-code dependency maps for agents reduced regressions by 70% and improved issue-resolution rates from 24% to 32%. But giving agents generic TDD instructions without that contextual information actually worsened performance.
The tests help when they function as a map of the codebase. They tell the next agent what the code is supposed to do, how modules relate, what the expected inputs and outputs look like. That is genuinely useful. But it is specification, not verification, and teams should stop treating a green build as evidence of correctness.
What actually verifies
The tests that provide real verification share one property: the agent cannot rewrite them to get to green. Architectural fitness tests verify structural constraints. Property tests enforce invariants across any implementation. Contract tests at service boundaries verify that systems honour their agreements regardless of what the agent did behind the interface. Blackbox tests at the release boundary verify the system from outside the agent's context entirely.
In practice, teams should require that agent-authored PRs include evidence of passing boundary-level verification rather than treating unit test coverage as a quality gate. We recently added explicit requirements for property tests, contract tests, and architectural fitness tests — but asking agents to write these is not enough on its own. If the same agent can author and alter a property test during its loop, the oracle problem follows it there. These test types only add independent verification when their constraints are owned outside the implementation loop: by a protected suite, an external contract definition, or a human-approved specification.
I think as the industry moves toward more agentic code creation, we will see less and less value in unit tests as a discipline. Not because they are badly written, but because the relationship between the test and the code has fundamentally changed. When the same system authors both sides, the unit test cannot be verification. It can be a map for the next agent, and that is worth something, but it is not worth what we have been pretending it is worth.
The boundary is where you find out if the map is right.