Black Box Tests Are Your Most Valuable Asset

Every engineer who's worked in a mature codebase already knows this principle: you can't safely refactor without tests. Nobody argues with that. If you're about to restructure something significant, the first question is always "what's the test coverage like?" If the answer is "poor", you either write tests first or you accept the risk.

Agent-generated code is a refactor on every PR. The agent is rewriting, restructuring, adding to your codebase constantly. So every change needs the same safety net that everyone already agreed refactoring needed. We are moving faster and the principle is still important.

Most teams hear "invest in testing" and think unit tests. More coverage, more assertions, more green checkmarks in CI. Unit tests are valuable: fast feedback, diagnosis, edge-case coverage. But an agent can write them for the code it just produced. They'll pass, and the implementation might still be wrong.

The tests that matter most as the final verification boundary are the ones that test the outcome, not the implementation.

Black box testing

Good unit tests can be independently derived from requirements, and when they are, they're useful. The risk with agent-generated code is that the same interpretation produces both the implementation and the tests. If the agent misunderstood the requirement, the unit tests encode the same misunderstanding. The tests pass but the feature is broken and the user finds out before you do.

Black box acceptance tests don't care how the code works. They test the compiled binary [or app] from the outside, the same way a user would interact with it. Did the command produce the right output? Did the error message actually help? The implementation could be completely restructured and the acceptance tests would still tell you whether the system behaves correctly.

We read a lot about teams saying they can't adopt AI because they're working in a brownfield codebase. A brownfield codebase with a solid acceptance test suite is actually a great candidate for agent work. The tests tell you whether the outcome is still correct regardless of how the internals changed. A greenfield codebase with nothing but unit tests can be more fragile than it looks. You are able to move faster and you can accumulate bugs and technical debt at a faster rate.

Tests are more accurate than the code under test

We have a principle baked into our user acceptance testing system at Swamp: tests are more accurate than the code under test. When a test fails, the default assumption is that the code needs to change, not the test. An agent is never allowed to weaken, relax, or update a test to match current behaviour unless a human explicitly asks for it.

Without that rule, agents do what's natural: they see a failing test, they fix the test. The implementation passes, the suite is green, and you've quietly lost a requirement. With it, the test suite becomes an immovable specification. The agent has to make the code satisfy the tests, not the other way around.

The agent writes the implementation and the unit tests, this is fine. But the acceptance tests were written independently, they test the binary from the outside, and the agent can't change them. The process that produces the code shouldn't also control the criteria for judging whether that code is correct.

The economics reversed

Writing code was interesting. Writing tests, especially acceptance tests, was a chore. Teams would ship features first and add tests later. "Later" was always a lie, but it was a survivable one because the rate of change was slow enough that you could get away with it.

Code is cheap now. An agent can produce a working implementation in minutes. The bottleneck moved to validation. CircleCI's 2026 State of Software Delivery analysed nearly 28 million workflows and found that the top 5% of teams nearly doubled their throughput year over year, while the median team improved by just 4%. Their conclusion was that generating more code isn't enough; the differentiator is the ability to validate, integrate, and ship at scale.

At Swamp, a PR takes about an hour from open to being shipped. to the end user in a stable binary. Acceptance tests account for about 20 of those minutes and we know that number only grows as we add more tests. We're fine with that tradeoff.

This isn't an argument for replacing the testing pyramid with sprawling end-to-end suites. Acceptance tests are slower, more expensive to maintain, and can become brittle when badly designed. Unit tests remain the right tool for fast, targeted feedback and diagnosis. The acceptance suite has a different job: it protects the externally observable behaviours we can't afford to accidentally change. That job becomes more valuable as implementation gets cheaper and the rate of change goes up.

The teams that invested in acceptance testing when it felt like a tax are sitting on the most valuable thing in their codebase. Everyone who skipped it is discovering that "later" just arrived all at once.

Technical debt is untested outcomes

Since I started building with agents full-time, I think about technical debt differently. Plenty of debt matters, but the kind that gets dangerous at the speed agents operate is code where nobody knows whether the outcome is correct after a change.

Messy code with good acceptance tests can be refactored. By a human, by an agent, it doesn't matter. An agent can restructure an entire subsystem in an hour if the acceptance tests are there to verify the result. Clean code without acceptance tests looks fine right up until someone changes it, and agents change code at a pace that makes manual verification impossible.

Defining what correct looks like

The human judgment in this workflow isn't about implementation anymore. Implementation is cheap. It's about acceptance criteria. What should this command output? What should happen when the input is malformed, or when something fails? Defining what "correct" means was always the harder problem. It just used to be buried under the cost of building it.

At Swamp, the humans are on triage, plan review, and PR authorisation. The agents are on implementation. The acceptance tests, the definition of what the system should actually do, are human-defined. That's where the judgment went. The acceptance tests reflect the outcome we want from the feature and if the agent has gone off on a tangent, we can see the outcome isn't correct.

What outlasts everything else

I've written a lot this year about what lasts and what doesn't in AI engineering. Practices have a half-life, prompting strategies expire and orchestration patterns get replaced.

Your acceptance test suite outlasts all of it. Everything else turns over — the code, the prompts, the practices — but the tests that define what the system should do from the outside don't have a half-life.

If you're going to invest in one thing that makes your codebase ready for agents, invest in black box acceptance tests. In a world where code is cheap and easy to replace, the specification of what the system should actually do is the only durable asset you have.