AI Broke the Assumptions Behind CI

CI queues are getting longer. Pipelines are backing up. CircleCI's 2026 State of Software Delivery report, analyzing 28 million workflows, found throughput up 59% year over year while main-branch success rates dropped to 70.8%, a five-year low. Recovery times climbed 13%. Every time GitHub has an outage, engineers talk about it like a weather event. Nothing you can do, just wait it out. The common explanation is more commits, more pressure on the system.

The obvious response is to add more runners. And you should, if queues are your problem. But imagine you had infinite CI capacity and zero queue time. An agent writes a change, opens a PR, and CI picks it up instantly. The compile fails, the agent pushes a fix, CI picks it up instantly again. A test fails, another fix, another instant run. Each iteration is fast, but the agent is still discovering that its change doesn't work only after it crosses the PR boundary. The feedback loop is in the wrong place regardless of how fast CI runs.

This is an architectural problem. CI has historically conflated two jobs: executing verification and coordinating what's ready to merge. When humans wrote all the code, that conflation was fine. A developer would run things locally, catch the obvious stuff, and the CI round trip was mostly a formality for the things that slipped through. Pre-PR verification happened, but it was informal, inconsistent, and entirely dependent on individual discipline.

Agents change this in a specific way, not because agents can't run tests locally. They can [and they should!]. The change is that pre-PR verification can now become a defined, repeatable, automated stage of the development lifecycle rather than something a developer might or might not do before pushing. When verification becomes a formal stage, the question shifts: if you can run the same checks before the PR opens and fix failures immediately in context, why should CI be the place where those failures are first discovered?

The feedback loop crosses the PR boundary

Every CI failure on a PR is a round trip. Find the failure, push a fix, wait for CI again. Each iteration costs wall-clock time whether CI is fast or slow, because the cost isn't just the run, it's the context switch. By the time CI reports a failure, the developer or agent has moved on. Coming back to fix it is often more expensive than the fix itself. And in a shared queue, every failed iteration delays everyone behind you.

This was tolerable when a team manually pushed code throughout the day. With agents generating changes at an order of magnitude faster, the iteration count goes up and each round trip through CI compounds. The volume isn't the issue, it's that we now have verification happening in a place that's architecturally late. The agent has the code, the tests, and the environment. It could verify the change before it ever leaves.

Moving verification before the PR

At Swamp, we're converting our CI process into Swamp workflows where the agent runs the full verification cycle in an isolated local environment before a PR is ever opened. Linting, tests, compilation, agent-powered code reviews, and adversarial reviews all run the same process CI would run, isolated from the authored code.

When something fails, the agent sees the failure immediately and fixes it in that iteration. If a test fails, the agent has the full error output right there and can iterate without leaving the environment. The loop keeps running until everything is green or the agent determines it needs human input. Failures that would have been CI round trips get resolved before the PR exists.

Could a pre-commit hook do this? A pre-commit hook is a trigger: it runs a fixed set of checks and reports pass or fail. Our verification loop is an iterative process. It runs the full set of controls that CI would run, observes failures, modifies the implementation, reruns the controls, and repeats until everything passes or the agent determines it needs human input. It also includes agent-powered review steps that traditional CI never had: adversarial review where a separate agent tries to break the implementation, UX review for user-facing changes. The attestation it produces is tied to the resulting commit after all iterations, not the original one. By the time a human sees the PR, it's been through a more thorough verification process than most CI pipelines provide.

You could run this verification remotely instead of locally. Nothing about the model requires it to run on the developer's machine, and for some controls (particularly those with specific infrastructure or security requirements) remote execution will be the right choice. The important architectural change isn't where the verification runs, it's that verification runs as a defined stage before the PR, produces structured evidence of what happened, and CI validates that evidence rather than re-executing the work.

The attestation model

When everything passes, the workflow produces an attestation: a structured manifest that records exactly what happened during verification. The commit hash that was verified, sha256 checksums of the config that governed it (review prompts, workflow definitions, CLAUDE.md), every step that ran with its result and duration, and the overall gate decision. Anyone can look at it and see precisely what was verified, under what configuration, and when.

The attestation is the architectural turning point. Before it, verification is something that happens and then disappears into logs. After it, verification produces a structured record that ties an exact commit to an exact verification configuration and its results. That record is the key. CI can independently verify that the attestation names the right commit, that the config checksums match the files checked into the repo at that commit, that all required steps are present, and that the record is fresh enough to be relevant. What CI can't verify from the attestation alone is whether the execution actually produced the claimed results. The attestation says the tests passed. It can't prove they ran. But once CI can independently validate config integrity, completeness, and freshness, it no longer needs to re-execute every control to answer the questions it cares about.

This separates execution from coordination. Verification environments execute controls. Attestations record what happened. CI validates and coordinates. Trusted infrastructure produces release artifacts. Acceptance tests independently verify the shipped outcome. Each layer does one job.

CI becomes a validator

By the time a PR opens, the verification has already happened. CI validates the attestation: does the commit match the PR, are all required steps present and green, do the config checksums match the files at that commit, is the attestation fresh enough that the verified state still represents the branch. Freshness matters because verification evidence must correspond to the exact commit being merged. If the branch has changed since verification ran, the attestation is no longer valid and verification needs to run again. If everything checks out, CI merges. Validation takes seconds because there's nothing to execute.

If your CI provider goes down for an hour, the verification loops keep running. Agents keep iterating, producing attestations, getting code ready. When CI comes back, it processes the backlog quickly because validation is cheap.

At Swamp, our PRs take about forty-five minutes from open to shipped, and about half of that time is acceptance tests. Before the verification loop, roughly 25% of our PRs would hit a CI failure (due to agent review) that needed a fix and another round trip. While we've been testing this model, we left the full CI pipeline running alongside the local verification checks as a shadow validation. 35 PRs ran through both systems. The verification loop predicted that CI would pass every time, and CI confirmed it every time. We're measuring what we call the CI escape rate: how often does traditional CI find a failure after the pre-PR verification loop declared the change ready? So far, across those 35 PRs, the escape rate is zero. CI hasn't found a single failure that the verification loop missed. Are there still suggestions that the code review found? Sure there are, but they were found locally too and deemed not at a level we needed to pick them up.

35 PRs is early evidence thats this new flow is working for us. The shadow validation is how we'll know if it stops working, and it's how we'd recommend anyone migrate: run both systems, measure escapes, investigate every one, and only consider removing duplicated CI execution once the escape rate gives you confidence that the verification loop is catching what CI would catch.

The compiled binary that ships to customers never comes from a developer's machine. The release build happens on shared infrastructure the developer doesn't control. Today that's a CI runner, but the build is just another workflow step that can run on any trusted infrastructure. For compliance, the attestation model produces structured, verifiable evidence of every control that ran, with checksums proving the right configuration was used. That's a more auditable trail than CI logs that scroll off a runner.

The trust model

The first question people ask is trust. The implementing agent and the reviewing agent run on the same developer machine. Isn't this having your homework marked by yourself?

An attestation is a claim, and CI should treat it as one. The trust model needs to be explicit about what can be independently verified from an attestation and what can't.

Config integrity — did the attestation reference the expected verification configuration — is independently verifiable. The attestation includes sha256 checksums of every review prompt, workflow definition, and CLAUDE.md at the verified commit. CI can checkout that commit, hash the files, and verify they match. A tampered prompt produces a different hash. The commit, the completeness of required steps, and the freshness of the attestation are all independently checkable.

Result integrity — did the execution actually produce the verdict being reported — is not provable from the attestation alone. The attestation says the tests passed. It can't prove the tests actually ran. This is a real limitation, and it applies to CI runners too. A compromised runner can report fake results. A stolen PAT with checks: write can forge check runs. But the practical security properties are different. An ephemeral CI runner on managed infrastructure, provisioned per-job and torn down afterward, presents a different attack surface than a long-lived developer laptop. Pretending these are identical doesn't help the argument.

What matters is whether the trust chain as a whole provides sufficient assurance for the controls being validated. The local machine produces the attestation with config integrity checksums. The attestation gets posted to Swamp Club as an attestation document. CI validates the attestation independently: checks the commit match, re-hashes the config files at that SHA, verifies completeness and freshness. Forging a check run requires a token with checks: write. Posting a fraudulent attestation that passes CI validation requires that the attestation's config checksums match the actual repo state at the claimed commit, which CI independently verifies. These are separate checks, and both must be defeated for a forged result to reach the merge queue. Whether they represent truly independent trust domains depends on how credentials are scoped in your environment.

For controls where result integrity requires a higher level of assurance, running them on managed infrastructure is the right call. The model doesn't require everything to run locally. It requires that verification run wherever its resource and trust requirements make sense, and that the results flow forward as structured evidence rather than being re-executed at the merge gate.

On a local machine, someone could edit a review prompt before the workflow reads it, get a green verdict, and revert the edit. On CI, someone could push a commit that weakens the review prompt, get it merged, then revert. On a local machine, someone could kill the agent process and write a fake pass to the result file. On CI, someone could add if: false to a review job temporarily. The attack vectors are different and have different likelihoods depending on your threat model. The important thing is to be explicit about which controls require which trust properties, rather than assuming CI provides them all by default.

The verification loop does reduce supply chain surface area in one specific way: the review and analysis controls. Many CI pipelines run reviews and analysis through third-party actions pulled from GitHub's CDN, which most teams never audit. The verification loop replaces those with review prompts and workflow definitions that are checked-in, versioned files in your repo. Anyone can read verification/review-prompts/adversarial-review.md and see exactly what the reviewer checks for. Compilation, tests, and builds still consume whatever external dependencies the project requires. The supply chain reduction applies to the verification controls themselves, not to the code being verified.

The implementing and reviewing agents are fully isolated. Separate processes, separate prompts, no shared conversation history. The reviewer can't be biased by "I wrote this so it must be right" because it didn't write it.

The supply chain risk that does matter in the AI era is the agent's instruction set. A poisoned CLAUDE.md or review prompt doesn't just break one build. It steers every agent on the team silently. The code looks correct because it follows the rules. The rules are just wrong. This works identically whether the review runs locally or on CI, because the agent reads those files from the repo regardless. Changes to agent instruction files should require a human reviewer. We are now conducting a difference style of reviews on those fils to ensure they are not poisioned or compromised as part of the verification.

Separating execution from coordination

Traditional CI bundles everything into a single pipeline: lint, test, compile, review, build, deploy. Every check runs in the same place, in the same queue, using the same shared resources. Adding a new check makes the pipeline slower for everyone.

Once verification produces structured evidence, the checks don't need to run in the same place anymore. Each verification environment executes controls independently. Attestations record what happened. CI validates and coordinates. Trusted infrastructure produces release artifacts. Acceptance tests independently verify the shipped outcome. Each agent works in its own isolated checkout. Multiple agents can verify different changes simultaneously without contending for the same queue. Adding adversarial testing, accessibility checks, or performance benchmarks doesn't affect the merge path because CI isn't running them.

What we've ended up with at Swamp looks a lot like trunk-based development. The verification loop catches implementation problems before the PR. CI validates the attestation in seconds. After merge, the acceptance test suite runs against the actual binary as the independent check on whether the outcome is correct. Verification moved to three places where each layer does a different job, instead of one pipeline trying to do all of them.

CI is the next weak link

I've written before about practices having a half-life. The CI pipeline you spent years perfecting was the right answer for human-speed development. It's not wrong because it was badly designed. The inputs changed.

The way code gets created, reviewed, and tested has already changed. Why should CI be the place where an agent first discovers whether its change works?