Your Tests Passed. On Which Version?

Here's a small thing that causes real incidents.

An AI agent writes a change. The test suite runs and passes. Someone notices a naming issue and tweaks the change by hand, or the agent is asked for one more small fix. The pull request still shows a green check, so it merges.

But the green check belongs to the version before the edit. Nobody tested what actually shipped.

Why this gets worse with AI

With people, edits after review tend to be few and visible. With AI agents, a change can go through several quick rounds: generate, review, fix, adjust. Each round is a new version. If your process treats "tests passed at some point" as "tests passed", the faster the iteration, the wider the gap between what was tested and what merges.

It's a quiet failure. Nothing looks wrong. The check is green, the approval is there, and the diff looks reasonable.

The rule: a result belongs to a version

The fix is a simple rule. A test result counts only for the version it ran on.

In AI CodeOps, that rule is built into the release gate:

• After an agent writes a change, the session runs the project's own test command in a sandbox and records the result against that version.
• If anyone edits the change after its tests ran, the old result neither passes nor blocks it. The gate marks the new version as untested.
• The approval opens again too, so a person decides before the edited version can ship.
• Re-running the session tests the new version and the gate can clear it.

Notice the second point. The old result doesn't count as a failure either. It simply stops counting, because it's evidence about a different change.

The same rule in CI

The gate your CI can call follows it too. It looks at every session that touched the pull request or commit. You can also ask it to deny while a session is still running (blockOnInFlight), so a merge can't sneak in between a new version and its test run.

What this means for your team

You don't need a special tool to apply the principle. Ask three questions about any green check before merging:

1. Which commit did this run on?
2. Has the change been edited since?
3. Does the approval cover the version I'm merging, or an earlier one?

If you can't answer all three from your tooling, your process has the gap. With AI writing more of the code, it's worth closing.

AI CodeOps is free during the open beta: https://aicodeops.dev

Drafted with AI assistance and edited by the AI CodeOps team. Originally published on Medium.