The evaluator that caught my own bug

I build my local agent console under a rule I carried over from decades in the trades: the person who does the work doesn’t get to grade their own work. Every build gets attacked by an independent AI evaluator — a separate adversarial pass whose only job is to break my guarantees before I call anything done.

During the portability build, it earned its keep. The console has a hard rule: if model-selection state is corrupt, chat must fail closed — block and explain, never silently fall back to a default model. My tests were green. The evaluator went around them: it corrupted the selection state and then explicitly requested a model by name. That request path slipped past the guard my tests were watching. Real bug, in code I had already trusted.

I fixed it, the evaluator re-attacked the fix, and the build stood. Then the second half of the lesson arrived: a later round claimed another bypass — a cached-state attack that sounded completely plausible. This one did not survive contact with a reproduction. I wrote a runnable test that performed the exact attack against the real code; the guard held on every variant, so the claim was recorded as refuted, with the script kept as the receipt.

The evaluator is not the authority.

Both outcomes are the system working. A finding is not trusted because an AI said it. A bug is not real until it reproduces, and a green bar is not safety until something adversarial has failed to break it. The evaluator supplies pressure and hypotheses; runnable evidence decides.

That distinction matters most when agents touch real files. An eloquent evaluator can be wrong. A familiar code path can be wrong. The useful process gives neither one special status: state the guarantee, attack it, reproduce the failure, fix it, and attack again.