A generated patch is a hypothesis until someone re-runs the exploit.
Across 6,080 AI-generated patches for six recently disclosed CVEs — 1Password's Off-by-1 Labs, using ChatGPT 5.5 and Claude Opus 4.8 — 26% closed the vulnerability without changing how the application behaved. 53.9% failed to fix it, introduced a new vulnerability, or both. Even among the passes, more than a third were judged fragile: blocking one specific route in while the vulnerable code sat untouched behind it.
But the number that should change how you work is a different one.
When the prompt carried a correct fix direction, the models closed the bug about two thirds of the time. When it carried a plausible but wrong direction — the kind a confident review bot or a rushed triage supplies — that collapsed to roughly one in six. And in those runs, the agents' own tool calls sometimes returned evidence contradicting the prompt. They went with the prompt anyway.
So the risk is not that the model is weak. It is that the model will execute your wrong hypothesis faster, and more convincingly, than you could have alone.
Which makes the pipeline straightforward. Reproduce the exploit first and keep that reproduction as the acceptance test. Re-run it against the patched build. Run the behavioural suite beside it. And if you cannot vouch for the fix direction you are handing the agent, hand it the bug and stay quiet.
"Can generate a fix" and "can own remediation" are still very different capabilities.