New research indicates that large language models (LLMs) tasked with patching software vulnerabilities frequently produce fixes that are incomplete, introduce new flaws, or alter expected program behavior. A study by Off-by-1 Labs, a security research group within 1Password, found that roughly three out of four AI-generated patches for real-world vulnerabilities left something broken.
The researchers evaluated 6,080 patches for six recently disclosed Common Vulnerabilities and Exposures (CVEs), using both ChatGPT 5.5 and Claude Opus 4.8. While many of the AI-generated patches appeared correct and often passed initial tests, closer inspection revealed significant issues.
A common failure mode was the "fragile fix," where models addressed a specific exploit path demonstrated in a reproducer but failed to resolve the underlying general problem. This left the vulnerable code intact and exploitable through alternative means. For instance, in CVE-2026-8512, a Chromium bug related to macOS folder change monitoring, both models routinely implemented only one of two necessary claims on a go-between object, leaving the flaw present but moved.
Approximately half of the evaluated patches failed to fully close the original vulnerability, leaving at least one exploitable path open. About one in twenty introduced entirely new vulnerabilities, sometimes in addition to not fixing the original issue. Other patches closed the original bug but inadvertently changed the software's behavior, such as rejecting previously accepted inputs.
The study highlighted a case with Freenginx, a web server, involving a use-after-free memory bug. An AI-generated fix for this flaw, submitted through the Patch the Planet initiative by Trail of Bits and OpenAI, was rejected by maintainers because it only repaired two of three vulnerable locations. Both the rejected AI patch and the maintainer-written fix also introduced a new way to crash the server. Off-by-1 Labs subsequently reported this new crash, which was fixed on July 2. In a separate case study, 270 attempts by ChatGPT 5.5 to patch the original Freenginx flaw, while 114 were judged to close the original hole, every one of those 114 introduced a new problem.
The Linux kernel privilege escalation vulnerability, "Copy Fail," disclosed in April, also presented challenges. The upstream fix for this bug involved reverting a memory optimization, which initially introduced an off-by-one heap write that required a subsequent correction. Roughly a third of the AI-generated patches for "Copy Fail" regenerated this flawed revert, including the off-by-one error. Additionally, a second, unrelated flaw in the same code section was overlooked by every AI patch, even those that edited the exact file.
The researchers noted that the models tend to fix only the specific bug described in the ticket, even when other vulnerabilities are present in the code they are modifying. This "patch the example, not the bug" approach was a recurring theme.
The quality of guidance provided to the LLMs significantly impacted their success. Prompts containing correct fix directions resulted in bug closure approximately two-thirds of the time. However, prompts offering plausible but incorrect directions drastically reduced success rates to about one in six. The study concluded that providing confidently wrong advice is more detrimental than offering no advice at all.
While an iterative patching process and richer correct context improved results, neither was as impactful as the accuracy of the initial guidance. The study also found that model performance varied wildly across different codebases; for example, Claude cleanly fixed an Exim remote code execution bug in about three-quarters of attempts but managed under one percent for a Gemini CLI trust-bypass advisory.
The cost of generating and validating each patch attempt ranged from two to three dollars, which is inexpensive compared to an engineer's time. However, the researchers emphasized that LLM-produced patches still necessitate review by a skilled engineer with domain expertise. This manual review is costly because understanding a patch well enough to certify its security implications requires at least as much effort as writing a known-good patch from scratch.
Due to the high volume of patches, automated validators were used, cross-checked against each other and spot-checked by humans. The automated reviewers did not catch all new vulnerabilities, such as the "Copy Fail" off-by-one error in many instances, suggesting that the study's reported new-vulnerability rate should be considered a floor. The tooling used in the study has been made public to allow organizations to run it against their own fixed bugs and derive repository-specific metrics.
