Autonomous patching of software vulnerabilities using large language models (LLMs) currently demonstrates a low success rate and often introduces new issues, according to research conducted by 1Password's Off-by-1 Labs. The study, which involved generating over 6,000 patches for six recently disclosed CVEs using ChatGPT 5.5 and Claude Opus 4.8, found that only 26.0 percent of the LLM-generated fixes fully resolved the vulnerability without altering application behavior.
The research indicates that human oversight remains critical for security remediation efforts. Of the patches analyzed, 20.1 percent fixed the original flaw but changed application logic, such as modifying allow list behavior to deny list behavior. A further 2.3 percent successfully addressed the vulnerability while simultaneously introducing new security issues. Nearly half of the patches, 49.3 percent, failed to fix at least one existing exploit path, and 2.2 percent both failed to fix the vulnerability and introduced a new exploit path.
Keith Hoodlet, director of security research at 1Password, highlighted in a blog post that even among patches deemed successful or partially successful, over a third were considered "fragile." This means the code adjustments might guard against a specific instance of a vulnerability, such as escaping particular input characters, but did not address the underlying problem comprehensively. The researchers, Axel Mierczuk, Spencer Michaels, and Keith Hoodlet, coined the acronym FLAWED (Fix-Like Artifacts With Embedded Defects) to describe these automated LLM patches, concluding that the expected value of a fully LLM-generated, unreviewed patch is significantly negative.
The effectiveness of LLM-generated patches is heavily influenced by the initial guidance provided. While both human developers and LLMs benefit from guidance, LLMs are more susceptible to being derailed by incorrect advice. When given correct guidance, LLMs achieved a 65.0 percent fix-success rate, compared to 50.4 percent with no guidance. Incorrect guidance drastically reduced their success rate to approximately 15.2 percent. Human developers, the authors noted, are more likely to identify and correct misleading information during their reasoning process.
Despite the low success rates, the cost of generating these patches is relatively low. The average successful, clean patch cost an estimated $6.74, including the expense of failed attempts. However, the researchers argue that a comprehensive cost-benefit analysis must account for the significant expert supervision required to make LLM-assisted patching genuinely useful.
The authors have released a patch evaluation harness, also named FLAWED, to assist organizations in assessing the efficacy of their security fixes. They caution that the cognitive load imposed by reviewing a large volume of mostly incorrect or subtly different LLM-generated patches could lead engineers to spend more effort than if they had patched the vulnerabilities themselves using standard LLM-assisted coding techniques where humans maintain control. The alternative, surrendering to an autonomous process with only a one-in-four success rate, poses substantial long-term risks for organizations considering LLM-driven patching.
