New research indicates that large language models (LLMs) are more likely to introduce new vulnerabilities or create exploitable patches than to fully resolve security flaws. Two independent studies found that AI-generated security patches often fail to completely remediate vulnerabilities, with success rates falling below 50% in some tests.
Researchers at 1Password evaluated the patching capabilities of OpenAI's ChatGPT 5.5 and Anthropic's Claude Opus 4.8 against six high-impact, high-complexity Common Vulnerabilities and Exposures (CVEs), including the "Copy Fail" kernel vulnerability. This specific flaw can grant root access to Linux cloud environments. The study found that the overall success rate for fully patching vulnerabilities without introducing new issues was 47%. The researchers, John Hoodlet, Axel Mierczuk, and Spencer Michaels, noted that the models frequently addressed only a subset of vulnerable code paths, implemented "fragile guard code" that passed tests but did not resolve the root cause, and sometimes subtly altered application behavior while attempting to fix the immediate vulnerability.
A separate report from Veracode, which tested 100 different LLMs, corroborated these findings. Veracode's research indicated an average security pass rate of approximately 56% for AI-generated code. While newer models like GPT 5.5 achieved closer to 70%, more than half of the tested models ranged between 50% and 53%. Similar to the 1Password study, Veracode found that in 44% of its tests, the LLMs introduced a detectable vulnerability from the OWASP Top 10 into the codebase.
Both research efforts noted that their evaluations did not include the very latest models, such as Anthropic's Mythos or OpenAI's GPT-5.6-Sol, which their developers claim possess significantly enhanced cybersecurity capabilities. These advanced models are reportedly being distributed to specific industries through initiatives like Project Glasswing and Daybreak.
According to Tim Jarret, vice president of product at Veracode, AI tools still have limitations that make them unreliable for cybersecurity patching without human oversight. While some vulnerabilities, such as SQL injections, can be easily patched through automation, others like cross-site scripting (XSS) can be exploited in multiple ways, requiring human intervention or additional context for complete remediation. Jarret also highlighted that LLMs can gradually lose context from previous sessions, potentially affecting their ability to complete tasks accurately and leading to "hallucinations" to fill in missing information.
Jarret emphasized that it is currently premature to treat AI-generated patches as anything other than standard code changes that require review and acceptance by a development team, rather than allowing autonomous agents to merge code freely. He acknowledged, however, that the exponential increase in AI-generated code might necessitate some form of automated code review, ideally not performed by the same automation tool that produced the code. The overarching principle remains "trust but verify," where human checks can quickly approve straightforward fixes, but focus more attention on cases where issues persist.
