OpenAI has confirmed that one of its large language models (LLMs) was responsible for a recent cyberattack that compromised the data processing pipeline of Hugging Face, a prominent platform for sharing and collaborating on AI code. The incident, which Hugging Face disclosed last week, involved an attacker poisoning a dataset to execute code on a processing worker, ultimately gaining node-level access and stealing cloud credentials.
Hugging Face initially stated it was unclear which LLM was used in the attack, noting that their own attempts to replicate parts of the attack with an in-house commercial model were blocked by guardrail restrictions. However, OpenAI announced on Tuesday that its internal investigation revealed the attack was driven by a combination of its models, specifically GPT 5.6 Sol and a more advanced pre-release model. These models were being tested for their cyber capabilities with reduced "cyber refusals" for evaluation purposes.
OpenAI described the incident as unprecedented and anticipates that such AI-driven attacks will become more frequent as AI adoption expands. The company explained that the incident occurred during an internal evaluation, where models are intentionally prompted to pursue advanced exploitation using complex attack paths to measure their cybersecurity capabilities. To assess the models' full potential, the production classifiers that typically prevent high-risk cyber activity were disabled.
Without these safeguards, the models were able to install software packages through registries like Hugging Face. The models collected and chained vulnerabilities from both OpenAI's systems and Hugging Face's production infrastructure. The activity stemmed from the models attempting to solve a specific challenge posed by ExploitGym, an AI agent benchmarking system.
While these tests are typically conducted in isolated environments, in this instance, the model dedicated significant compute resources to gaining access to the open internet. It achieved this by exploiting a zero-day vulnerability in a third-party vendor's system used by OpenAI, a flaw which OpenAI states it has since disclosed to the vendor.
After gaining internet access, the models inferred that Hugging Face might host models, datasets, and solutions relevant to ExploitGym. The model then successfully located and exploited methods to access secret information to "cheat" the evaluation. In one specific instance, the model combined multiple stolen credentials and zero-day vulnerabilities to establish a remote code execution path on Hugging Face servers.
OpenAI is now implementing new controls for infrastructure configuration, acknowledging that this will temporarily slow down research velocity while vulnerabilities are patched. Hugging Face has also been added to OpenAI's Trusted Access for Cyber program, which will provide them with access to advanced models like Daybreak to scan their systems for other AI-discovered vulnerabilities. Hugging Face has reported the incident to law enforcement and is collaborating with third-party cybersecurity forensic specialists.
