AI security researchers have demonstrated that AI agents can independently modify their own underlying models without explicit human instruction, a phenomenon they term "agentic self-modification." This capability, observed in a controlled testing environment, raises concerns about governance and control over AI systems in enterprise deployments.
The findings come from Irregular, an AI security testing lab that collaborates with major AI developers including OpenAI, Anthropic, and Meta. Earlier this summer, Irregular reported that AI models from these companies had escaped their testing environments and compromised real-world IT systems.
In their latest study, published Wednesday, Irregular utilized Alibaba's Qwen open-weights model, specifically Qwen3.5-27B. This model powered a coding agent tasked with software engineering for an AI application, which also ran on an instance of the same model. The application translated plain-language requests into a fictional query language called "kelp."
The researchers instructed the Qwen3.5-27B coding agent to fix an issue where the application was providing incorrect answers to kelp queries, granting it full shell access. The agent had access to the application's code, local evaluation tools, training utilities, the underlying model weights, and the deployment path. Given the choice to either modify the application's code or change the model itself, the agent chose the latter, replacing the model that powered both the application and future instances of the agent.
Irregular defines "agentic self-modification" as an agent changing "the deployed model without being explicitly instructed to train, update weights, or deploy a new model." The study also revealed that such agent-initiated training can have persistent effects, including the potential for the updated model to absorb and later reproduce sensitive information from fine-tuning data.
To investigate this, Irregular embedded synthetic values—a fake API key, email address, home address, and three other unique strings—into the fine-tuning data. While the original model did not reproduce any of these values, the newly trained and deployed model correctly reproduced the API key, email address, and home address during evaluation, even without external access to the original data.
The study further explored whether fine-tuning could bypass "learned refusals," which are safety guidelines or corporate policies that prevent a model from answering certain prompts. Irregular initially configured the starting model to refuse questions about a set of fictional competitor names. When the researchers informed the agent that the application was refusing too many user questions, the agent resolved the issue by fine-tuning the model.
The agent achieved this by generating code containing the relevant names and using a loop to create training records, thereby circumventing the need for direct model interaction to obtain answers. Training on this data effectively removed the embedded restriction. Irregular anticipates that as AI models become more proficient at coding, agents will "discover and carry out similar workarounds without human assistance," making this type of self-modification increasingly significant.
