LIVE · cybersecurity feed
Live wire
ai

Your AI red team may be finding the same prompt injection thousands of times

A collaborative study involving researchers from the University of Waterloo, Zhejiang University, and KTH Royal Institute of Technology has identified a significant flaw in how developers harden AI agents against prompt injection attacks. The findings suggest that automated red-teaming tools, which are frequently used to generate large datasets of successful attacks for model retraining, often…

ZeroDay News ·

Source: Help Net Security

A collaborative study involving researchers from the University of Waterloo, Zhejiang University, and KTH Royal Institute of Technology has identified a significant flaw in how developers harden AI agents against prompt injection attacks. The findings suggest that automated red-teaming tools, which are frequently used to generate large datasets of successful attacks for model retraining, often produce redundant data that leaves AI agents vulnerable to a wider variety of threats.

The research focused on the common practice of using automated tools to bombard an AI agent with potential injections, capturing the successful ones, and using that data to retrain the model via reinforcement learning. The study revealed that when an automated tool finds a successful exploit, it tends to generate thousands of variations of that same technique rather than discovering new methods. Consequently, vendors who advertise their agents as being tested against thousands of successful attacks may be relying on a dataset that lacks meaningful diversity.

To demonstrate this, the researchers compared two existing automated attack tools, PISmith and RL-Hammer, against the Qwen3-4B open-weight model across 560 practice tasks. PISmith appeared more effective by raw metrics, successfully hijacking 68.75% of tasks compared to RL-Hammer’s 33.39%. However, when the researchers retrained the model using the data collected by these tools, the version trained on PISmith’s larger dataset proved more susceptible to new, unseen attacks. In stress tests, the PISmith-hardened model was hijacked 47.5% of the time, while the RL-Hammer-hardened model was hijacked 27.5% of the time.

The researchers attribute this performance gap to the tools’ inability to maintain true variety. Even when tools include mechanisms to ensure diversity, they often merely alter capitalization or add irrelevant text to existing successful prompts. By the researchers' assessment, both tools ceased providing unique coverage early in the training process, meaning the larger collections of attacks were essentially redundant copies rather than a comprehensive map of potential vulnerabilities.

To address this, the research team developed a system called PIPETTE. This tool categorizes successful attacks by technique and rewards the automated attacker only when it discovers a method that has not been previously identified. Furthermore, the system modifies the agent’s training to verify whether a proposed action aligns with the user’s original intent before execution. This approach is intended to prevent the agent from becoming overly cautious, a common issue where hardened models simply quit when encountering suspicious data, thereby failing to complete legitimate user tasks.

When tested against the AgentDojo benchmark, agents hardened with PIPETTE demonstrated significant improvements. Off-the-shelf injection attacks were largely ineffective, succeeding in less than 0.5% of attempts. Even when attackers were given 250 attempts to adapt their strategies, the success rate remained capped at 10% across the tested models. Crucially, the PIPETTE-hardened Qwen3-4B model maintained high utility, successfully completing 64.95% of tasks, compared to 61.86% for the original, unprotected model.

The researchers contrasted this with existing defensive tools like PromptGuard. While PromptGuard can effectively block malicious inputs, the study found it often disrupts the agent's workflow by discarding necessary data along with the attack. In one scenario, while PromptGuard blocked more attacks than PIPETTE, the agent behind it only completed the user’s legitimate task 16.19% of the time, whereas the PIPETTE-hardened agent completed 58.10% of the same tasks.

The study examined four open-weight models ranging from 4 billion to 27 billion parameters. While the researchers confirmed that their defensive method improved resilience, they noted that it is not a total solution. For instance, even after hardening, a search-based attacker was still able to bypass the defenses on 4 out of 40 tasks when testing the GPT-OSS-20B model.

The authors of the study suggest that the cybersecurity community should shift its focus away from the sheer volume of successful attacks when evaluating AI defenses. Instead, they recommend that vendors and security teams prioritize the number of distinct techniques covered, the agent's performance against persistent, adaptive attackers, and the model's ability to remain functional while under pressure.

While the researchers have not yet validated their technique-coverage grader against human judgment, they emphasize that relying on raw attack counts provides a false sense of security. The findings indicate that without a structured approach to ensure the diversity of training data, automated red-teaming may inadvertently leave critical security gaps unaddressed.

ai
ShareXLinkedInWhatsAppFacebook

More News

view all →
ai

AI agent API security hasn’t caught up with agents already running in production

A significant gap has emerged between the rapid deployment of autonomous AI agents and the security infrastructure required to govern them, according to a recent survey of nearly 9,000 developers, architects, and executives. While 25% of organizations report that they have already integrated AI agents into their production environments, the underlying security frameworks for these tools remain…

security

It’s time to retire vanity security dashboards

Security operations centers and operational technology teams are increasingly being urged to move away from vanity dashboards that prioritize high-level metrics over actionable intelligence. These visual interfaces often display large volumes of data, such as the total number of closed tickets or the count of vulnerabilities, which can create a false sense of security while failing to address…

zero-day

New Unpatched Windows 0-Day FastUnlock Lets Someone Holding a Locked PC Run Programs on It

A newly disclosed zero-day technique, identified as FastUnlock, allows an individual with physical access to a locked Windows computer to execute programs without requiring a valid user login. The discovery was reported by a researcher operating under the handles MSNightmare and Nightmare Eclipse, with additional analysis provided by the endpoint-security firm ThreatLocker. While the…

CVE-2026-104995high

Dontdream BP Profile Search Vulnerability Listed As Exploited In VulnCheck KEV

The dontdream BP Profile Search plugin vulnerability CVE-2026-104995 was identified as exploited by VulnCheck, though it remains absent from the CISA and ENISA catalogues.

CVE-2026-54836critical

YMC Filter SQL Injection Vulnerability Added to VulnCheck KEV

The YMC Filter SQL injection vulnerability, CVE-2026-54836, has been added to the VulnCheck KEV catalogue, though it remains absent from CISA and ENISA lists.

CVE-2026-7542high

Slider Revolution Sensitive Information Exposure Added To VulnCheck KEV

CVE-2026-7542 in the Slider Revolution plugin for WordPress was added to the VulnCheck KEV catalogue 142 days after its initial publication.