A browser-based game designed to test human oversight of AI coding agents suggests that users frequently approve dangerous commands, with approximately one in three malicious requests slipping past human reviewers. The game's creator, Belgian software developer Alex Wauters, developed the tool after observing the impracticality of requiring users to approve every command in an AI agent's workflow, noting that many developers resort to disabling permission prompts entirely.
The game simulates permission requests from an AI coding agent, similar to those from tools like Claude Code, challenging players to approve or deny commands within a 60-second time limit. Correct decisions earn points, while approving risky commands or denying safe ones reduces a player's score. Wauters launched the game in late May, coinciding with a blog post highlighting the human-in-the-loop challenge.
Data from over 40,000 game runs, encompassing 409,000 approved or denied commands, indicates significant human error. Scope violations, such as requests to access Kubernetes configuration files or AWS credentials, were the most commonly missed malicious commands, approved in 35% of instances. These types of requests could easily lead to the exfiltration of sensitive data.
While obviously destructive commands, like `rm -rf /` or recursively granting full permissions to the root directory, were frequently caught, other subtle threats were often overlooked. These included `curl` requests to unknown APIs and typosquatted package names, which were missed almost as often as scope violations. The single most frequently missed potentially malicious command was `npm run analyze`, approved nearly 65% of the time, despite its ability to execute arbitrary code defined in a project's `package.json` file. Wauters noted that the game's history log explicitly shows the script's contents, suggesting players may not be closely reviewing the provided context.
Wauters emphasized that limited context makes approval decisions difficult. While coding agents offer some information, commands like `npm run analyze` can be modified to run any payload. To ensure safety, human reviewers would need to investigate all files an agent intends to call, which can be a significant time investment. This issue is not confined to game scenarios; Anthropic, the developer of Claude Code, reported in May that telemetry shows users approve approximately 93% of permission prompts, indicating that approval fatigue is a real problem.
The high volume of permission requests can lead to user fatigue, causing developers to become less diligent in their supervision over time. This suggests a need for improved permission models for AI agents. Wauters advocates for running AI coding models in sandboxed environments, such as cloud-based devcontainers, and implementing hooks to contextualize and intercept potentially malicious actions before automatic approval.
Anthropic has introduced an "auto mode" for Claude Code to combat approval fatigue, which delegates some command-approval decisions to a model-based classifier. This system reportedly catches about 83% of "overeager behaviors," though 17% still bypass it. Anthropic stresses that auto mode is an additional layer of defense within a sandbox, not a replacement for one.
Wauters concludes that the emergence of AI coding agents introduces a new set of attack vectors, necessitating increased awareness of risks and robust strategies to mitigate them. He believes that tooling needs to be made easier to enhance the safety of these systems, rather than solely relying on human-in-the-loop oversight as a complete solution.
