Amazon Web Services (AWS) has introduced a new benchmark designed to evaluate how effectively artificial intelligence models can differentiate between genuine security vulnerabilities and code that merely appears risky but is actually safe. The "Deception Benchmark" aims to address the challenge of high false-positive rates in AI-driven vulnerability detection, which can lead to increased workload, alert fatigue, and reduced trust in AI findings.
The benchmark, which AWS is making publicly available, comprises 14,822 code samples across 16 programming languages and more than 70 Common Weakness Enumeration (CWE) categories. Of these, 9,695 samples are scored, including 6,988 code-level challenges and 2,707 environment-gated challenges. The remaining 5,127 samples are unscored and mixed with the scored ones to prevent models from optimizing specifically for the benchmark.
Unlike other cybersecurity benchmarks that focus on tasks like exploit generation or capture-the-flag challenges, AWS's benchmark specifically tests a model's ability to understand whether a dangerous code pattern is truly exploitable, considering surrounding protections. For instance, an environment-gated challenge might present code that suggests an SSRF attack is possible, but a Kubernetes Network Policy in the deployment environment prevents it. Models must account for such external security controls.
AWS generated these samples by testing against existing "frontier models" and iteratively making them more difficult when models classified them correctly. Samples that were too easily classified were excluded. According to AWS, the process of generating and refining these samples consumed tens of billions of tokens.
To ensure the accuracy of the labels, AWS employs a rigorous review process. Multiple independent reviewers check each label without knowledge of other reviewers' decisions or the original labeling rationale. Disagreements are further reviewed, and unresolved cases are escalated to human reviewers. This process continues until fewer than 3% of scored samples remain contested, with a target of less than 1% after human review. Any samples that remain disputed are moved to the unscored set. A human review of 100 randomly selected scored samples reportedly found no labeling errors.
AWS evaluated 12 models from five providers using the benchmark. The results indicated that none of the tested configurations met the minimum production-use threshold of false-positive and false-negative rates below 10%. With direct prompting, models generally identified nearly all real vulnerabilities but incorrectly flagged between 41% and 99% of safe code, resulting in precision ranging from 52% to 71%.
When models were prompted to provide proof that a vulnerability could be exploited, false positives decreased significantly, by 17 to 74 percentage points. However, this approach also led to an increase in missed vulnerabilities, with false-negative rates ranging from 7% to 44%. Models demonstrated the most difficulty when external security controls rendered suspicious-looking code unexploitable.
It is important to note that AWS tested general-purpose models with single-turn prompts, not purpose-built security systems that incorporate tools and multiple validation steps. Therefore, these results do not directly reflect the performance of complete security products. The benchmark is designed to measure the model's inherent understanding of code and its protections, rather than the effectiveness of multi-step agentic workflows often used in production tools.
