LIVE · cybersecurity feed
Live wire
CVE-2026-103898 · Wpmet wp_social_login_and_register_social_counter Flaw Exploited Upon DisclosureCVE-2026-116512 · weDevs wp_project_manager Flaw Exploited Before CVE PublicationCVE-2026-119096 · Uncanny Toolkit for LearnDash Improper Authentication Exploited on DisclosureCVE-2026-119977 · WordPress Core Time-of-Check Time-of-Use Vulnerability Exploited Before DisclosureCVE-2026-111357 · Ivanti Connect Secure Vulnerability Exploited Before CVE PublicationCVE-2010-20125 · DedeCMS Vulnerability CVE-2010-20125 Exploited On Day Of DisclosureCVE-2015-20130 · Netsweeper Improper Neutralization Flaw Exploited Upon DisclosureCVE-2025-31704 · Dahua Intelligent Connection Center Flaw Exploited on Day of DisclosureCVE-2026-103039 · Gravity Forms Deserialization Flaw Exploited Upon DisclosureCVE-2026-118539 · Drupal Commerce Core Access Bypass Exploited Before CVE Publication
vulnerability

AWS puts AI vulnerability detection to the test, and false positives pile up

Amazon Web Services (AWS) has introduced a new benchmark designed to evaluate how effectively artificial intelligence models can differentiate between genuine security vulnerabilities and code that merely appears risky but is actually safe. The "Deception Benchmark" aims to address the challenge of high false-positive rates in AI-driven vulnerability detection, which can lead to increased…

ZeroDay News ·

Source: Help Net Security

Amazon Web Services (AWS) has introduced a new benchmark designed to evaluate how effectively artificial intelligence models can differentiate between genuine security vulnerabilities and code that merely appears risky but is actually safe. The "Deception Benchmark" aims to address the challenge of high false-positive rates in AI-driven vulnerability detection, which can lead to increased workload, alert fatigue, and reduced trust in AI findings.

The benchmark, which AWS is making publicly available, comprises 14,822 code samples across 16 programming languages and more than 70 Common Weakness Enumeration (CWE) categories. Of these, 9,695 samples are scored, including 6,988 code-level challenges and 2,707 environment-gated challenges. The remaining 5,127 samples are unscored and mixed with the scored ones to prevent models from optimizing specifically for the benchmark.

Unlike other cybersecurity benchmarks that focus on tasks like exploit generation or capture-the-flag challenges, AWS's benchmark specifically tests a model's ability to understand whether a dangerous code pattern is truly exploitable, considering surrounding protections. For instance, an environment-gated challenge might present code that suggests an SSRF attack is possible, but a Kubernetes Network Policy in the deployment environment prevents it. Models must account for such external security controls.

AWS generated these samples by testing against existing "frontier models" and iteratively making them more difficult when models classified them correctly. Samples that were too easily classified were excluded. According to AWS, the process of generating and refining these samples consumed tens of billions of tokens.

To ensure the accuracy of the labels, AWS employs a rigorous review process. Multiple independent reviewers check each label without knowledge of other reviewers' decisions or the original labeling rationale. Disagreements are further reviewed, and unresolved cases are escalated to human reviewers. This process continues until fewer than 3% of scored samples remain contested, with a target of less than 1% after human review. Any samples that remain disputed are moved to the unscored set. A human review of 100 randomly selected scored samples reportedly found no labeling errors.

AWS evaluated 12 models from five providers using the benchmark. The results indicated that none of the tested configurations met the minimum production-use threshold of false-positive and false-negative rates below 10%. With direct prompting, models generally identified nearly all real vulnerabilities but incorrectly flagged between 41% and 99% of safe code, resulting in precision ranging from 52% to 71%.

When models were prompted to provide proof that a vulnerability could be exploited, false positives decreased significantly, by 17 to 74 percentage points. However, this approach also led to an increase in missed vulnerabilities, with false-negative rates ranging from 7% to 44%. Models demonstrated the most difficulty when external security controls rendered suspicious-looking code unexploitable.

It is important to note that AWS tested general-purpose models with single-turn prompts, not purpose-built security systems that incorporate tools and multiple validation steps. Therefore, these results do not directly reflect the performance of complete security products. The benchmark is designed to measure the model's inherent understanding of code and its protections, rather than the effectiveness of multi-step agentic workflows often used in production tools.

vulnerabilityaicloud
ShareXLinkedInWhatsAppFacebook

More News

view all →
CVE-2026-103898high

Wpmet wp_social_login_and_register_social_counter Flaw Exploited Upon Disclosure

The Wpmet wp_social_login_and_register_social_counter plugin vulnerability CVE-2026-103898 was identified as exploited on the same day it was published.

CVE-2026-116512high

weDevs wp_project_manager Flaw Exploited Before CVE Publication

The weDevs wp_project_manager plugin was identified as under exploitation one day before the CVE was officially published, leaving users with no patch window.

CVE-2026-119096high

Uncanny Toolkit for LearnDash Improper Authentication Exploited on Disclosure

The Uncanny Toolkit for LearnDash plugin was identified as exploited on the same day its CVE was published, leaving no window for remediation.

CVE-2026-119977high

WordPress Core Time-of-Check Time-of-Use Vulnerability Exploited Before Disclosure

The WordPress Core TOCTOU race condition, CVE-2026-119977, was identified as exploited one day before its official publication, leaving no patch window for administrators.

CVE-2026-111357high

Ivanti Connect Secure Vulnerability Exploited Before CVE Publication

The Ivanti Connect Secure missing authentication vulnerability was identified as exploited two days before its official CVE publication, leaving no patch window for administrators.

breach

Inside ARTEX: AI Hacking Tools, Exposed Servers and South Korean Bank Breaches

A series of cyberattacks between late September and early October 2026 resulted in the compromise of seven South Korean financial institutions. The breaches led to the exposure of sensitive personal, employment, income, and loan-related data belonging to approximately 66,000 individuals and 2,200 corporations. Security researchers have linked these incidents to the use of an open-source…