Threat actors are increasingly leveraging artificial intelligence to accelerate their offensive operations, from reconnaissance and vulnerability discovery to social engineering. In response, Rapid7's Red Team has developed a formalized, multi-agent AI system designed to mirror their end-to-end penetration testing methodology. This production system, built over the past year, aims to automate repetitive tasks while retaining human oversight for critical decision-making points.
The motivation behind building this AI-driven system stems from the labor-intensive nature of penetration testing. Tasks such as enumerating attack surfaces, tracing data flows, and documenting findings are structured and repeatable, making them suitable for AI agents. This allows human testers to focus on higher-level judgment, such as determining the next steps, assessing exploitability, and understanding business impact – areas where experience, context, and human insight are crucial and where AI models currently struggle.
Furthermore, developing this system provides Rapid7's internal security team with direct architectural insights into how AI agents behave in adversarial contexts. This understanding is vital for assessing and securing Rapid7's own AI-powered products, offering a proactive approach to internal security.
The architecture is designed as a coordinated team of specialist agents, rather than a single autonomous entity. An orchestrator manages the engagement, assessing the current state, assigning tasks to appropriate specialist agents (e.g., enumeration, code review, dynamic testing, reporting), and processing their results. This supervisor-style orchestration separates decision-making from execution, enhancing predictability, auditability, and control, which are essential for operations in sensitive environments.
A key design principle was reverse-engineering the agent architecture from the daily task lists of human red teamers. The sequence of tasks, decision branching, and triggers for returning to earlier phases in real engagements formed the specification for the orchestration logic. This methodological approach, rather than a purely technical one, was central to the system's success.
Early lessons learned highlighted the inefficiency of presenting an entire engagement scope to a single AI agent. Due to the finite context windows of large language models, complex applications could overwhelm the analysis, leading to shallow and scattered results. The solution was deliberate scope decomposition, breaking down the engagement into discrete, manageable chunks. Each chunk undergoes the full architectural process independently, ensuring that each component receives the agent's full analytical attention and enabling parallelization and clear progress tracking.
Recognizing that real penetration tests are non-linear, the system incorporates feedback loops. The orchestrator manages progression gates and feedback triggers, allowing the engagement to loop back to earlier phases when new, actionable data emerges, such as code reviews revealing new endpoints or dynamic testing uncovering previously unseen attack surfaces. This creates a directed graph with re-entry points, avoiding a rigid waterfall approach.
Significant design effort was dedicated to safety guardrails, given the potential for AI agents to operate in malicious contexts. A tiered safety model includes scope enforcement to ensure actions remain within authorized boundaries, action classification to categorize proposed tests as non-destructive, destructive, or ambiguous, and a human-in-the-loop default for approving dynamic tests. While the system is designed for eventual semi-automation of low-risk actions, state-modifying operations will continue to require human approval, with the level of autonomy being context-dependent.
Practical considerations, such as the cost of AI token consumption, also shaped design decisions. Mechanical tasks that do not require complex AI reasoning, such as DNS lookups, header checks, and certificate enumeration, were replaced with deterministic scripts and microservices. This significantly reduced token consumption during enumeration-heavy phases, allowing the AI to focus its reasoning capabilities on analysis, correlation, and judgment, demonstrating that not every step in an AI workflow necessitates AI.
