Researchers have developed a new diagnostic method, "perturbation probing," that can identify the specific neural pathways responsible for safety behaviors in large language models (LLMs). This technique, detailed in an academic paper on arXiv titled "Perturbation Probing: A Two-Pass-per-Prompt Diagnostic for FFN Behavioral Circuits in Aligned LLMs," reveals that LLM safety mechanisms are often concentrated in a very small fraction of the model's neurons, making them potentially fragile and susceptible to manipulation.
The research indicates that on the open-source LLM Qwen3-4B, merely 50 out of 350,208 feed-forward neurons—approximately 0.014% of the total—are causally linked to the model's safety refusal template. Disabling these 50 neurons altered the response format for 80% of 520 standard harmful-prompt benchmarks and was replicated on 200 prompts of a second benchmark. For the smaller Qwen3.5-2B model, only 20 neurons were sufficient to eliminate false agreement in multi-turn conversations, reducing this behavior from 36.7% to 0% across 30 questions.
This concentration suggests that an aligned LLM's refusal behavior is not a robust, distributed defense but rather resides in a "thin template layer." This thin layer could be disabled by an attacker with internal manipulation capabilities or even shifted during routine optimization processes. This finding implies that relying solely on internal model alignment for safety is akin to depending on a single perimeter firewall, which is structurally insufficient. A more robust approach would involve a defense-in-depth strategy, incorporating external content filters and runtime guardrails.
Beyond identifying these critical neurons, the perturbation probing method also generates a diagnostic metric called the FFN/Skip ratio. This single number, computable in seconds per model, predicts how easily a model's safety circuit can be steered with minimal modifications. Across 13 tested models, this ratio accounted for 81% of the variance in the vulnerability of their safety behaviors to small, targeted changes. This makes the FFN/Skip ratio a candidate for a quantitative safety fragility score, enabling security teams to compare models' alignment robustness without extensive adversarial red-teaming.
The researchers propose that perturbation probing can serve as a pre-deployment diagnostic, allowing security teams to assess the fragility of a model's safety mechanisms before production. The same toolkit that exposes fragility can also be used for repair; for instance, amplifying just 10 identified neurons on a small model improved factual self-correction from 52% to 88% on 200 TruthfulQA prompts without retraining.
This research builds upon previous work on logit-gap steering, which demonstrated how alignment guardrails could be bypassed by closing a measurable gap in a model's output scores. The current study further investigates where this alignment resides within the model and its concentration. The goal is to empower the AI and security community to build LLMs with measurable, auditable, and reinforceable safety properties, rather than merely asserted ones.
The researchers encourage the integration of fragility diagnostics into evaluation pipelines and recommend external content filters and inline guardrails for organizations deploying LLMs. The study utilized publicly available open-weight models under their respective licenses for local mechanistic and safety evaluation, reporting aggregate rates and model-internal measurements without releasing harmful generations or attack artifacts.
