Large Language Models (LLMs) that utilize persistent memory for personalization and task enhancement face significant risks concerning the inappropriate disclosure of sensitive information, a problem identified as a challenge to "contextual integrity." This issue becomes more pronounced as LLMs are increasingly deployed as autonomous agents making decisions on behalf of users.
Recent research highlights the difficulty LLMs have in controlling information flow from their memory based on the specific context of a task. A benchmark called CIMemories was developed to evaluate this problem, using synthetic user profiles with over 100 attributes per user and diverse task contexts where each attribute's relevance varies.
Evaluations using CIMemories revealed that current frontier models exhibit up to a 69% rate of attribute-level violations, meaning they inappropriately leak information. While lower violation rates can be achieved, this often comes at the expense of overall task utility. The research also found that these violations accumulate over time and across multiple interactions. For instance, as usage increased from 1 to 40 tasks, violations in a model identified as GPT-5 rose from 0.1% to 9.6%. When the same prompt was executed five times, violations reached 25.1%, indicating arbitrary and unstable behavior where different attributes were leaked for identical prompts.
Attempts to mitigate these issues through "privacy-conscious prompting" were largely ineffective. Models tended to overgeneralize, either sharing all information or none, rather than making nuanced, context-dependent decisions about what information is appropriate to disclose. These findings suggest that the problem stems from fundamental limitations in the LLMs' ability to reason contextually, rather than merely requiring better prompting strategies or increased model scaling.
Further research has explored methods to instill contextual integrity in LLMs through reasoning and reinforcement learning. One approach involves explicitly prompting LLMs to reason about contextual integrity when deciding what information to disclose. This method was extended by developing a reinforcement learning (RL) framework designed to further embed the necessary reasoning into models.
Using a synthetic dataset of 700 examples featuring diverse contexts and information disclosure norms, this method demonstrated a substantial reduction in inappropriate information disclosure while maintaining task performance across various model sizes and families. Crucially, the improvements achieved on this synthetic dataset transferred to established contextual integrity benchmarks, such as PrivacyLens, which uses human annotations to evaluate privacy leakage in AI assistant actions and tool calls. This suggests that a combination of explicit reasoning and reinforcement learning could be a viable path toward addressing the contextual integrity challenges in LLMs.
