Enterprise AI agents are increasingly deployed in document-intensive workflows—from booking flights using passport numbers to processing customs forms with confidential trade data. In such settings, agents must use private information to complete tasks accurately but must never expose that information in responses, because they cannot verify who is actually at the keyboard. These two obligations are in fundamental tension, according to a paper titled TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction published on arXiv.
The Privacy-Task Accuracy Trade-off
The paper introduces TRAP (Task-completion and Resistance to Active Privacy-extraction), a benchmark designed to evaluate the trade-off between task accuracy and privacy leakage. Each scenario in TRAP includes a document containing private information, a task query that requires the agent to invoke the correct tool using private fields, and an attack query that attempts to elicit the same information in natural language. The authors argue that a model capable enough to use private information for task completion can, by the same capability, be induced to reveal it.
TRAP Benchmark: Evaluation of 22 Models
The study evaluated 22 models spanning frontier proprietary and open-source models at multiple scales. According to the paper, all model families exhibit non-trivial leakage. Furthermore, instruction-following ability correlates with leakage rate—meaning models that are better at following instructions are also more prone to leaking private data when attacked. Existing prompt-based defenses reduce leakage but at a significant cost to task accuracy. The authors found that prompt optimization fails to escape this trade-off.
For any softmax-based model, no soft-constraint defense, e.g., prompt-based defenses, can jointly achieve high task success with zero leakage probability.
This impossibility result, the paper states, is not incidental—it is a fundamental limitation of softmax-based architectures.
Structural Private Field Isolation: A Hard Solution
Motivated by this impossibility, the authors propose structural private field isolation, which replaces private fields with hash keys before they reach the model. This approach largely prevents leakage while keeping task accuracy, according to the paper. By removing raw private data from the model's input, the agent can still use the information via hash-keyed representations without ever exposing the underlying values.
Implications for Enterprise AI in Trade and Logistics
For enterprise decision-makers deploying AI in supply chain, logistics, and trade finance, the TRAP benchmark highlights a critical risk: any AI agent handling sensitive documents—such as invoices, customs declarations, or shipping manifests—may inadvertently leak private data. The finding that instruction-following ability correlates with leakage is particularly concerning for highly capable agents used in automation. The proposed structural isolation approach offers a practical path forward, but it requires modifying how private fields are ingested. Enterprise technology leaders should evaluate whether their AI pipelines incorporate similar safeguards, especially when deploying agents in compliance-heavy, document-driven workflows.
In summary, TRAP provides concrete evidence that privacy leakage is not just a theoretical risk but a measurable property of current AI systems. The benchmark and the proposed defense give CTOs and procurement leaders a framework to assess and mitigate these risks in their own deployments.