Enterprise AI agents deployed in supply chain and logistics systems often forget critical information—shipment updates, customs clearance status, or trade document amendments—leading to costly operational failures. A new architectural study from the arXiv preprint server (ID 2606.15903) systematically analyzes this problem across 13 system configurations, finding that forgetting failures are predominantly the cause of production errors, while existing benchmarks measure only recall (the ability to retrieve stored facts).
The Forgetting Problem in Agent Memory
The research, authored by Dongxu Yang, examines where an LLM (large language model) is placed within an agent's memory pipeline. The pipeline consists of a recall plane (retrieving stored facts, extensively benchmarked) and a control plane (mutating facts via supersedure, release, or purge—largely untested). The study defines ForgetEval, a 1000-case templated suite plus a 385-case adversarial layer (132 hand-crafted and 253 LLM-drafted, oracle-validated) scored by deterministic substring match. The benchmark includes a 77-case external-authored subset from four blind contributors that replicated the canonicalization asymmetry and amplified the joint-placement lift by +27.8 percentage points.
Three Placement Regimes Compared
The study identifies three placement regimes for LLM intervention and evaluates them on a 385-case adversarial surface:
| Regime | Strengths | Weaknesses | Accuracy (Overall) | Cost per 385-case run | Latency per case |
|---|---|---|---|---|---|
| Deterministic primitives | Lexical/temporal categories | Fails canonicalization (5% on identifier-obfuscation, 0% on cross-lingual) | Not explicitly provided overall | Not provided | 64–191 ms/case |
| Inscribe-time LLM | Recovers canonicalization (100%) | Cannot help intent-aware deletion (0% on prefix-collision and compound-fact) | Not explicitly provided overall | Not provided | Not provided |
| Mutation-time hook | Recovers intent-aware deletion (78–85%) and brightens nearly all categories simultaneously | None reported | 91.7–93.2% | $0.17 | 2.3 s/case (mutation latency) |
The mutation-time hook placement—where an LLM directly modifies stored facts at mutation time—achieves the highest overall accuracy at a modest cost, though its latency is approximately 12x slower than deterministic primitives (2.3s vs. 64–191ms). The deterministic path remains unchanged for recall.
Adapter Protocol Enables Integration
To allow heterogeneous memory stores to participate in the benchmark, the study introduces a six-method Adapter Protocol with honest N/A scoring, implementable in 130 lines of code. Admission is corroborated by a 10-annotator inter-annotator agreement (IAA) with Fleiss' kappa = 0.958, indicating near-perfect consistency.
Implications for Enterprise AI in Trade and Logistics
For CTOs and technology leaders deploying AI agents in supply chain—such as automating customs documentation, tracking cargo, or managing trade finance—the finding that forgetting failures dominate recall failures is a critical design consideration. The control-plane placement of LLM intervention directly affects whether an agent can handle canonicalization failures (e.g., recognizing that "PO-1234" and "Purchase Order 1234" refer to the same document) and intent-aware deletion (e.g., removing only expired trade agreements that share a prefix with active ones).
The mutation-time hook approach, despite higher latency, may be justified for high-stakes transactions where accuracy is paramount. The research provides a structured way to evaluate these trade-offs using ForgetEval, which is released under the MIT license along with all adapters.
Enterprise buyers evaluating AI agent platforms—especially those from vendors integrating LLMs into logistics execution systems—should request evidence of performance on forgetting benchmarks, not just recall benchmarks. The study demonstrates that production failures are predominantly forgetting failures, making control-plane design a key architectural decision for reliable AI in global trade.
