Organizations across industries — including supply chain, logistics, and trade technology — are increasingly turning to large language models (LLMs) to replace human participants in A/B tests, hoping to experiment faster and at lower cost. But a new statistical analysis reveals that raw LLM predictions may severely underestimate real human responses, recovering only 39% of the human treatment effect in a benchmark study.
The research, Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference, was published on arXiv (arXiv:2606.17165) by authors Persson, Joel; Schultzberg, Mårten; and Ankargren, Sebastian. It provides a rigorous framework to determine when an LLM-based estimate can validly substitute for a human-population causal effect.
The Surrogacy Framework
The core insight is that assuming distributional equivalence between LLM and human outcomes — the condition that would make any standard estimator valid — is unrealistic. Instead, the authors adapt surrogate endpoint theory from clinical trials to the LLM context. They show that calibrating LLM outcomes to human outcomes can identify the average treatment effect under two conditions: surrogacy and comparability.
- Surrogacy: The LLM outcome captures all information about the human outcome that is relevant to the treatment effect.
- Comparability: The relationship between LLM and human outcomes is the same across treatment and control groups, and across the observed sample.
These conditions are jointly weaker than distributional equivalence, making the approach practical. The paper also presents a falsification test for surrogacy and a bound on worst-case bias from limited overlap between LLM and human samples.
Empirical Findings: Upworthy Headlines
To test the framework, the authors applied it to A/B tests on Upworthy headlines — a common benchmark in experimentation. The results starkly illustrate the pitfall of relying on raw LLM predictions:
| Metric | Value |
|---|---|
| Raw LLM predictions' recovery of human treatment effect | 39% |
| Improvement after nonparametric calibration | Closes the gap entirely |
According to the paper, raw LLM predictions recover only 39% of the human treatment effect, while nonparametric calibration successfully closes the gap. This demonstrates that calibration is essential before LLM-based A/B testing can be trusted for decision-making.
Implications for Enterprise Experimentation
For CTOs and digital transformation leaders in logistics and trade tech, the findings carry a clear warning: LLM-based A/B testing yields correct results only by assumption, whereas A/B testing on humans is correct by design. The required assumptions are hardest to justify precisely where LLM-based testing promises the greatest benefit — i.e., when human data is scarce, costly, or slow to collect.
The paper identifies several design variables that affect surrogacy quality:
- LLM choice (model architecture and training data)
- Prompting strategy (how the task is framed)
- Temperature setting (controlling output stochasticity)
Stochasticity inherent to LLMs can weaken surrogacy for identification while introducing bias and variance during estimation. However, using an average over multiple LLM draws per unit as the surrogate mitigates these issues.
"A/B testing on LLMs yields correct results only by assumption, whereas A/B testing on humans is correct by design." — Authors of the study
Limitations and Validation
The paper acknowledges that surrogacy conditions are not universally satisfied. It provides a falsification test to check surrogacy in specific applications. For long-term outcomes, the challenge is compounded because the surrogacy relationship may shift over time. The authors recommend sizing human pilot studies for validation, rather than entirely replacing human experiments.
For enterprise buyers considering LLM-based experimentation platforms — whether for optimizing logistics interfaces, trade documentation workflows, or supply chain dashboards — this research underscores that a human validation step remains indispensable. The statistical framework offers a path to more reliable LLM surrogates, but only when calibration and surrogacy checks are rigorously applied.