Enterprise technology leaders deploying large language models (LLMs) for structured data tasks face a critical risk: the models do not know what they do not know. According to a preprint on arXiv by Dasula, Akshat, Desikan, Prasanna, and Srivastava, LLMs applied to clinical tabular data display epistemically vacuous confidence — their verbalized confidence remains nearly constant regardless of whether accuracy is 49% or 75.3%, tracking prompt format rather than prediction quality.
The Inverse Difficulty Effect
The study compared Qwen 2.5 7B with XGBoost on a prediction task using attribution divergence analysis. A striking finding is the inverse difficulty effect: when XGBoost is 99% correct, the LLM accuracy drops to 64.8%. Conversely, when XGBoost is moderately uncertain, the LLM matches it (73.8% vs. 73.1%). This pattern suggests that the LLM fails to align its confidence with actual task difficulty, a serious concern for high-stakes applications.
Super-Additive Interventions Without Training
Two orthogonal interventions proved effective. The researchers found that few-shot examples and SHAP-derived feature evidence act as super-additive interventions. Together, they reduced the Attribution Disagreement Score (ADS) from 1.54 to 0.38 and improved accuracy from 49% to 75.3% — all without model retraining. SHAP (SHapley Additive exPlanations) provides feature importance values, which when combined with few-shot examples, dramatically align the LLM's predictions with reliable signals.
Cross-Model Calibrator Replaces Uninformative Confidence
The team developed a cross-model calibrator that uses attribution divergence signals to estimate LLM reliability per patient. This method does not require access to model internals or repeated inference. It reduces the expected calibration error from 0.254 to 0.080, replacing uninformative verbalized confidence scores with patient-specific reliability estimates.
| Metric | Before Calibration | After Calibration |
|---|---|---|
| Expected calibration error | 0.254 | 0.080 |
| Accuracy (few-shot + SHAP) | 49% | 75.3% |
| Attribution Disagreement Score (ADS) | 1.54 | 0.38 |
Implications for Enterprise AI Deployment
These findings frame LLM usage on structured data as a cold start problem, where models lack genuine epistemic self-awareness. For CTOs and technology buyers, this study underscores that off-the-shelf LLM confidence scores cannot be trusted without external validation. The proposed calibrator offers a practical path to reliable predictions without retraining costs. While tested on clinical data, the methodology is applicable to any structured tabular domain — including supply chain forecasting, trade finance risk assessment, and customs classification — domains where overconfident models could lead to costly errors.
