Enterprises deploying AI for reasoning tasks rely on difficulty signals like pass@k to curate training data, train verifiers, and guide reinforcement learning. A new study reveals that this canonical metric systematically underestimates the difficulty of the hardest examples, potentially distorting model evaluation and improvement.
"We show this proxy has a persistent blind spot on its hardest stratum" — according to the authors Zhou, Luca, Shah, Sajel, Rodolà, Emanuele, Dessì, and Roberto, in their paper on arXiv.
The Blind Spot in Pass@K
The pass@k metric measures the fraction of sampled reasoning chains that reach the correct answer. When every sampling seed fails (pass@k=0%), the example is typically labelled as extremely hard. However, the researchers demonstrate that this conclusion can be premature. Across eight free-form math cells from GSM8K and MATH, tested on four open-weight models, 10.3% to 22.9% of such "zero-pass" examples are actually solvable when using a different generation strategy under the same compute budget.
Methodology: Activation Grafting
Instead of relying on random sampling, the team employed a deterministic regime: greedy decoding plus five cheap residual-stream perturbations applied via activation grafting. This technique intervenes on internal representations rather than modifying the decoding method. The grafted perturbations are mechanistically distinct — the cross-kind fix-set Jaccard index never exceeds 0.47 across all twelve cells tested. Greedy decoding alone solves at most 6% on these math cells, underscoring that the gains come from the combination of perturbations.
| Measure | Value Range |
|---|---|
| Examples recovered (pass@k=0 → solved) | 10.3% – 22.9% |
| Greedy-only coverage on same cells | ≤ 6% |
| Cross-kind fix-set Jaccard (mechanistic distinctness) | ≤ 0.47 in every setup |
Key Findings
Recovery scales with the additional budget across perturbations. The authors emphasize that activation grafting is used purely as a diagnostic and diversification tool. The recovered items show that the pass@k=0% stratum is structurally identifiable in the residual stream — the unmodified model can reach these solutions under ordinary inference, but not via the naive sampling approach.
Implications for AI Evaluation
For enterprise AI teams, these results highlight a critical limitation of pass@k as a difficulty signal. If difficult examples are misclassified, RL with verifiable rewards may be trained on skewed data, and synthetic curricula may omit recoverable cases. The study suggests that complementary methods — such as activation grafting — could improve the robustness of difficulty estimation, leading to more efficient data curation and stronger verifier training.
While the research focuses on math reasoning, the blind spot likely extends to other domains where pass@k is used. CTOs and AI procurement leaders should scrutinize how model evaluation metrics are constructed and whether they might understate actual model capability on the hardest cases. The work is available on arXiv under a Creative Commons license.
