A new class of agentic review systems is emerging as a potential remedy to the pressure placed on peer review by AI-assisted research. But how should these systems be evaluated? A preprint by Nguyen et al. on arXiv (arXiv:2606.19749) takes on this question, testing two open-source systems (OpenAIReview and coarse), one proprietary system (Reviewer3), and a zero-shot baseline across six LLMs spanning frontier and efficient models. The study evaluates whether AI reviews on ICLR/NeurIPS papers track with paper quality as approximated by external signals such as citations and acceptance decisions, and whether systems can catch errors with known ground truth.
Pairwise Accuracy: How Well Do AI Reviews Track Human Judgments?
The first benchmark measures pairwise accuracy: given two papers, does the AI review correctly identify which has higher quality based on citations and acceptance decisions? According to the study, every system performed above chance. The best configuration was OpenAIReview paired with GPT-5.5, achieving 83.0% pairwise accuracy. This result shows that current AI review systems can align reasonably well with human quality signals, making them potentially useful for triage or initial screening.
Error Detection: A Perturbation Benchmark
The second benchmark constructs a perturbation benchmark that injects four categories of errors into papers across eight arXiv subject classes. The goal is to measure detection recall. The strongest configuration (OpenAIReview + GPT-5.5) caught 71.6% of injected errors, leaving substantial room for improvement. However, the study found that the union of detections across all six models reached 83.3% recall. This indicates that different models detect different errors, and better harness design could potentially increase performance.
User Feedback: Positive Sentiment but Nitpicks
Beyond these benchmarks, the researchers studied a public deployment of OpenAIReview with real users. Votes on its comments skewed positive at a ratio of 1.44 to 1. The most common complaints were about false positives and minor nitpicks. This suggests that while users appreciate the system's contributions, they are frustrated by inaccuracies and overly detailed critiques.
Implications for Enterprise AI Agent Evaluation
For enterprise technology leaders evaluating AI agents for tasks such as document review, quality assurance, or compliance checking, this study offers a methodological template. The combination of pairwise accuracy against ground truth and recall on injected errors provides a rigorous framework for assessing agent reliability. The finding that different LLMs catch different errors suggests that ensemble approaches may be more robust than relying on a single model. The negative user feedback on false positives also highlights the need to tune sensitivity. While the study focuses on academic peer review, the same evaluation principles apply to any domain where AI agents are used to assess or critique content produced by humans or AI.
