Topic
ai evaluation
IHBench: Evaluating Post-Interruption Recovery in Voice Agents with Structured Workflows
IHBench, a new benchmark from researchers including Salimi et al., evaluates how voice agents recover after interruptions in structured enterprise workflows. The benchmark tests 27 audio-language models from OpenAI, Google, and the open-weight community, finding that closed-weight models are consistently more robust, degrading 3.3x more slowly in long conversations.
RTSGameBench Benchmark Tests Strategic Reasoning in Vision-Language Models
A new benchmark called RTSGameBench evaluates strategic reasoning in vision-language models (VLMs) using the real-time strategy game Beyond All Reason. The benchmark includes diagnostic mini-games, diverse matchup structures, and a self-evolving generation framework. Initial tests show state-of-the-art VLMs struggle with tighter coordination, multiagent tasks, and increased scale.
The Scaffold Effect: How Prompt Framing Skews AI Evaluation in Clinical Vision-Language Models
A study on arXiv evaluating 12 open-weight vision-language models (VLMs) on clinical neuroimaging datasets found that up to 58% of apparent multimodal performance gains are due to prompt framing rather than genuine reasoning. The researchers identified a 'scaffold effect' where merely mentioning MRI availability in the task prompt accounts for 70-80% of F1 improvement, even when no imaging data is present. Expert evaluation also revealed fabrication of neuroimaging-grounded justifications, raising concerns about the reliability of VLM evaluations in clinical settings.
CombEval: A Framework for Evaluating Combinatorial Counting in Large Language Models
CombEval is a dynamic benchmark for evaluating combinatorial counting in large language models. It uses typed Cofola specifications to generate problems with verified answers. Tests on 11 LLMs reveal persistent failures on ordered objects, indistinguishable elements, and nested dependencies.
Process-Level Evaluation of Web Agents Reveals Hidden Performance Differences in AI Systems
Researchers introduce WebStep, a benchmark of 1,800 task instances that evaluates web agents at the process level using semantic state tracking. Key findings show that agents with similar success rates have divergent process metrics, with OpenAI CUA outperforming Qwen3.5 on commit actions but underperforming on filtering on the Housing website.