Topic
coding agents
New StaminaBench Benchmark Reveals Coding Agents Fail After 5-6 Turns
Researchers introduce StaminaBench, a benchmark that measures how many consecutive interaction turns coding agents can handle. Testing six harnesses and seven open-source LLMs over 100-turn scenarios, they found all models fail within 5-6 turns. Providing test feedback improved passed turn count by up to 12x, highlighting the importance of iterative testing.
ScaleWoB Framework Synthesizes Realistic Environments to Evaluate GUI Agents at Scale
ScaleWoB is a new framework that generates high-fidelity synthesized interactive environments for evaluating GUI agents across mobile, desktop, and automotive platforms. It includes 100+ environments and 1000+ verifiable tasks. Experiments on five state-of-the-art mobile GUI agents show an average success rate of only 27.92%, compared to 92.08% for humans, highlighting substantial room for improvement.
DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents
Researchers present DualGauge, an automated framework for jointly evaluating correctness and security of code generated by LLMs from natural-language specifications. A benchmark of 307 tasks across three languages shows that even the strongest models achieve under 15% joint security-functionality success, while factors like scale and instruction tuning do not reliably improve outcomes. Three leading agentic coding systems also show no advantage over direct generation.