The prevailing metric for coding agents—fraction of tasks solved—masks a critical weakness: their inability to sustain performance over long sessions. A new benchmark, StaminaBench, directly stress-tests this stamina by requiring agents to handle up to 100 consecutive change requests. According to a paper by researchers Sobal, Vlad, Yang, Shuo, Zhang, Yuting, Xia, Wei, and Soatto (affiliated with an unnamed institution), all tested models failed within 5 to 6 turns when operating without feedback, confirming that "vibe-coding-style programming without thorough testing produces bugs."
How StaminaBench Works
StaminaBench simulates real-world coding sessions where an agent receives a series of procedural change requests. The agent implements a REST API server and modifies it across a tunable number of turns. The benchmark generates tests fully programmatically—without any LLM involvement—ensuring reproducibility. Change sequences are drawn from either a hardcoded or an LLM-driven sampler, both constrained to a structured action space. The agent and server run in an isolated environment and communicate via HTTP, making the test language-agnostic and black-box.
Key Findings from 100-Turn Trials
The researchers evaluated six agent harnesses paired with seven open-source LLMs across 20 scenarios of 100 turns each. The results reveal stark contrasts in performance:
| Condition | Passed Turns (Avg.) | Improvement vs. No Feedback |
|---|---|---|
| No feedback | 5–6 turns | — |
| With test feedback & retry | Up to 12× more | 12× |
| Strong model, best harness | — | Up to 6× vs. worst harness |
| Weak model, any harness | ~5–6 turns | No improvement |
As the table shows, allowing the agent to receive test feedback and retry increased passed turn count by up to 12×. However, the quality of the agent harness mattered greatly: stronger LLMs exhibited up to a 6× gap in performance between their best and worst harness, while weaker models failed regardless of the harness.
Implications for Enterprise Software Development
For enterprise technology leaders evaluating AI coding assistants, StaminaBench highlights a crucial limitation. Even state-of-the-art open-source LLMs cannot maintain reliable coding output over extended interaction sessions without robust test feedback loops. The benchmark's code and generated tasks are publicly available at https://github.com/... (paper URL: https://arxiv.org/abs/2606.19613), enabling further research into multi-turn agent behavior.
The findings suggest that organizations deploying coding agents should prioritize integration with continuous testing pipelines and be wary of trusting agents for long, unsupervised coding sessions. As the paper notes, all tested models "fail within 5-6 turns," underscoring the gap between current AI capabilities and the demands of real-world software maintenance.