Multimodal large language models (MLLMs) are increasingly deployed to act on visual information, but a new benchmark reveals a significant gap between perception and context-specific action. The ROSE (Reference-conditioned Oddity and Symbolic Execution) benchmark, described in a paper on arXiv, systematically tests how reliably a model can turn the same visual evidence into the action required by the current task context.
The Perception-to-Action Gap
According to the paper, ROSE holds the visual scene fixed while varying region constraints and required symbolic outputs. Through coupled counting and coordinate-action tasks, the benchmark tests whether models can infer an implicit majority reference and act on resulting fine-grained visual evidence under changing contexts. Across nine recent MLLMs, performance drops by as much as 44.5 percentage points from counting-oriented tasks to region-conditioned action. In contrast, human performance stands at 98.8%. The gap persists even on paired scenes and regions for which the same model returns the correct count.
How ROSE Works
The benchmark uses a controlled setup: the visual scene remains constant, but the task switches between counting objects and performing actions based on coordinates. This isolates the model's ability to adapt shared visual evidence to different task contexts. Global-click and matched local controls show that coordinate grounding explains only part of the loss, revealing a distinct, model-dependent bottleneck in turning shared visual evidence into context-specific actions.
Implications for Enterprise AI
For enterprise technology leaders evaluating MLLMs for automation tasks — such as visual inspection in logistics or document processing in trade — the ROSE findings highlight a critical reliability issue. A model that correctly counts items in a scene may fail to act on that same scene when required to, for example, select a specific object or region. The 44.5 percentage point drop underscores that current MLLMs lack robust context-sensitive decision-making, a prerequisite for deployment in safety-critical or high-stakes supply chain operations.
Model-Level Variability
The paper tested nine recent MLLMs, but does not name specific models in the abstract. The model-dependent bottleneck suggests that different architectures or training regimes handle the perception-to-action transition differently. Enterprises should demand benchmarks like ROSE that isolate this specific capability, rather than relying solely on general vision-language accuracy metrics.
"How reliably can a model turn the same visual evidence into the action required by the current context?" — This question, posed by the authors, is central to the ROSE benchmark and to enterprise AI deployment.
As MLLMs move into production environments, benchmarks that expose the perception-to-action gap become essential procurement criteria. ROSE provides a controlled, reproducible method for assessing this capability, contributing to more trustworthy AI for trade, logistics, and beyond.