Vision-language models (VLMs) can now produce natural-language reasoning traces, but these traces often leave the supporting image regions implicit, making verification and supervision difficult. A new paper on arXiv presents a solution called visually grounded thinking, where models interleave thoughts with explicit point or box groundings of the visual evidence used at each step.
The Problem with Implicit Visual Reasoning
According to the arXiv paper, while recent VLMs can generate step-by-step reasoning, the intermediate connections to the image are often unstated. This lack of explicit grounding means the reasoning is hard to verify and difficult to supervise. The researchers argue that visual thinking should not only sound right—it should show its evidence.
Visually Grounded Thinking: A New Approach
The authors propose a reasoning process in which the model expresses intermediate reasoning in language while simultaneously grounding key objects in the image regions they refer to. This is achieved through point groundings (for localizing objects in counting tasks) and box groundings (for spatial relationships). The approach allows the model to make its visual reasoning transparent and auditable.
The Training Pipeline: Scalable Synthesis and Reinforcement Learning
To train this behavior, the team built a scalable synthesis pipeline. It first distills correct visual reasoning traces from a teacher model, then extracts the visual objects required by those traces. A SAM3-based agent is used to ground the objects, deriving aligned point and box supervision from the resulting masks. Further, the paper introduces grounding-aware reinforcement learning, which combines answer correctness rewards with dense grounding rewards. These dense rewards score whether generated object references match the correct image evidence, encouraging the model to attend to the right regions.
Performance Gains Across Benchmarks
The researchers tested their method on two counting benchmarks and four spatial reasoning benchmarks. Adding visually grounded thinking to Gemma3-4B-IT consistently improved performance over both the original model and a non-grounded thinking baseline. On spatial reasoning, the visually grounded thinking 4B models match, and in some cases surpass, Gemma3-27B-IT from the same model family.
| Model Variant | Counting Tasks | Spatial Reasoning Tasks |
|---|---|---|
| Gemma3-4B-IT (original) | Baseline performance | Baseline performance |
| Gemma3-4B-IT + non-grounded thinking | Improved over original | Improved over original |
| Gemma3-4B-IT + visually grounded thinking | Consistently improved over baselines | Matches or surpasses Gemma3-27B-IT |
| Gemma3-27B-IT | Best on some tasks | Strong performance |
Analysis by the authors shows that point grounding is well suited to counting tasks, while box grounding benefits most from explicit grounding rewards on spatial tasks. Overall, the results indicate that VLMs think better when their intermediate thoughts are tied to the image regions that make them true.
Implications for Enterprise AI
Although the paper is academic research, its findings have practical relevance for enterprise AI applications that depend on visual reasoning—such as automated inspection, inventory counting, and spatial layout analysis. By making reasoning explicit and grounded, these models become more reliable and easier to audit, which is critical for regulated industries. The ability to match a 27B-parameter model with a 4B model also suggests significant efficiency gains, potentially reducing computational costs for deployment.