Embodied agents that must navigate and interact with humans rely on dialog to ensure safety and effectiveness. However, training such agents requires large amounts of data, which is often scarce. A new paper on arXiv addresses this challenge for the DialNav framework, which evaluates the full dialog--execution loop in photorealistic indoor navigation. The researchers propose an automatic generation pipeline that creates the RAINbow dataset, a large-scale training dataset with 238K episodes, and combine it with two complementary advances to achieve substantial improvements in navigation success rate.
The Challenge of Training Data Scarcity
DialNav, introduced by Han et al. in 2025, provides a holistic evaluation framework for dialog-driven navigation. However, its performance has been limited by a critical scarcity of training data: only 2K episodes were available. This constraint hindered the agent's ability to generalize across different environments and dialog scenarios.
The RAINbow Dataset Generation Pipeline
The researchers' automatic generation pipeline converts existing Vision-and-Language Navigation (VLN) datasets into multi-turn dialog episodes, creating a cost-efficient and high-quality dataset. The resulting RAINbow dataset contains 238K episodes, a 119-fold increase over the original DialNav training set. This pipeline addresses the data bottleneck without requiring manual annotation.
Dual-Strategy Training and Localization Model
To unlock the full potential of the larger dataset, the team introduced two additional advances:
- Dual-Strategy Training: A navigation training scheme designed to align the training process with the dynamic dialog-navigation loop, ensuring the agent learns to handle interactive dialog.
- Localization Model: A model that leverages knowledge from VLN tasks to improve the agent's ability to determine its position within the environment.
These components work together to turn the large-scale dataset into actionable performance gains.
Results and Benchmark Performance
When evaluated on the DialNav benchmark, the combined system substantially outperforms the baseline, establishing a new state of the art. The key results are summarized in the table below:
| Metric | Baseline (estimated) | Our Model | Improvement |
|---|---|---|---|
| Success Rate (Val Seen) | ~30.81 | 58.24 | +89% |
| Success Rate (Val Unseen) | ~14.53 | 29.05 | +100% |
Baseline values are derived from the reported improvements: success rates of 58.24 (+89%) and 29.05 (+100%).
On the Val Seen split (environments seen during training), the model achieved a success rate of 58.24, an 89% improvement over the baseline. On the Val Unseen split (novel environments), it reached 29.05, a 100% improvement. These results demonstrate that the automatic augmentation pipeline and complementary techniques significantly enhance the agent's ability to generalize.
Future Directions and Implications
The paper's authors — Han, Leekyeung, Jung, Sangwon, Hyunji, Jeong, Jinseong, Kim, Minyoung, and Seo, Paul Hongsuck — have shown that automatic dialog augmentation can overcome data scarcity in embodied navigation tasks. While this research is focused on indoor navigation, the principles of automatic dataset generation and training alignment could be applied to other domains where dialog and physical interaction are required. The code, data, and media associated with this article are available through arXiv.
