In-context imitation learning (ICIL) enables robots to learn tasks from just a handful of demonstration prompts, eliminating the need for parameter updates at deployment time and supporting few-shot adaptation to novel tasks. However, according to a new paper on arXiv, recent ICIL methods rely on Transformers, which have computational limitations and tend to underperform when handling longer prompts than those seen during training. To address this, researchers introduce RoboSSM, a scalable recipe for ICIL based on state-space models (SSM).
The Transformer Bottleneck in In-Context Imitation Learning
Transformers are the dominant architecture for ICIL, but they suffer from quadratic computational complexity and poor extrapolation to context lengths beyond those seen during training. The paper notes that this limits their effectiveness for long-horizon tasks where robots must process extended demonstration sequences. These limitations motivated the search for a more efficient and scalable backbone.
RoboSSM and the Longhorn State-Space Model
RoboSSM replaces Transformers with Longhorn, a state-of-the-art SSM that provides linear-time inference and strong extrapolation capabilities. According to the paper, SSMs like Longhorn are well-suited for long-context prompts because they maintain constant memory usage regardless of sequence length, avoiding the quadratic cost of Transformers. This makes RoboSSM particularly effective for tasks requiring many demonstration steps.
Experimental Validation on the LIBERO Benchmark
The researchers conducted diverse experiments on the LIBERO benchmark to evaluate RoboSSM. The results demonstrate that applying SSMs to ICIL achieves improved generalization to both unseen tasks and long-horizon tasks compared to Transformer-based ICIL methods. Specifically, RoboSSM handles longer contexts at test-time more effectively, leading to better performance on tasks that require extended reasoning over multiple steps.
| Aspect | Transformer-based ICIL | RoboSSM (SSM-based) |
|---|---|---|
| Inference complexity | Quadratic | Linear |
| Generalization to longer prompts | Poor | Strong |
| Performance on long-horizon tasks | Underperforms | Improved |
First Demonstration of SSMs for ICIL
The paper states that these results show for the first time that SSMs are an efficient and scalable backbone for ICIL. By leveraging the linear-time inference and extrapolation strengths of Longhorn, RoboSSM opens a new direction for robot learning that can handle more complex, real-world tasks without retraining.
Code Availability
The researchers have released the code for RoboSSM, which is available at the link provided in the paper. This allows other teams to reproduce results and build upon the method for their own robotics applications.
While the work is currently applied to robotics benchmarks, the underlying principle of replacing Transformers with SSMs for in-context learning could extend to other domains where long-sequence processing is critical, such as supply chain automation or logistics planning, though such applications are not explored in the paper.