Autonomous vehicles rely on motion forecasting to predict future trajectories of dynamic agents like cars, pedestrians, and cyclists. However, most existing approaches assume a closed-world setting with a fixed object taxonomy and high-quality perception inputs. In reality, new object classes can emerge over time, and perception is often imperfect. A team of researchers has introduced class-incremental motion forecasting, a novel setting where new object classes are sequentially introduced, and future trajectories are predicted directly from camera images.
The Problem: Fixed Object Taxonomies Limit Real-World Deployment
Current motion forecasting systems are typically trained on a static set of object classes. When a new type of agent appears — say, an electric scooter or a delivery robot — the model either fails to recognize it or must be retrained from scratch. Retraining can cause catastrophic forgetting, where the model loses accuracy on previously learned classes. According to the arXiv paper, this limitation stems from assuming a closed-world setting with perfect perception, which is not realistic in dynamic environments.
A First-of-Its-Kind Framework for Incremental Learning
The researchers propose the first end-to-end framework for class-incremental motion forecasting. The system generates motion forecasting pseudo-labels for known classes and matches them with 2D instance masks from an open-vocabulary segmentation model. A 3D-to-2D keypoint voting mechanism filters out inconsistent and overconfident predictions. Additionally, a query feature variance-based replay strategy selects informative past sequences to preserve prior knowledge.
This approach allows the model to adapt to newly introduced classes without catastrophic forgetting. The framework is designed to work directly from camera images, avoiding the need for expensive LiDAR data.
Evaluation and Results
The team evaluated their framework on two large-scale autonomous driving datasets: nuScenes and Argoverse 2. The results show that the method successfully preserves performance on known classes while effectively adapting to novel ones. They also demonstrated zero-shot transfer to real-world driving scenarios, indicating the framework's generalizability. Furthermore, the system extends naturally to both open- and closed-loop end-to-end class-incremental planning on nuScenes and NeuroNCAP, a neural driving simulator.
| Aspect | Traditional Closed-World Approach | Proposed Class-Incremental Framework |
|---|---|---|
| Object taxonomy | Fixed, known in advance | Sequential introduction of new classes |
| Perception requirement | High-quality (often LiDAR) | Camera images (direct) |
| Handling new classes | Retraining or failure | Adapts without forgetting |
| Forgetting mitigation | Not considered | Pseudo-labels, keypoint voting, replay |
| Evaluation datasets | Typically one dataset | nuScenes, Argoverse 2, real-world transfer |
Implications for Autonomous Driving and Beyond
While the paper focuses on autonomous driving, the underlying technology has potential applications in logistics and supply chain. Autonomous trucks, warehouse robots, and delivery drones operate in environments where new object types (e.g., new packaging, different vehicle models) are common. A system that can incrementally learn these classes without retraining could reduce downtime and improve safety. The researchers have stated that code and models will be made publicly available, which may accelerate industry adoption.
In summary, this work addresses a critical gap in motion forecasting by enabling continuous learning in dynamic environments. The combination of pseudo-labels, open-vocabulary segmentation, and selective replay offers a practical path toward more adaptable autonomous systems.