Human-robot co-creation faces a fundamental challenge: connecting high-level semantic understanding of music with the real-time physical constraints of performance. A new framework called Co-policy, described in a paper published on arXiv, addresses this by decomposing the task into three modular components: semantic intent grounding, constrained musical variation, and visuomotor execution.
Technical Architecture: Separation of Concerns
Co-policy employs pre-inference semantic anchors and a fine-tuned Qwen-vl planner (F-Qwen) to transform speech, live musical seeds, and visual observations into structured co-creation plans. The planner handles semantic understanding, converting ambiguous human input into actionable musical goals. For low-latency execution, Co-policy introduces a Gaussian-Mixture Visuomotor Policy (GMP), implemented as a conditional mixture-density policy. GMP maps target notes and visual context to multimodal robot actions in a single forward pass, bypassing slower iterative methods.
Performance Validated in Chime Experiments
The researchers conducted real-robot chime experiments, along with ablations and expert evaluations, to compare Co-policy against diffusion-policy and ablated baselines. According to the paper, Co-policy demonstrated improved intent alignment, execution accuracy, and response frequency. Unlike robotic playback systems that merely reproduce user-specified notes, Co-policy generates complementary musical responses under both musical and physical constraints, enabling true co-creation.
Implications for Embodied AI Co-Creation
By grounding generative models in physical action, Co-policy offers a template for embodied human-AI collaboration beyond music. The framework's separation of semantic planning from motor control could be adapted to manufacturing, logistics, or service robotics, where real-time responsiveness and physical feasibility are equally critical. The authors note that "physically grounded action generation" is a key requirement for embodied co-creation, a principle that extends to any domain where humans and robots collaborate dynamically.
