MIND: Multi-Scale Intent Diffusion for Text-Driven Physics-Based Humanoid Control

Bin Li *1   Ruichi Zhang *2    Han Liang† 3    Jingyan Zhang1   Juze Zhang4   Xin Chen3   Jingya Wang† 1,5
1 ShanghaiTech University    2 University of Pennsylvania    3 Bytedance Seed    4 Stanford University    5 InstAdapt
* Equal contribution        † Corresponding author
Arxiv Code (Coming soon) Video
 

Abstract

Enabling physics-based humanoids to execute diverse behaviors from high-level textual commands remains a significant challenge. Existing methods typically follow either a two-stage paradigm that combines kinematic motion generation with physics-based tracking, or an end-to-end imitation-learning paradigm that directly generates actions from text. However, the former suffers from the inherent domain shift between kinematic generation and physics-based tracking, while the latter struggles with the substantial modality gap between textual commands and low-level actions, limiting effective semantic alignment. Notably, humanoid states encode rich motion dynamics that are more semantically aligned with textual descriptions than low-level actions, making them a natural basis for deriving behavioral intent. Building upon this insight, we propose MIND, a novel end-to-end diffusion framework for text-driven physics-based humanoid control that leverages behavioral intent as a semantic bridge between textual commands and low-level actions. At its core, MIND introduces a multi-scale intent diffusion mechanism, where a holistic intent predictor captures global behavioral dynamics to guide overall behavior synthesis, while an immediate intent predictor provides step-wise, fine-grained signals for local behavior refinement at each diffusion step. This hierarchical intent formulation imposes a structured inductive bias for humanoid control, improving semantic alignment and behavioral naturalness. Furthermore, MIND encodes humanoid states into a latent space to enable more effective semantic intent modeling. Extensive experiments demonstrate that MIND outperforms existing methods and synthesizes coherent, physically plausible, and semantically aligned humanoid behaviors from text commands.

Method Overview

Overview of MIND. Given text commands and humanoid states, MIND models humanoid intent at multiple temporal scales under a within diffusion framework. Specifically, the Holistic Intent Predictor (HIP) captures global behavioral dynamics to provide high-level planning guidance, while the Immediate Intent Predictor (IIP) models step-wise intent from current states for fine-grained action refinement. The predicted holistic and immediate intents, together with the textual commands, are jointly used to condition the Action Diffusion Transformer (ADiT), guiding autoregressive (AR) action generation. Dashed lines indicate components that are used only during training.
Pipeline Overview

Demo Videos

Comparison & Ablation