MixReact: Physics-Based Reactive Humanoid Control from Multimodal Cues
via a Discrete–Continuous Mixture of Transformers

Bin Li* 1Xing Zhu* 1Jingya Wang† 1,2

1 ShanghaiTech University2 InstAdapt

* Equal contribution.   † Corresponding author.

Abstract

Humanoid agents in everyday environments are expected to react to diverse cues as humans do, with semantically appropriate and physically executable behaviors. Existing cue-driven reaction generation methods support only a narrow range of cue modalities and are limited to purely kinematic generation, leaving the resulting reactions susceptible to physical artifacts. To this end, we present MixReact, the first unified framework for physics-based reactive humanoid control from text, audio, and video cues, integrating a Discrete–Continuous Mixture of Transformers (DisCo-MoT) for state prediction with a History-aware Action Transformer (HAT) for physical execution. DisCo-MoT couples a discrete masked-modeling branch with a continuous autoregressive diffusion branch via one-way mixture attention within a MoT architecture. Guidance from discrete context mitigates collapse into averaged behaviors in one-to-many reaction generation, while direct prediction in continuous state space bypasses the decoding bottleneck of discrete representations. HAT then maps the predicted next state and observed execution history to executable actions, enabling robust closed-loop control in physics-based simulation. Experiments across all three modalities demonstrate consistent improvements over the baselines in fidelity, diversity, and physical consistency, while ablations validate the complementary contributions of the components. Our code will be released to facilitate future research.

Motivation

Everyday cues.
More than one response.

“Congratulations!” may elicit a bow, a handshake, or a celebratory jump. MixReact models these distinct choices and executes reactions in physics-based simulation.

Method

Plan globally. Adapt continuously.

DisCo-MoT unites discrete reaction planning and continuous state prediction. HAT closes the loop with physical execution.

Persistent reaction context

A complete discrete plan maintains behavioral structure across prediction windows.

Fine-grained state prediction

One-way attention guides continuous denoising while protecting discrete representations.

History-aware control

HAT uses predicted states and observed history to adapt actions during execution.

Comparison

Expressive reactions.
Stronger physical consistency.

MixReact produces diverse, balanced reactions across text, audio, and video cues. The side-by-side comparisons show each method under physical execution.

Text, audio, and video comparisons play in sequence. Enable sound for narration.

Text- and audio-driven reactions · ReactMotionNet
CueMethodFID ↓Diversity →MModality ↑Floating ↓mmSkating ↓mmJerk ↓mm / frame3
—Phys-GT0.000±0.0001.359±0.046—13.559±0.0001.064±0.0001.976±0.000
TextReactMotion1.478±0.0021.176±0.0191.035±0.02129.213±0.2692.523±0.0152.279±0.010
ReactMotion++1.379±0.0051.301±0.0111.208±0.03519.035±0.6591.560±0.0804.372±0.354
CLoSD1.543±0.0101.091±0.0170.443±0.03416.493±0.1191.341±0.0293.045±0.022
MixReact (Ours)1.328±0.0051.356±0.0031.320±0.00313.827±0.0860.917±0.0282.626±0.036
AudioReactMotion1.515±0.0111.175±0.0391.165±0.09428.193±0.2572.352±0.0952.280±0.024
ReactMotion++1.386±0.0101.292±0.0091.290±0.05223.534±0.7912.369±0.1357.662±0.623
CLoSD1.543±0.0111.090±0.0050.451±0.04916.552±0.0701.329±0.0223.080±0.042
MixReact (Ours)1.326±0.0061.361±0.0031.324±0.00313.920±0.0610.987±0.0282.645±0.034
Video-driven reactions · ViMo
CueMethodFID ↓Diversity →MModality ↑Floating ↓mmSkating ↓mmJerk ↓mm / frame3
—Phys-GT0.000±0.0001.402±0.006—22.378±0.0003.851±0.00018.623±0.000
VideoHERO1.036±0.0031.314±0.0060.786±0.01038.036±1.0009.642±0.1264.477±0.050
NOUS1.026±0.0031.321±0.0070.773±0.00539.004±0.58410.959±0.0784.661±0.047
HERO++0.862±0.0041.367±0.0051.005±0.00728.173±0.2114.600±0.09810.242±0.229
NOUS++0.856±0.0051.366±0.0071.002±0.01429.471±0.1444.683±0.17711.217±0.091
CLoSD0.888±0.0021.362±0.0140.843±0.02923.669±0.1965.021±0.1468.869±0.268
MixReact (Ours)0.706±0.0031.382±0.0031.244±0.00320.908±0.1853.081±0.0627.505±0.075

Mean ± 95% confidence interval across 10 evaluation seeds. ↓ Lower is better; ↑ higher is better; → closer to Phys-GT is better. Bold and underline indicate the best and second-best results. Phys-GT denotes physically tracked reference motions.

User study · baseline comparison

20 participants · 30 cues per modality · ratings from 1 (very poor) to 5 (very good)

MixReact receives higher user ratings across all three cue modalities.

Ablation

Complementary by design.

Discrete context supports reaction diversity; continuous prediction refines state evolution. One-way attention protects global behavioral modeling from local denoising.

MixReact alongside variants without the discrete branch, continuous branch, or one-way attention.

Ablation study · contributions of key components
CueMethodFID ↓Diversity →MModality ↑Floating ↓mmSkating ↓mmJerk ↓mm / frame3
TextMixReact (Ours)1.328±0.0051.356±0.0031.320±0.00313.827±0.0860.917±0.0282.626±0.036
w/o Discrete branch1.514±0.0111.178±0.0071.001±0.06010.808±0.0630.133±0.0061.025±0.028
w/o Continuous branch1.423±0.0201.306±0.0201.280±0.01414.751±0.2081.898±0.0802.768±0.108
w/o One-way attention1.354±0.0261.361±0.0111.329±0.02214.863±0.2441.392±0.0895.066±0.159
AudioMixReact (Ours)1.326±0.0061.361±0.0031.324±0.00313.920±0.0610.987±0.0282.645±0.034
w/o Discrete branch1.616±0.0141.011±0.0380.816±0.08911.258±0.0070.131±0.0010.905±0.031
w/o Continuous branch1.417±0.0181.311±0.0271.285±0.01214.813±0.06421.880±0.0962.796±0.246
w/o One-way attention1.362±0.0091.363±0.0121.327±0.01415.160±0.2701.543±0.1505.330±0.297
VideoMixReact (Ours)0.706±0.0031.382±0.0031.244±0.00320.908±0.1853.081±0.0627.505±0.075
w/o Discrete branch0.831±0.0191.347±0.0151.101±0.02526.158±0.1445.502±0.12516.886±0.201
w/o Continuous branch0.848±0.0121.353±0.0061.269±0.02418.521±0.6713.305±0.0844.659±0.218
w/o One-way attention0.722±0.0161.375±0.0121.229±0.02120.249±1.0093.040±0.1219.601±0.411

Mean ± 95% confidence interval. Bold and underline indicate the best and second-best results. Diversity is evaluated by proximity to Phys-GT. All values are reproduced from the paper.

User study · ablation variants

20 participants · 30 cues per modality · ratings from 1 (very poor) to 5 (very good)

Removing the discrete branch produces the largest rating decrease across all three modalities.

One-to-many

One cue. Many possible reactions.

Given the same cue, MixReact generates distinct reactions while maintaining coherent behavior throughout physical execution.

Closed-loop performance

Faster than real time, on average.

34.51 steps / second

1.15× the nominal 30 Hz simulation rate.
Single RTX 3090 · FP32 · full closed loop