Persistent reaction context
A complete discrete plan maintains behavioral structure across prediction windows.
1 ShanghaiTech University2 InstAdapt
Humanoid agents in everyday environments are expected to react to diverse cues as humans do, with semantically appropriate and physically executable behaviors. Existing cue-driven reaction generation methods support only a narrow range of cue modalities and are limited to purely kinematic generation, leaving the resulting reactions susceptible to physical artifacts. To this end, we present MixReact, the first unified framework for physics-based reactive humanoid control from text, audio, and video cues, integrating a Discrete–Continuous Mixture of Transformers (DisCo-MoT) for state prediction with a History-aware Action Transformer (HAT) for physical execution. DisCo-MoT couples a discrete masked-modeling branch with a continuous autoregressive diffusion branch via one-way mixture attention within a MoT architecture. Guidance from discrete context mitigates collapse into averaged behaviors in one-to-many reaction generation, while direct prediction in continuous state space bypasses the decoding bottleneck of discrete representations. HAT then maps the predicted next state and observed execution history to executable actions, enabling robust closed-loop control in physics-based simulation. Experiments across all three modalities demonstrate consistent improvements over the baselines in fidelity, diversity, and physical consistency, while ablations validate the complementary contributions of the components. Our code will be released to facilitate future research.
Motivation
“Congratulations!” may elicit a bow, a handshake, or a celebratory jump. MixReact models these distinct choices and executes reactions in physics-based simulation.
Method
DisCo-MoT unites discrete reaction planning and continuous state prediction. HAT closes the loop with physical execution.
A complete discrete plan maintains behavioral structure across prediction windows.
One-way attention guides continuous denoising while protecting discrete representations.
HAT uses predicted states and observed history to adapt actions during execution.
Comparison
MixReact produces diverse, balanced reactions across text, audio, and video cues. The side-by-side comparisons show each method under physical execution.
Text, audio, and video comparisons play in sequence. Enable sound for narration.
| Cue | Method | FID ↓ | Diversity → | MModality ↑ | Floating ↓mm | Skating ↓mm | Jerk ↓mm / frame3 |
|---|---|---|---|---|---|---|---|
| — | Phys-GT | 0.000±0.000 | 1.359±0.046 | — | 13.559±0.000 | 1.064±0.000 | 1.976±0.000 |
| Text | ReactMotion | 1.478±0.002 | 1.176±0.019 | 1.035±0.021 | 29.213±0.269 | 2.523±0.015 | 2.279±0.010 |
| ReactMotion++ | 1.379±0.005 | 1.301±0.011 | 1.208±0.035 | 19.035±0.659 | 1.560±0.080 | 4.372±0.354 | |
| CLoSD | 1.543±0.010 | 1.091±0.017 | 0.443±0.034 | 16.493±0.119 | 1.341±0.029 | 3.045±0.022 | |
| MixReact (Ours) | 1.328±0.005 | 1.356±0.003 | 1.320±0.003 | 13.827±0.086 | 0.917±0.028 | 2.626±0.036 | |
| Audio | ReactMotion | 1.515±0.011 | 1.175±0.039 | 1.165±0.094 | 28.193±0.257 | 2.352±0.095 | 2.280±0.024 |
| ReactMotion++ | 1.386±0.010 | 1.292±0.009 | 1.290±0.052 | 23.534±0.791 | 2.369±0.135 | 7.662±0.623 | |
| CLoSD | 1.543±0.011 | 1.090±0.005 | 0.451±0.049 | 16.552±0.070 | 1.329±0.022 | 3.080±0.042 | |
| MixReact (Ours) | 1.326±0.006 | 1.361±0.003 | 1.324±0.003 | 13.920±0.061 | 0.987±0.028 | 2.645±0.034 |
| Cue | Method | FID ↓ | Diversity → | MModality ↑ | Floating ↓mm | Skating ↓mm | Jerk ↓mm / frame3 |
|---|---|---|---|---|---|---|---|
| — | Phys-GT | 0.000±0.000 | 1.402±0.006 | — | 22.378±0.000 | 3.851±0.000 | 18.623±0.000 |
| Video | HERO | 1.036±0.003 | 1.314±0.006 | 0.786±0.010 | 38.036±1.000 | 9.642±0.126 | 4.477±0.050 |
| NOUS | 1.026±0.003 | 1.321±0.007 | 0.773±0.005 | 39.004±0.584 | 10.959±0.078 | 4.661±0.047 | |
| HERO++ | 0.862±0.004 | 1.367±0.005 | 1.005±0.007 | 28.173±0.211 | 4.600±0.098 | 10.242±0.229 | |
| NOUS++ | 0.856±0.005 | 1.366±0.007 | 1.002±0.014 | 29.471±0.144 | 4.683±0.177 | 11.217±0.091 | |
| CLoSD | 0.888±0.002 | 1.362±0.014 | 0.843±0.029 | 23.669±0.196 | 5.021±0.146 | 8.869±0.268 | |
| MixReact (Ours) | 0.706±0.003 | 1.382±0.003 | 1.244±0.003 | 20.908±0.185 | 3.081±0.062 | 7.505±0.075 |
Mean ± 95% confidence interval across 10 evaluation seeds. ↓ Lower is better; ↑ higher is better; → closer to Phys-GT is better. Bold and underline indicate the best and second-best results. Phys-GT denotes physically tracked reference motions.
20 participants · 30 cues per modality · ratings from 1 (very poor) to 5 (very good)
Ablation
Discrete context supports reaction diversity; continuous prediction refines state evolution. One-way attention protects global behavioral modeling from local denoising.
MixReact alongside variants without the discrete branch, continuous branch, or one-way attention.
| Cue | Method | FID ↓ | Diversity → | MModality ↑ | Floating ↓mm | Skating ↓mm | Jerk ↓mm / frame3 |
|---|---|---|---|---|---|---|---|
| Text | MixReact (Ours) | 1.328±0.005 | 1.356±0.003 | 1.320±0.003 | 13.827±0.086 | 0.917±0.028 | 2.626±0.036 |
| w/o Discrete branch | 1.514±0.011 | 1.178±0.007 | 1.001±0.060 | 10.808±0.063 | 0.133±0.006 | 1.025±0.028 | |
| w/o Continuous branch | 1.423±0.020 | 1.306±0.020 | 1.280±0.014 | 14.751±0.208 | 1.898±0.080 | 2.768±0.108 | |
| w/o One-way attention | 1.354±0.026 | 1.361±0.011 | 1.329±0.022 | 14.863±0.244 | 1.392±0.089 | 5.066±0.159 | |
| Audio | MixReact (Ours) | 1.326±0.006 | 1.361±0.003 | 1.324±0.003 | 13.920±0.061 | 0.987±0.028 | 2.645±0.034 |
| w/o Discrete branch | 1.616±0.014 | 1.011±0.038 | 0.816±0.089 | 11.258±0.007 | 0.131±0.001 | 0.905±0.031 | |
| w/o Continuous branch | 1.417±0.018 | 1.311±0.027 | 1.285±0.012 | 14.813±0.0642 | 1.880±0.096 | 2.796±0.246 | |
| w/o One-way attention | 1.362±0.009 | 1.363±0.012 | 1.327±0.014 | 15.160±0.270 | 1.543±0.150 | 5.330±0.297 | |
| Video | MixReact (Ours) | 0.706±0.003 | 1.382±0.003 | 1.244±0.003 | 20.908±0.185 | 3.081±0.062 | 7.505±0.075 |
| w/o Discrete branch | 0.831±0.019 | 1.347±0.015 | 1.101±0.025 | 26.158±0.144 | 5.502±0.125 | 16.886±0.201 | |
| w/o Continuous branch | 0.848±0.012 | 1.353±0.006 | 1.269±0.024 | 18.521±0.671 | 3.305±0.084 | 4.659±0.218 | |
| w/o One-way attention | 0.722±0.016 | 1.375±0.012 | 1.229±0.021 | 20.249±1.009 | 3.040±0.121 | 9.601±0.411 |
Mean ± 95% confidence interval. Bold and underline indicate the best and second-best results. Diversity is evaluated by proximity to Phys-GT. All values are reproduced from the paper.
20 participants · 30 cues per modality · ratings from 1 (very poor) to 5 (very good)
One-to-many
Given the same cue, MixReact generates distinct reactions while maintaining coherent behavior throughout physical execution.
1.15× the nominal 30 Hz simulation rate.
Single RTX 3090 · FP32 · full closed loop