Sim to real · Whole-body control · October 2026
One policy moves
the whole robot.
A learned whole-body controller for MILO, a wheeled robot with a lift, a pitching torso, a 3-DoF neck and two 7-DoF arms. Give it where the head and both wrists should be; it moves the base, lift, torso and arms together to put them there. Trained with PPO in Isaac Lab 3 on the Newton physics backend. At run time it reads only what the real robot measures: its posture reference is solved by inverse kinematics from the targets, not taken from the simulator.
Results
99.4 % of episodes
at full difficulty.
4096 evaluation episodes of 20 s, curriculum at its widest, every domain randomisation on (masses, hand payload, centre of mass, joint friction, actuator gains, gravity feed-forward error, command delay, lift speed, base acceleration and traction, shoves). An episode succeeds if the robot neither falls, hits a joint end-stop, holds a joint at its torque limit for 0.25 s, collides with itself or loses a target, and its mean errors stay under 5 cm and 0.25 rad for the wrists and the head.
| Policy | Success | Episodes | Wrist pos. | Wrist rot. | Head pos. | Head rot. |
|---|---|---|---|---|---|---|
| Deployable: reference posture from IK on the targets | 99.4 % | 4096 | 1.4 cm | 0.047 rad | 2.0 cm | 0.038 rad |
| Same network, simulator reference posture (upper bound) | 99.9 % | 4096 | 1.1 cm | 0.047 rad | 0.7 cm | 0.039 rad |
Mean errors over each episode after the first second. Failed episodes, by first failed criterion: early termination 9, wrist pos 10, head pos 8. Mean change of the policy's action per 20 ms step: 0.198 (actions are in units of the per-joint scale). All numbers are in simulation; nothing here has run on the robot yet. Evaluation ↗
Design
What the robot
is asked, and how it answers.
The structure follows Dyna Robotics' Dyna-2.1 controller: task-space targets in, joint targets and base velocity out, priorities in the reward weights, gains, masses, friction and delays randomised. The controller is adapted to MILO's hardware interface, read from the MILO code base.
Rewards: position and orientation tracking (wrists and head equal), footprint, posture (towards the human-like reference), mechanical power, torque, action rate, drift (motion the target does not ask for), support margin of the zero-moment point in the wheel triangle, tilt, self-collision, joint-range proximity. Terminations: fall (tilt over 0.25 rad), joint end-stop, sustained torque saturation, collision, lost target, time-out.
Targets come from 125 k feasible keyframes: inside the joint ranges with margin, collision-free, holdable within the torque limits, wrists in front of the body and below 1.85 m, and with the head looking at a hand in 70 % of them. They are chained into human-paced behaviours (working along a table, reaching up to a shelf, reaching low, carrying while driving, side reaches) and a curriculum grows them from small motions around the robot to the full workspace.
Rollouts
Behaviours.
Segments of evaluation episodes (full difficulty, randomisation on), rendered at 1x.
Engineering log
What broke,
and what fixed it.
| Symptom | Cause | Fix |
|---|---|---|
| Arm roll joints oscillate at zero action | explicit 200 Hz PD on low-inertia joints | MIT gains as the solver's implicit drive (the motor loop runs at kHz) |
| Base does not move | MuJoCo takes the larger friction of a contact pair | low floor and wheel friction; traction lives in the base model |
| Joints hit their end-stops | ready pose 0.08 rad from the J2/J4 stops; torso sagging | IK ready pose with 0.3 rad clearance; keyframes 0.15 rad and setpoints 0.12 rad from every stop |
| Torso saturates under exploration | one action scale for all joints | per-joint action scales |
| Curriculum never starts near the robot | shelf and side poses are far from the default by definition | each goal = default + level x (library pose - default) |
| Commands dither up to 0.8 rad per step | the hardware slew hides dithering from tracking | action-rate penalty x10 |
| A 99 % policy degrades with more training | running observation normaliser drifts with the policy | normaliser folded into the first layer, then frozen |
| Base heading drifts, arms compensate | no feed-forward for the footprint; weak yaw loop | footprint tracking layer under the policy (reference velocity + odometry feedback) |
| Neck and torso learn stuck offsets | PPO settles in coordinated local optima of a 20-DoF redundant body | neck IK for head orientation; posture reward; a reference posture input |
| High shelf and side reaches never converge | actions clipped at 1.5 x 0.8 rad: setpoints limited to +-1.2 rad around the ready pose | wider action clip; velocity feed-forward as MILO's MIT stream sends it |
| Policy without the reference posture stays at 26 % at full difficulty | the posture is what resolves the redundancy | solve it: whole-body IK from the targets (15 constraints, 16 joints) gives it to within 0.003 rad |
| Training degrades as targets grow | 8 % of keyframes needed more than the torso's torque limit to hold | keyframes must be holdable at <= 45 % (torso) / 60 % (arms) of the clamp |
Limitations
What is measured.
What is not.
Code: the wbc/ harness in the Rehearse repository (private), with the model assumptions in docs/MODEL.md. Isaac Lab 3.0 with the Newton (MuJoCo-Warp) backend, RSL-RL 5.5. Videos are kinematic replays of simulator states rendered in MuJoCo.