Imitation learning for humanoids

What the problem looks like for a humanoid

The authors of EgoDex describe a well-known data scarcity problem for imitation learning for manipulation: unlike natural language and 2D computer vision, there is no Internet-scale corpus of data for dexterous manipulation. ^[6]

The TWIST2 authors argue that humanoid robotics lacks data collection frameworks as effective as those behind recent vision-language-action models for bimanual manipulation. ^[3] The EgoHumanoid authors add that learning humanoid loco-manipulation, the close coordination of whole-body locomotion and dexterous manipulation, is bottlenecked by the scarcity of diverse, large-scale demonstration data. ^[5]

In TWIST2, the human teleoperator together with the motion retargeter produces pairs of visual observation, proprioceptive state and motion command, and these are then used to train an autonomous policy such as a Diffusion Policy. ^[3]

Collecting demonstrations on the robot

TWIST2 is a worked example of whole-body teleoperation for data collection. It uses PICO VR hardware to stream human body motion in real time, retargets it to a Unitree G1, and does not need a motion-capture studio. ^[3]

Several design decisions in that system are worth copying or at least weighing:

• Give the operator the robot's own view. The authors found that neck degrees of freedom are essential for effective, long-horizon teleoperation, so they added a neck with yaw and pitch. ^[3]
• Provide depth. In their user study, operators without stereo vision tended to grasp higher than the real object position, and operators without the neck could not see objects outside a fixed field of view. ^[3]
• Make one person enough. The handheld controllers are programmed so that the demonstrator can start, pause and end a session without an assistant. ^[3]
• Interpolate on resume. When a pause ends, the system interpolates from the last robot pose to the current target pose, to avoid a sudden jump. ^[3]
• Filter afterwards. A post-processing interface splits long recordings into episodes, reduces idle actions and removes failed episodes. ^[3]

The authors report that an expert could collect around 100 successful bimanual pick-and-place demonstrations in 20 minutes. ^[3] They also name limits: PICO whole-body pose estimation is less accurate than costly motion capture, particularly at the elbows and knees where no tracker is worn. ^[3]

Structuring the policy around a whole-body controller

TWIST2 splits control into two levels: a low-level motion tracker that is task-agnostic and a high-level visuomotor policy that outputs the commands the tracker follows. The same tracker serves during teleoperation and during autonomous execution. ^[3]

For the commands, TWIST2 uses relative root translations and rotations, so the system does not rely on accurate global state estimation, and it includes whole-body joint positions rather than reducing the lower body to a root velocity. The authors say this enables tasks such as legged manipulation. ^[3]

For the high-level policy, they feed in the history of commands rather than raw robot states. They give two reasons: it decouples the high-level policy from the low-level controller, and it avoids dependence on noisy raw states. ^[3] The policy predicts a chunk of 64 actions, covering 2 seconds of future commands, and 48 of them are executed. ^[3] During training they inject 10% Gaussian noise into the proprioceptive inputs so that the policy relies more on vision. ^[3]

NVIDIA's repository documents whole-body humanoid control through an embodiment tag for the Unitree G1 and the GEAR-SONIC controller. In that workflow the vision-language-action model predicts compact latent action tokens, which a learned whole-body controller decodes into full-body joint commands, including legs, arms and hands. ^[2]

Learning from human video and robot-free demonstrations

EgoScale pretrains a vision-language-action policy on 20,854 hours of egocentric human video, supervised with relative wrist motion and retargeted dexterous hand joint actions. ^[4] A smaller mid-training stage of aligned human and robot data follows, and the authors say it anchors the representation to robot sensing and control. ^[4] They report that average task completion rises from 0.30 at 1k hours of human data to 0.71 at 20k hours. ^[4]

The EgoScale authors also test transfer to a substantially different robot platform, with a low-DoF tri-finger hand. They report that adding a small amount of play data from that robot during mid-training enables strong transfer, outperforming mid-training or pre-training alone. ^[4]

EgoScale is not the only source of egocentric human data. The EgoDex dataset has 829 hours of egocentric video with paired 3D hand and finger tracking, covering 194 tabletop tasks. ^[6]

EgoHumanoid co-trains on human demonstrations and a smaller amount of robot data, aligning the views by reprojecting human images to the robot camera viewpoint and aligning actions through delta end-effector poses and discrete locomotion commands. ^[5] It reports 78% average score for co-training against 59% for robot-only in the lab, and 82% against 31% in scenes covered only by human data. ^[5] Human demonstrations were also about twice as fast to collect, at 39.7 s per episode on average against 62.1 s for robot teleoperation. ^[5]

In the EgoHumanoid analysis, a human-only model reached 100% on the navigation-dominated first sub-steps of each task, but it fell behind on precision-critical manipulation, and the authors found the best ratio of robot to human data depends on the task. ^[5]

Open tooling for datasets, training and deployment

You do not have to build the whole stack. LeRobot provides a hardware-agnostic Python interface that standardises control across platforms, from low-cost arms to humanoids. ^[1] It lists the Unitree G1 among its supported hardware, and a custom robot can implement the same Robot interface to reuse its recording, training and visualisation tools. ^[1]

Its LeRobotDataset format pairs synchronised MP4 videos or images for vision with Parquet files for state and action data. ^[1] The imitation learning policies it implements include ACT, Diffusion and VQ-BeT, and a training run is started with the lerobot-train command. ^[1]

Evaluating and bounding what you have learned

Report results in the setting you care about. EgoHumanoid evaluated both in-domain laboratory scenes and generalisation scenes, using 20 trials per setting, and the gap between robot-only and co-trained policies was far larger in the unseen scenes. ^[5]

Expect more data per task than for fixed-base manipulation. The same authors found that loco-manipulation needed roughly 2–3 times as many demonstrations as fixed-base tasks of similar difficulty, because the robot rarely stops at precisely the same position and the manipulation phase starts from varied viewpoints. ^[5]

Look at the failures, not only the success rate. In TWIST2, a policy for kicking a T-shaped box reached the target in 6 out of 7 trials, but it could only kick forward. ^[3] In the whole-body cup-pick task the authors observe that the policy can reach the cup in most cases, but because the cup is very light, even a slight drift often results in grasp failure. ^[3] EgoHumanoid reports that its human-only model had three times more manipulation failures than locomotion failures, whereas the robot-only model's failures were balanced across both phases. ^[5]

LeRobot evaluates policies in simulation with lerobot-eval on benchmarks such as LIBERO and MetaWorld, and it runs a trained policy on a real robot with lerobot-rollout. ^[1] NVIDIA's repository describes open-loop evaluation, which compares predicted actions against ground truth from your dataset, and closed-loop evaluation, which tests the model in simulation or on real hardware. ^[2]