Vision systems

What a humanoid's vision system has to do

The Humanoid-OmniOcc paper describes occupancy as estimating, for each voxel around the robot, whether it is free or occupied. The authors say this abstraction supports collision-aware planning and fits real-time control loops. ^[1]

The same authors argue that humanoids work indoors, with egocentric viewpoints, frequent occlusions and dynamic human activity. In their view this is why occupancy datasets built for autonomous driving, with forward-facing cameras and long-range geometry, fit humanoids poorly. ^[1]

Figure says the visuomotor policy of Figure 03 takes head cameras, palm cameras, fingertip tactile sensors and full-body proprioception as inputs. ^[6]

• Booster lists a stereo depth camera and a 9-axis IMU for the Booster K1. ^[8]
• EngineAI lists two ultra-wide field-of-view binocular cameras, on the head and the waist, for the EngineAI PM01. ^[9]
• Boston Dynamics lists tactile sensing and a 360° camera view for Atlas. ^[10]

Stereo, monocular and LiDAR sensing

The Humanoid-OmniOcc authors compare three ways of getting 3D structure. They say monocular systems are prone to depth ambiguity and weak cross-domain generalisation, and that LiDAR is accurate but costly and cumbersome for head-mounted humanoid platforms. For near-field perception they argue that stereo is a middle ground, with metrically grounded dense depth at arm's reach, lower cost and easier deployment. ^[1]

The paper's data collection setup mounts four stereo camera modules on the head of a Unitree G1, facing front, rear, left and right. Each stereo pair estimates depth through disparity computation, which ensures 360° spatial coverage. The authors report a baseline of 6 cm for the stereo cameras they use. ^[1]

The Humanoid-OmniOcc paper introduces a stereo-based occupancy model called HS2Occ and compares it with monocular occupancy models. On real-world captures HS2Occ reaches 35.45 IoU and 19.26 mIoU, while the monocular SurroundOcc reaches 20.35 IoU and the monocular GaussianFormer 17.11. ^[1] An ablation in the same paper swaps in different stereo backbones and concludes that occupancy quality is highly sensitive to the reliability of stereo depth. ^[1]

Field of view, occlusion and close-range cameras

In the HERO system, the authors note that when arm and waist movements are large, the egocentric view is too narrow to keep the object in sight, so they rely on odometry to adjust the reaching goal. ^[2] They also list the restricted egocentric field of view for far or high targets as a remaining limitation, which they say motivates active neck control and visual servoing. ^[2]

One hardware answer is to add cameras closer to the hand. Figure says each hand of Figure 03 integrates an embedded palm camera that gives redundant, close-range visual feedback during grasps, so that Helix keeps visual awareness when the main cameras are occluded, for example when reaching into a cabinet. ^[7]

Figure's introduction of Figure 03 also covers touch: the company says each fingertip sensor can detect forces as small as three grams of pressure. ^[7]

The RoboTacDex dataset, recorded on a Unitree G1, records multi-view RGB and depth together with tactile feedback, and its authors describe a multi-camera synchronisation system that gives millisecond synchronisation across modalities. ^[5]

From pixels to action: modular and end-to-end designs

HERO is modular. It uses open-vocabulary vision models to segment the queried object, proposes grasps with the AnyGrasp model, retargets the grasp to a Dex-3 hand, and then hands the reach to a learned whole-body end-effector tracker. ^[2] The authors say that separating action planning from action execution lets the system inherit visual generalisation from pre-trained models and control quality from simulated training. ^[2] In real-world tests they report a 90% success rate on 10 daily objects. ^[2]

VIRAL trains an RGB-based policy entirely in simulation and deploys it zero-shot to a Unitree G1. The policy performs continuous loco-manipulation for up to 54 cycles. ^[3] It follows a teacher-student design: a privileged teacher that sees the full state is distilled into a vision-based student. ^[3]

The DoorMan work follows a similar pattern for opening doors from RGB alone. The student sees an image and proprioception, while the teacher uses privileged simulator information such as the transform between the robot root and the door. ^[4] On hardware the authors report an 83% success rate, against 80% for an expert teleoperator and 60% for a non-expert. ^[4]

Closing the visual sim-to-real gap

VIRAL combines large-scale visual domain randomisation over lighting, materials, camera parameters, image quality and sensor delays with real-to-sim alignment of the dexterous hands and cameras. ^[3] Humanoid-OmniOcc likewise copies the intrinsics, extrinsics, field of view and baseline of the Unitree G1 cameras into the simulator. ^[1]

The DoorMan ablations test the visual randomisation. With no visual randomisation, success on unseen doors fell to between 5% and 20%. ^[4] Using only 10% of the available textures was comparable to using all of them, with a drop of only 4% to 8%. ^[4]

HERO reports that the default analytical forward kinematics of the Unitree G1 are not accurate enough for hand placement. A learned residual model reduced the end-effector error from 1.30 cm to 0.50 cm. ^[2]

Questions to ask when evaluating a vision stack

The sources above suggest several concrete checks for anyone building or comparing humanoid perception.

• Which sensors does the policy actually consume? Figure names head cameras, palm cameras, fingertip tactile sensors and proprioception as inputs for Figure 03. ^[6]
• What happens when the target leaves the view? HERO found that large arm and waist motions can take the object out of the egocentric view. ^[2]
• Was the result measured on hardware? Humanoid-OmniOcc evaluates models trained exclusively on simulated data on real-world captures from the same Unitree G1 platform. ^[1]
• Was the comparison fair? DoorMan compares the policy with human teleoperators who use the same whole-body controller. ^[4]