Unitree releases UnifoLM-ER-Flow, a vision-language model that predicts scene changes alongside robot actions

SUMMARY

Unitree Robotics has published UnifoLM-ER-Flow on Hugging Face, a successor to its UnifoLM-ER-1 model. According to the model card, it adds two capabilities to its predecessor: anticipating how a scene is about to change during an interaction, and expressing robot actions as discrete tokens, both handled inside one vision-language model. The weights are released under the Apache 2.0 licence, and the card gives a model size of 4B parameters.

To anticipate what an action will change, Unitree isolates the moving parts of a scene with optical flow and compresses them with a VQ-VAE into short token sequences of a set length. Given the current camera image and either a task description or an action, the model then predicts these tokens for the regions that are about to move, which keeps its attention on the objects being handled and on the effects of handling them.

Actions are split into three streams: the pose of the end effector, the joints of the end effector itself, such as a gripper or a dexterous hand, and the joints of the lower body. Each stream is discretised by its own residual vector quantisation model, and the resulting token sequences are aligned in time and fed to the model together. The card presents this as the link between understanding a scene, predicting its changes and coordinating the whole body.

The model card shows demonstrations of cleaning a table, folding a towel, packaging a phone and placing plates. UnifoLM-ER-Flow is part of Unitree's UnifoLM-WLA-1.0 collection on Hugging Face.

Drafted with AI assistance from the source and reviewed by WBH. Follow the source link for the full text.