MV-UMI turns human-aligned data to cross-embodiment data for robot policy learning.
Cross-Embodiment Deployment
What if we naively trained a policy on the human-aligned unsegmented data? ☹️
In our ablation study, we found that the policy trained on human-aligned data failed to complete any tasks, despite often moving in sensible directions. This failure is likely due to a distribution shift in the observation space between training and deployment.
Interestingly, the policy trained on segmented data performed well even when the robot was unsegmented at inference. We hypothesize that removing the human helped the model avoid overfitting to strong correlations between human and gripper actions, and focusing on taking the learning signal from the task-relevant parts of the scene.
This is also evident from the attention maps coming from our vision encoders:
a) Unsegmented model focusing on the embodiment
b) MV-UMI model focusing on the objects manipulated
c) MV-UMI model focusing on the objects manipulated (even when robot is unsegmented!)
Why not stick to the egocentric camera alone?
Unable to find the cup
Unable to find the empty bottles rack
MV-UMI Pipeline
Results
Bottles-Rack-Placer
Cans-Shelf-Placer
Markers-Placements
Mechanical Design
The design of MV-UMI is intended to be modular. Mounts are designed for the gripper, allowing it to be used handheld or when attached to a robot.
