DIPOLE
Fusing Vision and Geometry for Robust Visuomotor Generalization

Yikai Tang1*, Haoran Geng1*†, Jindou Jia2, Yuxuan Hu2, Sheng Zang2, Jianfei Yang2†, Pieter Abbeel1†, Jitendra Malik1†
1 University of California, Berkeley
 2 Nanyang Technological University
*Indicates Equal Contribution, †Indicates Equal Advising
arxiv Paper Code Data Contact

Abstract

Imitation learning has emerged as a crucial approach for acquiring visuomotor skills from demonstrations, where designing effective observation encoders is essential for policy generalization. However, existing methods tend to struggle once test-time conditions differ from the demonstrations, such as changes in lighting, texture, viewpoint, object placement, or object identity. To address this challenge, we propose DIffusion POlicy with compLementarity Encoders (DIPOLE), a visuomotor policy that learns to fuse complementary modalities through a training-time mechanism rather than a specialized fusion architecture. A modality-wise dropout masks one branch at each training step, encouraging each modality to remain individually informative. A lightweight cross-attention layer then exchanges complementary cues between the two. This design endows DIPOLE with five core strengths: stable high performance across diverse tasks, robustness to visual changes, spatial generalization at sub-centimeter precision, emergent capability beyond either modality, and zero-shot transfer to unseen objects. Across 18 simulated and 4 real-world tasks, DIPOLE outperforms six baselines by 39.1% on average, with gains of 41.5% under unseen visual distractors and 15.2% under randomized object placement.

Key Attributes

We evaluate DIPOLE on a unified benchmark of 22 manipulation tasks — 18 simulated tasks drawn from LIBERO, RLBench, and ManiSkill, plus 4 real-world tasks on a Franka arm — against six baselines spanning RGB, RGB-D, and point-cloud encoders. Across the full suite DIPOLE outperforms all baselines by 39.1% on average, and the gains follow a single theme: a policy trained to keep each modality individually informative converts complementarity into stability, robustness, precision, transfer — and capabilities neither modality can deliver alone.

Stable high performance across diverse tasks

One recipe, no per-task tuning. On 18 simulated tasks — each run under three levels of domain randomization — DIPOLE attains a 65% mean success rate, two to six times the 11%–33% that the six baselines span, and it leads on every real-world task as well. Just as telling is how it wins: competing encoders shine on one task family and collapse on another (point-cloud encoders ace CloseBox yet drop to 0% on Butter), whereas DIPOLE stays near the top everywhere, with the lowest cross-task variation of any method (CoV 30%, 1.5–4.4× smaller than the baselines). High mean with low variance is the signature of a single generalist policy rather than one tuned to a favorable task family.

Success rate per task (%, ↑)

Cross-task variation (CoV %, ↓)

Success rates averaged over three domain-randomization levels; lower CoV = more stable across tasks.
Data table
Task — success rate (%)DIPOLERGB-ResNet18RGBD-ResNet18RGBD-ViTRGBD-MultiMAEDP3PonderV2
CloseBox96355348495875
StackCube31060100
AlphabetSoup792224614800
Butter887262686900
OrangeJuice531718313378
Tomato72811131400
Average65272932331114
Cross-task CoV (%)3052.848.4845.4943.86126.19133.33

Robustness to visual changes

What happens when vision becomes unreliable? Material, table-position, and viewpoint shifts attack exactly the cues an RGB policy depends on — so we vary all three, progressively, across randomization levels L0–L2. DIPOLE's success rate moves by only 2.53% across levels, roughly 11× steadier than the 28.5% baseline average, and at the most aggressive L2 it still holds 64% while every baseline falls to 8–21%, leaving an average margin of 41.5%. The invariance carries beyond simulation: DIPOLE passes the real-world color-cycling Light-Strip test zero-shot, where both DP(RGB) and DP(RGBD) fail. When appearance drifts, the geometric stream quietly absorbs the perturbations that most degrade unimodal policies.

Success rate by randomization level (%, ↑)

Cross-randomization dispersion (%, ↓)

Average success rate over all 18 simulated tasks at each randomization level; lower dispersion = steadier under visual shifts.
Data table
Randomization level — success rate (%)DIPOLERGB-ResNet18RGBD-ResNet18RGBD-ViTRGBD-MultiMAEDP3PonderV2
L0 · original68464344481219
L1 · +material & table position64172336321315
L2 · +camera viewpoint641921202088
Cross-randomization dispersion (%)2.5332.7235.0145.2837.235.0615.43

Spatial generalization at sub-centimeter precision

Geometry, in turn, is not always enough on its own: sub-centimeter tasks demand a precision that a downsampled point cloud is too coarse to resolve. By inheriting RGB's fine-grained visual detail on top of the point cloud's placement-invariant structure, DIPOLE handles both at once. Trained on only 30 grid positions and evaluated across the full 143-position workspace, it fetches a bottle with a 2 cm cap at 22.4% success against 1.4%–18% for the baselines — a +15.2% margin under randomized placement. On contact-rich plug insertion, which demands sub-centimeter vertical accuracy, it leads at 36% while DP3 fails entirely at the precision-critical grasp, a +18.77% average gain on precision tasks. The success grids below make the difference visible at a glance.

Real-world spatial randomization results: per-position success grids for Fetch Bottle (143 positions) and Insert Plug (25 positions), comparing DIPOLE with three baselines
Real-world spatial randomization results. Fetch Bottle: policies trained on the 30 dotted grid positions and evaluated on all 143; Insert Plug: evaluated on 25 evenly distributed positions. Dark green and light green mark IID and OOD success, white marks failure.
Data table
Task — success rate (%)DIPOLEDP (RGB)DP (RGBD)DP3
Fetch Bottle (randomized placement)22.38182.101.40
Insert Plug (sub-cm insertion)3625160

Zero-shot transfer to unseen objects

Can one policy survive objects it has never seen? We replace the Pour-Cereal plate with a transparent drawer, a bowl, and a yogurt cup; swap the Insert-Plug iPad for an AirPods charging case (or move it to a novel position); and re-light Fetch-Bottle with strong color-cycling illumination. DIPOLE passes all six transfers, while no unimodal baseline exceeds three of six — each one fails precisely where its own modality breaks. The pattern is telling: DIPOLE generalizes by leaning on whichever modality the novel scene still preserves in distribution, rather than requiring every stream to remain intact. Robust generalization need not come from scaling demonstrations alone — it can emerge from observation representations that encode the task-relevant structure of the scene more faithfully.

Zero-shot transfer settings: Pour-Cereal target replaced with a drawer, bowl, and cup; Insert-Plug iPad moved or replaced with an AirPods case; Fetch-Bottle under color-cycling illumination
Zero-shot transfer settings. Top: in Pour-Cereal, the target plate is replaced with containers of varying sizes and shapes. Bottom-left: in Insert-Plug, the iPad is moved to a novel position or replaced with an AirPods charging case. Bottom-right: in Fetch-Bottle, the original lighting is swapped for strong color-cycling illumination.
Data table
ScenarioDIPOLEDP (RGB)DP (RGBD)DP3
Drawer✓✓✓✓
Bowl✓✓✗✓
Yogurt✓✗✗✗
Position✓✓✗✗
AirPods✓✗✓✗
Light Strip✓✗✗✓
Scenarios passed6/63/62/63/6

Emergent capability beyond either modality

The most striking result comes last: fusion that exceeds the sum of its parts. On four task settings where both unimodal encoders fail almost completely when trained alone — ≤1% success for the RGB-only ResNet and the point-cloud-only DP3 — DIPOLE recovers to 72.3% mean success. Ablations isolate modality-wise dropout as the mechanism: remove it, and the fused model reverts to its strongest unimodal baseline no matter which fusion architecture is used. In other words, the emergent gain comes from how DIPOLE is trained, not from the fusion module itself — even a modality that is weak on a task still supplies complementary cues, so fusion produces capabilities neither stream can deliver alone.

Setting — success rate (%)ResNet (RGB only)DP3 (PC only)DIPOLE
AlphabetSoup L10067
AlphabetSoup L20073
Tomato L11068
Tomato L21081
Mean0.5072.3

Settings where each unimodal encoder fails (≤1%) when trained alone; L1/L2 are domain-randomization levels.

Methodology

DIPOLE (DIffusion POlicy with compLementarity Encoders) predicts actions from RGB and geometric observations through a pair of complementary encoders — a ResNet-18 for RGB images and a DP3-style encoder for point clouds, alongside a small MLP for proprioception. Rather than relying on a specialized fusion architecture, DIPOLE enforces complementarity through a training-time mechanism: at each training step, modality-wise dropout masks either the RGB or the point-cloud feature, forcing each modality to remain individually informative instead of collapsing onto the locally easier one. A lightweight cross-attention layer then exchanges complementary cues between the two streams, and the fused representation conditions a diffusion action head.

DIPOLE overview: point-cloud and RGB streams fused through modality-wise dropout and a complementarity fusion module, evaluated on challenging, perturbed, and zero-shot manipulation tasks, with a radar chart showing DIPOLE leading six baselines on all five axes

Experiments and Results

Insert Plug

In the insert plug task, the robot should insert a plug into a socket. The position of the plug is randomized evenly in 100 positions in a 20cm*20cm square.

DIPOLE

Diffusion Policy(RGB)

Diffusion Policy(RGBD)

DP3

During evaluation, we also test the policy with the tablet placed in different positions and replaced with a pair of Airpods.

Tablet in Different Position

Replaced with Airpods

Pour Cereal

In the pour cereal task, the robot should pour cereal from a bowl into a plate. We train the policy in a fixed environment, while testing its strengths in both fitting into the distribution and 0-shot transferring to unseen object and lighting conditions.

Training: a consistent scene with fixed lighting and object

Transfer: various distributions of flowing rgb light strip and objects of different shapes and sizes

Pick Butter

THe Pick Butter task imitates the same task in Libero-objects, where we aim to pick the butter from a cluttered table and place it into the basket.

DIPOLE

Aside of the findings in DIPOLE's effectiveness, we also discovered the surprisingly aligning results with the same task in simulation. We find that strongly domain-randomized simulation benchmarks can well reflect the performance in real-world settings.
Simulation vs Real-world Comparison
Performance comparison between simulation and real-world settings
Demo Rollout in Simulation
Mimic Task in Real-world

Pick Bottle

The Pick Bottle task evaluates spatial generalization in a precision-critical setting. The robot must reach the bottle cap with sub-centimeter accuracy (≤1 cm) in order to successfully grasp it. During training, the bottle is placed at 30 distinct locations, and performance is assessed across all 143 grid positions in the workspace.

Generalize in training distribution

Transfer to extreme lighting shift

Simulation Tasks

We conduct our simulation benchmarks in RoboVerse, evaluating tasks from LIBERO, ManiSkill, and RLBench. All tasks are domain-randomized at three levels using a unified simulation codebase.

Simulation Overview

BibTeX

@misc{tang2025dipole,
        title={DIPOLE: Fusing Vision and Geometry for Robust Visuomotor Generalization},
        author={Yikai Tang and Haoran Geng and Jindou Jia and Yuxuan Hu and Sheng Zang and Jianfei Yang and Pieter Abbeel and Jitendra Malik},
        year={2025},
        eprint={2511.22445},
        archivePrefix={arXiv},
        primaryClass={cs.RO},
        url={https://arxiv.org/abs/2511.22445},
  }