Research project · World Action Models

SelfWAM

A self-grounded, unified World Action Model for fast robot control.

Bikang Pan1,2,* Fan Liu1,2,* Haotao Lu1,2 Jingya Wang1 Ye Shi1,2
1ShanghaiTech University 2InstAdapt *Equal contribution

World Action Models can strengthen robot policies by learning actions and future observations together. But when future prediction sees only the task and current observation, it may learn a generic continuation instead of the consequences of the action actually taken. SelfWAM adds a clean-action path that conditions future RGB and robot self-mask prediction on the demonstrated action chunk. The result is unified consequence modeling with the same lightweight action expert preserved for fast, action-only control.

Explore results ↓ Paper ↗ Code ↗ BibTeX ↘
92.6%
RoboTwin 2.0 avg. successclean 92.2 · random 93.1
32.5%↓
lower future-video LPIPSvs. action-agnostic Fast-WAM
+3.55dB
PSNR gainon 275 future-video samples
<1%
added action-only latency0% peak-memory increase

01 · Headline result

Action-Sensitive Futures

Change the action. Change the future.

We scale the executed action by λ and ask each world model to roll the future forward from the head camera. SelfWAM follows the same hold-to-expert progression as the ground truth: at λ = 0 the arm stays still; at λ = 1.0 it completes the expert motion. The action-agnostic Fast-WAM baseline cannot reproduce this response—the future changes little as the commanded action changes.

Columns scale the executed action chunk by λ. Rows compare future RGB (head cam). Ground truth Fast-WAM (uncond.) SelfWAM (ours)

Task 01 · Beat block with hammer

λ = 0.00HOLD
λ = 0.25SLOW
λ = 0.75MEDIUM
λ = 1.00EXPERT
GROUND
TRUTHfuture RGB
FAST-WAMuncond.
SELFWAMpredicted

Task 02 · Adjust bottle

λ = 0.00HOLD
λ = 0.25SLOW
λ = 0.75MEDIUM
λ = 1.00EXPERT
GROUND
TRUTHfuture RGB
FAST-WAMuncond.
SELFWAMpredicted

02 · Method

How it works

A modality-specialized Mixture-of-Transformers with deliberately separated information flow.

SelfWAM overview

Overview. Compared with action-unconditioned video prediction, SelfWAM shifts the model from a passive observer to an actor that predicts the visual consequences of its own actions. Its predicted motion follows the same slow-to-fast progression as the ground truth when the executed action is scaled (top). Given robot videos, trajectories and instructions, it couples a lightweight action expert with a shared video backbone; the executed action conditions the backbone to jointly predict future RGB and robot self-mask video. At deployment the video branch is removed and the action expert runs alone for fast action-only inference.

→ 01

Unified routing

A clean copy of the executed action is routed only to future-video queries — aligning predicted dynamics with the commanded control while staying hidden from the noisy future-action path, so the policy cannot recover its target.

→ 02

Self-grounded masks

The same backbone jointly generates RGB and robot self-mask video, redirecting learning from background appearance toward the robot as the agent of change throughout the motion.

→ 03

Action-only deployment

Once the current context is encoded, action denoising runs entirely through the lightweight action expert — no video or mask experts in the loop — preserving fast closed-loop control.

SelfWAM architecture and attention design

Architecture & attention design. Left: the MoT contains a video backbone and a lightweight action expert coupled through mixed attention. Middle: during training, clean executed-action tokens condition only future-video queries. Right: two inference modes — fast control uses the action expert alone; optional rollout conditions the video backbone on a supplied action to generate its visual consequences.

03 · Policy performance

RoboTwin 2.0 success rate

Bimanual manipulation across all tasks, clean and randomized scenes.

Baselines Fast-WAM SelfWAM w/o masks SelfWAM (ours)
light bar = Clean  ·  dark bar = Random
100
75
50
25
65.9
58.4
82.7
76.8
88.7
87.0
86.4
85.0
82.8
84.8
90.8
90.8
91.8
91.9
92.2
93.1
π0
avg 62.2
π0.5
avg 79.8
Motus
avg 87.8
GigaWorld
avg 85.7
Policy only
avg 83.8
SelfWAM
w/o masks
avg 90.8
Fast-WAM
avg 91.8
SelfWAM
(ours)
avg 92.6

Qualitative policy rollout

Open microwave

Same clean-scene episode and initial state · 2× playback

Fail
Fast-WAMThe arm leaves the target while the door remains closed.
Success
SelfWAMThe policy reaches the handle and opens the microwave door.
SelfWAM reaches 92.6% average success on RoboTwin 2.0. It preserves the strong policy performance of Fast-WAM (91.8%) while adding unified world modeling, and improves randomized-scene success from 91.9% to 93.1%.
MethodClean ↑Random ↑Average ↑
π065.9258.4062.16
π0.582.7476.7679.75
Motus88.6687.0287.84
GigaWorld-Policy86.3685.0485.70
Policy only †82.7684.8083.78
SelfWAM w/o self masks90.8090.8090.80
Fast-WAM91.8291.8691.84
SelfWAM (ours)92.1693.0892.62

† Reported Fast-WAM variant without video co-training; success rates in %.

04 · World-model video quality

Future-video quality

275 future-video clips with an identical backbone, isolating the effect of conditioning on the demonstrated action.

LPIPS ↓

0.0636
0.0429
Fast-WAMSelfWAM

PSNR ↑

29.24
32.79
Fast-WAMSelfWAM

FVD-I3D ↓

45.92
34.55
Fast-WAMSelfWAM
LPIPS
Learned Perceptual Image Patch Similarity — perceptual distance between predicted and ground-truth frames; lower means more faithful to the real future.
PSNR
Peak Signal-to-Noise Ratio — pixel-level reconstruction quality; higher means sharper, less noisy frames.
FVD-I3D
Fréchet Video Distance under an I3D feature extractor — distribution-level realism of the clip over time; lower means more realistic temporal dynamics.
Conditioning on the executed action reduces mean LPIPS by 32.5%, improves PSNR by 3.55 dB, and lowers FVD-I3D by 24.8% relative to unconditioned Fast-WAM.

Task · Beat block with hammer — expert action λ = 1.0, future RGB rollout

Ground truth Fast-WAM (uncond.) SelfWAM (action-cond.)

05 · WorldArena consistency

Action-following & consistency

Standard WorldArena protocol on generated future videos.

Both unified variants achieve non-zero Action Following, while Fast-WAM remains at 0.0000. Full SelfWAM scores highest at 0.0184, showing that its predicted futures respond to the commanded action instead of replaying a generic continuation.
ModelAction Following ↑Subject Consistency ↑Background Consistency ↑Photometric Consistency ↑
Fast-WAM0.00000.91380.93090.8126
SelfWAM w/o mask0.01810.93210.93801.5168
SelfWAM (ours)0.01840.93260.93851.4361

SelfWAM improves on Fast-WAM across action-following, subject and background consistency.

Action Following
Whether the generated future actually changes with the commanded action — the core action-sensitivity signal; higher means the model responds to the action instead of replaying a generic continuation.
Subject Consistency
The main subject (robot / manipulated object) preserves its appearance across generated frames.
Background Consistency
The static scene stays stable, with motion concentrated on the acting agent.
Photometric Consistency
Lighting, color and exposure remain coherent from frame to frame.

06 · Inference efficiency

Fast action-only inference

NVIDIA H200, 32-action window, 10 denoising steps, averaged over four runs after warm-up.

Action only (ms)
320.7
323.9
Fast-WAMSelfWAM
Only +0.97% latency
Action + video (ms)
668.4
677.5
Fast-WAMSelfWAM
Only +1.36% latency
Peak memory (GiB)
13.959
13.959
Fast-WAMSelfWAM
No additional memory
Clean-action conditioning adds less than 1% action-only latency and 1.36% joint latency, with no measured increase in peak memory. The heavier video branch stays outside the iterative action-denoising path.

07 · Analysis

Counterfactual rollouts

Same observation, different candidate actions — SelfWAM imagines action-dependent futures for diagnosis.

Action perturbation rollouts

Action perturbation. Candidate actions that differ in direction, gripper command or temporal scale produce visibly different futures; object change begins near a plausible contact region while unrelated background stays stable.

Action interpolation rollouts

Action interpolation. SelfWAM predicts action-dependent future RGB and robot self-masks, revealing whether object motion is spatially and temporally compatible with the robot's own motion.

08 · Physical evaluation

Real-Robot Experiments

Four complete manipulation trials on a physical dual-arm platform.

Task 01Pick up the cup
Task 02Pick up the mouse
Task 03Pick up the pen
Task 04Pick up the paper ball
01/01

09 · Citation

Cite this work

If SelfWAM is useful in your research, please consider citing our arXiv preprint.

@misc{pan2026selfwam,
  title   = {SelfWAM: A Self-Grounded Unified World
             Action Model for Fast Robot Control},
  author  = {Pan, Bikang and Liu, Fan and Lu, Haotao and
             Wang, Jingya and Shi, Ye},
  year    = {2026},
  eprint  = {2608.00725},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url     = {https://arxiv.org/abs/2608.00725}
}