A4A: Action-Oriented 4D Affordances

A4A teaser: from human interaction demonstrations to action-oriented 4D affordances to affordance-to-action transfer — **What should robots learn from human demonstrations?** We identify the task-conditioned future motion of interaction-relevant 3D geometry as a transferable signal across embodiments. **A4A** uses these action-oriented 4D affordances to pretrain VLA policies before adapting them to robot control.

Abstract

Human demonstrations contain rich manipulation knowledge, but it remains unclear what information can be transferred effectively to robot control. Existing affordance representations are typically formulated as 2D masks, 3D regions, contact points, or actionability scores, and therefore primarily identify where interaction may occur. However, effective manipulation also requires modeling how interaction-relevant geometry evolves during task execution. To bridge this gap, we introduce action-oriented 4D affordances, which represent the language-conditioned future trajectories of interaction-relevant 3D points. These trajectories capture task-conditioned geometric evolution rather than embodiment-specific actions, enabling transferable interaction priors across humans and robots. Based on this representation, we construct a large-scale action-oriented 4D affordance dataset from existing human–object interaction video data and complementary RGB-D demonstrations, and introduce A4A, an affordance-to-action framework that uses 4D affordance trajectory prediction to pretrain robot policies before manipulation finetuning. Experiments in both simulation and the real world validate the effectiveness of A4A, showing that pretraining with action-oriented 4D affordance data consistently improves the manipulation performance of diverse VLA policies. These results establish action-oriented 4D affordances as an effective cross-embodiment representation for transferring manipulation knowledge from human demonstrations to robot control.

>80K4D affordance interaction clips

5VLA families improved

+22 ppaverage gain on RLBench

56.7 → 72.5%real-robot success, averaged over policies

Contributions

Action-Oriented 4D Affordances

A task-conditioned representation of the future evolution of interaction-relevant 3D geometry captured from human demonstrations. Unlike static affordance localization, it characterizes how an interaction should unfold, providing a transferable, action-aligned supervision signal across embodiments.

Affordance-to-Action Transfer

A cross-embodiment framework that uses 4D point trajectories as the pretraining prediction space of VLA policies, then adapts the learned representation to robot control through each policy's native state and action interfaces — no extra temporal network is introduced.

Consistent Downstream Gains

Evaluated across five VLA policy families, simulated manipulation benchmarks, and real-world tasks: consistent gains for autoregressive, regression, diffusion, and flow-matching policies, strong RGB-only results against methods that explicitly consume 3D or 4D observations, and more effective transfer than generic scene-wide 4D pretraining.

Method

Over a short horizon, end-effector motion and the 4D motion of interaction-relevant local points are two geometric descriptions of the same manipulation transition. A4A turns that correspondence into a pretraining objective.

A4A pipeline: stage I interaction-centric 4D affordance pretraining and stage II embodiment-specific robot finetuning sharing one latent module — **Affordance-to-action representation transfer.** Guided by the short-horizon geometric alignment between interaction-relevant 4D point motion and robot end-effector motion, A4A pretrains a vision–language policy to predict future 3D point trajectories and transfers the learned representation to robot control through the policy's native state and action interfaces.

Stage I — 4D Pretraining

A point projector P_Q maps interaction-relevant query-point coordinates into the policy's state-token space; the shared latent module F_θ consumes them alongside the native vision–language features, and the head H_Q predicts future point motion, parameterized as a residual over constant velocity.

Interface Substitution

The transition to robot control is localized to the external interfaces: P_Q → P_S and H_Q → H_A. The vision–language stack and the shared latent parameters F_θ are preserved across both stages.

Paradigm-Matched Objective

Each base policy keeps its own objective family, applied to valid 4D point targets: token cross-entropy for OpenVLA, continuous regression for OpenVLA-OFT, DDPM denoising for Octo, and flow matching for π₀ and π_0.5 — so pretraining and control share one prediction mechanism.

Action-Oriented 4D Affordance Corpus

Over 80K interaction clips in total, built from HOI4D and EPIC-KITCHENS together with roughly 30K in-house RGB-D demonstrations. Clips from the public datasets are processed using their existing interaction annotations — reconstructed directly in 3D where reliable depth, object poses, or 3D labels are available, and otherwise lifted from image-space tracks with estimated depth and camera-motion compensation. For the in-house subset, query points are localized with GroundingDINO + SAM 2, tracked with CoTracker3, and back-projected to metric 3D using LingBot-refined depth. Both routes yield identity-preserving 3D point trajectories.

Visualization of action-oriented 4D affordances across sixteen human interaction clips — **Visualization of action-oriented 4D affordances.** The colored point flows represent the displacement of interaction-relevant 3D points from the current timestep to the next, with the color gradient indicating the temporal evolution of the motion. The corpus spans translation-dominant motion, rotation-dominant motion, constrained articulated motion, and tool-mediated interactions.

Stage-One 4D Affordance Prediction

Predicted versus ground-truth point-motion targets on four interactions — **Qualitative results of stage-one 4D affordance prediction.** Predicted and ground-truth point-motion targets across four representative interactions: closing an oven, cutting fruit, dragging a drawer, and pouring from one cup into another.

Effectiveness Across VLA Paradigms

Per-task success rate (%) on LIBERO-Object. Each task picks up the named object and places it in the basket. In every pair, A4A pretraining is the only variable.

Object	Octo			OpenVLA			OpenVLA-OFT			π₀			π_0.5
Object	Base	A4A	Δ	Base	A4A	Δ	Base	A4A	Δ	Base	A4A	Δ	Base	A4A	Δ
Alphabet soup	36	64	+28	76	88	+12	100	100	0	76	88	+12	100	100	0
Cream cheese	20	56	+36	68	78	+10	98	98	0	86	92	+6	90	94	+4
Salad dressing	30	68	+38	84	92	+8	98	98	0	86	94	+8	100	98	−2
BBQ sauce	16	36	+20	42	54	+12	98	100	+2	56	78	+22	84	92	+8
Ketchup	12	48	+36	74	86	+12	100	98	−2	88	92	+4	100	100	0
Tomato sauce	38	58	+20	58	74	+16	100	100	0	72	86	+14	96	98	+2
Butter	18	38	+20	62	68	+6	98	100	+2	88	96	+8	92	96	+4
Milk	14	56	+42	78	84	+6	100	100	0	62	74	+12	84	88	+4
Chocolate pudding	54	72	+18	56	62	+6	88	94	+6	90	96	+6	100	96	−4
Orange juice	44	76	+32	66	78	+12	100	98	−2	74	82	+8	94	98	+4
Average	28.2	57.2	+29.0	66.4	76.4	+10.0	98.0	98.6	+0.6	77.8	87.8	+10.0	94.0	96.0	+2.0

The five policies span diffusion (Octo), discrete autoregressive generation (OpenVLA), continuous regression (OpenVLA-OFT), end-to-end flow matching (π₀), and hierarchical flow matching (π_0.5). A4A improves the average success rate of every policy; the smaller gains for OpenVLA-OFT and π_0.5 are consistent with their already high baselines of 98.0% and 94.0%.

Comparison with Geometry-Aware Policies

Success rate (%) on four RLBench tasks. Our RGB-only policy is compared against methods that consume explicit 3D or 4D observations. Comparison numbers are quoted from prior work, except ARM4R, which we finetune ourselves from its released stage-one weights.

Method	Meat	Sweep	Turn Tap	Slide	Avg.
Image-BC (ViT)	0.0	0.0	16.0	0.0	4.0
C2FARM-BC	20.0	0.0	68.0	16.0	26.0
ManiGaussian	60.0	64.0	56.0	24.0	51.0
ARM4R	68.0	48.0	28.0	20.0	41.0
Ours w/o pretrain	56.0	44.0	44.0	12.0	39.0
Ours w/ pretrain	88.0	60.0	68.0	28.0	61.0
Δ	+32.0	+16.0	+24.0	+16.0	+22.0

Meat, Sweep, Turn Tap, and Slide denote meat off grill, sweep to dustpan, turn tap, and slide block to target. Δ is the gain from 4D affordance pretraining. Without any explicit 3D observation at execution time, the pretrained RGB policy is best on meat off grill and slide block to target, and matches the best result on turn tap.

What Matters Is the Pretraining Data

Architecture-controlled comparison: ARM4R with its original pretraining versus the same architecture pretrained on our 4D affordance corpus — **Architecture-controlled comparison.** Both variants use the same ARM4R architecture, robot demonstrations, finetuning recipe, and evaluation protocol. Gray uses the released initialization learned from generic scene-wide 4D point tracks; green uses the same architecture pretrained on our action-oriented 4D affordance dataset — raising the four-task average from 41.0% to 60.0%, with *turn tap* going from 28% to 60%.

Real-World Manipulation

A4A policies deployed on a physical AgileX Piper arm. 100 teleoperated demonstrations per skill; 10 rollouts per skill at evaluation.

Open the microwave

Place the cup on the mug

Cut the peach

Cook rice is trained as three independent skills, each evaluated separately.

Open the pot

Pour rice

Pour water

Success on the physical Piper arm — successful trials out of 10.

Method	Microwave	Cup	Peach	Cook Rice			Avg.
Method	Microwave	Cup	Peach	Pot	Rice	Water	Avg.
Octo	7/10	3/10	3/10	2/10	4/10	3/10	36.7%
+ A4A	8/10	5/10	6/10	5/10	6/10	6/10	60.0%
OpenVLA	6/10	6/10	4/10	3/10	6/10	4/10	48.3%
+ A4A	9/10	6/10	7/10	5/10	7/10	7/10	68.3%
OpenVLA-OFT	9/10	7/10	7/10	5/10	8/10	7/10	71.7%
+ A4A	9/10	8/10	9/10	6/10	8/10	8/10	80.0%
π_0.5	9/10	8/10	7/10	5/10	7/10	6/10	70.0%
+ A4A	9/10	9/10	9/10	7/10	8/10	7/10	81.7%

Incorporating action-oriented 4D affordances improves every policy, raising the mean success rate across all four from 56.7% to 72.5%. Clear improvements are observed on pouring rice and pouring water, where execution requires coordinated object translation together with substantial orientation change — exactly the rotational structure the 4D representation carries over from human demonstrations.

**Real-world manipulation.** Representative A4A rollouts on placing a cup on a mug, opening a microwave, cutting a peach, and cooking rice.

BibTeX

@article{han2026a4a,
  title   = {A4A: Cross-Embodiment Transfer of Action-Oriented 4D
             Affordances from Human Demonstrations},
  author  = {Han, Yifan and Liu, Litao and Gu, Yuqi and Lu, Ye and
             Wang, Hanqing and Wai, Sidney and Myrie, Ishaan and
             Zhang, Qi and Yu, Jingjin and Li, Gen},
  year    = {2026}
}