Dissecting Advantage-Guided Post-Training for Vision-Language-Action Policies

Cao, Jiahang; Zhao, Hanye; Lai, Hang; Zhang, Shenyu; Han, Xiaoshen; Li, Xinghang; Liu, Futeng; Peng, Wanli; Wang, Heyun; Wang, Yunhong; Li, Jason; Yu, Yong; Zhang, Weinan

Dissecting Advantage-Guided Post-Training for
Vision-Language-Action Policies

Jiahang Cao^1†*, Hanye Zhao^1†*, Hang Lai^2✉, Shenyu Zhang², Xiaoshen Han^1†, Xinghang Li²,
Futeng Liu², Wanli Peng², Heyun Wang², Yunhong Wang², Jason Li², Yong Yu¹, Weinan Zhang^1✉

¹School of Computer Science, Shanghai Jiao Tong University ²Xiaomi Robotics

^*Equal contribution · ^✉Corresponding author · ^†Work done during internship at Xiaomi Robotics

Paper arXiv

One-take videos for four real-robot tasks.

Abstract

Advantage-guided reinforcement learning provides a practical way to post-train vision-language-action (VLA) policies using limited robot data. However, its performance depends on several coupled choices, including how critic-derived advantages are constructed, calibrated, and used for policy training. Existing recipes often combine these choices into a single end-to-end procedure, making their individual effects difficult to identify.

In this work, we dissect advantage-guided VLA post-training through a controlled empirical study that separates these design choices while accounting for their distinct estimands. We develop stage-specific offline evaluation methods to screen alternative choices efficiently, without requiring extensive real-robot policy evaluations for every possible combination. The staged evaluation identifies a modular recipe that combines temporal-difference advantage construction, group-wise calibration, and continuous advantage weighting. Across four real-world bimanual tasks, the resulting recipe improves mean task progress and success over the SFT initialization by 0.42 and 0.63, respectively. Moreover, the proposed evaluation diagnostics show an overall alignment with downstream real-world performance, supporting their use for interpreting empirical outcomes and selecting advantage-guided post-training designs in practice.

Three Stages, Three Questions

We decompose advantage-guided post-training into construction, calibration, and utilization.

Three-stage study design: Stage I constructs advantages, Stage II calibrates advantage scale, Stage III extracts the policy.

Stage I · RQ1

Advantage construction

Four critics: IQL, SARSA, ALOE, MC — each with Q−V or n-step TD
Screened by two ordering diagnostics: R_int, R_prog

Selected: IQL + n-step TD

Stage II · RQ2

Advantage calibration

Four grouping rules: Raw, Global, Temporal, Value
Screened by η²: cross-sub-task variance share, lower is better

Selected: Value-based grouping

Stage III · RQ3

Advantage utilization

Four conditions: SFT Init, DAgger, Weight, Filter
Matched actor, data, and budget — only the sample weight differs

Selected: continuous Weight

IQL critic + n-step TD advantage Value-based group calibration Continuous advantage weighting

Offline Diagnostics

Stage I (RQ1): advantage construction

Stage I asks which advantages recover the expected quality orderings among dataset actions. Because the compared constructions estimate different quantities, we evaluate the orderings they induce instead of their raw values.

Representative intervention and progress pairs across the four real-world tasks. — **What the diagnostics compare.** Intervention pairs (left) contrast failing policy behavior with the recovery after a human takeover; progress pairs (right) contrast stalled and advancing behavior within the same sub-task. Both ask whether the better behavior receives the higher estimated advantage.

Intervention and progress ordering diagnostics for seven advantage-construction configurations on four tasks. — **Stage I (RQ1).** For IQL, SARSA, and ALOE critics, n-step TD beats *Q−V* on every task and both diagnostics, with R_int from 0.639 to 0.773 and R_prog from 0.805 to 0.953. *Q−V* and Monte Carlo stay close to chance.

Stage II (RQ2): calibration comparability

Stage II asks whether calibrated advantages stay comparable across sub-tasks.

η² is the share of total advantage variance explained by sub-task identity; lower is better. Value-based grouping gives the lowest mean, so it is carried into Stage III.
Calibration	PhoneE	PhoneH	Box	Printer	Mean
Raw	0.072	0.063	0.106	0.127	0.092
Global	0.057	0.051	0.090	0.120	0.080
Temporal	0.071	0.023	0.053	0.028	0.044
Value	0.053	0.027	0.013	0.010	0.026

Real-World Tasks

Four bimanual manipulation tasks on a platform with two 6-DoF arms and three RGB fisheye cameras. The policy predicts chunks of 30 end-effector actions executed at 30 Hz.

PhoneE

Pack Phone Easy

Places a phone into its packaging box and closes the lid.

PhoneH

Pack Phone Hard

A more complex box plus an accessory-insertion step, giving a longer horizon and higher difficulty.

Box

Pack Box

Places three objects into a carton, folds the flaps, and presses the side tabs to close it.

Printer

Refill Printer Paper

Picks up a small stack of paper and loads it into the printer tray.

Real-World Results

Stage III (RQ3): advantage utilization

All conditions start from the same SFT checkpoint and share data, budget, and evaluation schedule; only the per-sample weight differs. Each task column averages 20 real-robot trials per condition.

Task progress is the fraction of completed sub-tasks at termination; task success requires all sub-tasks to be completed. Higher is better.
	Task progress					Task success
Condition	PhoneE	PhoneH	Box	Printer	Mean	PhoneE	PhoneH	Box	Printer	Mean
SFT Init	0.50	0.32	0.69	0.24	0.44	0.30	0.00	0.15	0.00	0.11
DAgger	0.65	0.62	0.91	0.66	0.71	0.35	0.30	0.70	0.45	0.45
Weight (ours)	0.88	0.81	0.95	0.79	0.86	0.80	0.70	0.80	0.65	0.74
Filter	0.78	0.69	0.91	0.60	0.75	0.70	0.40	0.55	0.35	0.50

Rollouts of the selected recipe

Policy rollouts on each of the four tasks, executed by the Weight condition from the table above.

Pack Phone Easy — places a phone into its packaging box and closes the lid.

Ablation study: do the offline diagnostics predict real-robot outcomes?

Keeping the Weight rule fixed, we change one earlier stage at a time and re-evaluate on PhoneH and Printer: the first four conditions vary the Stage-I construction, the last three vary the Stage-II calibration, and the shared fourth condition is the reference recipe above. Higher Stage-I concordance and lower Stage-II η² come with higher real-world progress and success, which is what licenses screening these choices offline.

Ablations of advantage construction and calibration under fixed continuous weighting on PhoneH and Printer, with offline diagnostics overlaid. — Bars show real-robot policy outcomes (left axis); lines show the offline diagnostics for the same configuration (right axis). Stage-I and Stage-II labels below each panel identify what changed.

Full Video

Complete walkthrough of the study: the three staged comparisons, results, and uncut real-robot rollouts.

BibTeX

@article{cao2026dissecting,
  title   = {Dissecting Advantage-Guided Post-Training for Vision-Language-Action Policies},
  author  = {Cao, Jiahang and Zhao, Hanye and Lai, Hang and Zhang, Shenyu and Han, Xiaoshen
             and Li, Xinghang and Liu, Futeng and Peng, Wanli and Wang, Heyun
             and Wang, Yunhong and Li, Jason and Yu, Yong and Zhang, Weinan},
  year    = {2026}
}

Dissecting Advantage-Guided Post-Training forVision-Language-Action Policies