RL Accuracy Verification (GRPO + OPD)

Verify a code change keeps GRPO and OPD training accuracy aligned with baselines. Both runs need a single 8-GPU node and take ~40 min (GRPO) / up to a few hours (OPD, 50 steps).

Before launching anything, present an experiment card (目标 / 判读标准 / 最小规模 / 前置条件) per the collaboration workflow and wait for confirmation.

0. Prerequisites

1. Record the code version under test: git rev-parse HEAD (+ git diff stat if the tree is dirty). The report must name the exact commit the numbers came from.
2. Check inputs exist (fail fast if any is missing). Paths below use the share roots CI_SHARE_MODEL / CI_SHARE_DATA — export them first, pointing at your cluster's model and data share directories:
   • GRPO: ${CI_SHARE_MODEL}/Qwen3-8B, ${CI_SHARE_DATA}/gsm8k/train-mini.jsonl, ${CI_SHARE_DATA}/gsm8k/test.jsonl
   • OPD: MODEL_PATH ${CI_SHARE_MODEL}/models--Qwen--Qwen3-VL-2B-Instruct/snapshots/89644892e4d85e24eaac8bacfd4f463576704203, teachers ${CI_SHARE_MODEL}/Qwen3-4B-Instruct-2507 and ${CI_SHARE_MODEL}/Qwen3-VL-4B-Instruct, data ${CI_SHARE_DATA}/on_policy_distillation/gsm8k_geo3k_{train,eval}.json

1. GRPO accuracy run

Launch (from the repo root):

bash -x examples/v1/scripts/run_rl.sh autotest/config/rl_qwen3_gsm8k_grpo.py "lmdeploy" \
    ${CI_SHARE_MODEL}/Qwen3-8B \
    ${CI_SHARE_DATA}/gsm8k/train-mini.jsonl \
    ${CI_SHARE_DATA}/gsm8k/test.jsonl

Shape: 3 epochs over 1000 gsm8k prompts, global_batch_size 64 → exactly 45 RL steps (the baseline has 45 step-summary lines; check_rl_result asserts equal step counts, so do not change total_epochs/global_batch_size/dataset size). Expected wall time: under 40 min.

Baseline: ${CI_SHARE_BASELINE}/qwen3-rl-lmdeploy/tracker.jsonl (CI_SHARE_BASELINE = the e2e baseline store root, i.e. the directory configured as base_path.base_baseline_path in autotest/config.yaml). Average of 5 historical runs.

Thresholds (from the GRPO e2e spec):

Compare by calling the existing checker — do not reimplement comparison logic. Run it from the results directory (it imports matplotlib and publish_comparison_report writes plots into $CWD/$GITHUB_RUN_ID, default ./0/):

import sys
sys.path.insert(0, "autotest")  # path to the repo's autotest dir
from utils.check_metric import check_rl_result

assert_info = {"check_metrics": [
    {"metric": "response/rewards/mean", "threshold": 0.1, "method": "absolute", "operator": "<"},
    {"metric": "mismatch/mismatch_k3_kl", "threshold": 0.0001, "method": "absolute", "operator": "<="},
    {"metric": "response/response_len/mean", "threshold": 0.12, "method": "relative", "operator": "<"},
    {"metric": "time/step", "threshold": 10, "method": "absolute", "operator": "<"},
]}
ok, msg = check_rl_result("grpo-gsm8k", BASE_TRACKER, CUR_TRACKER, assert_info)

Note: check_rl_result aggregates response/rewards/mean and response/response_len/mean with p80 across steps by default (RL_PERCENTILE_METRICS); other metrics are compared stepwise.

eval/accuracy must be compared separately: the baseline tracker records it only once, at the final step (step 45, value 0.9007), while the config evaluates at steps 0/15/30/45. Putting it in check_metrics fails with a series-length mismatch. Instead compare the final-step value directly (final-point-only check is deliberate — the stored baseline has no earlier eval points; OPD below checks every eval point because its golden values were recorded per point):

import json

def final_eval_acc(path):
    vals = [r["eval/accuracy"] for line in open(path)
            if "eval/accuracy" in (r := json.loads(line))]
    return vals[-1]

assert abs(final_eval_acc(CUR_TRACKER) - final_eval_acc(BASE_TRACKER)) < 0.05

2. OPD accuracy run (multi-teacher MOPD, rollout teachers)

Single 8-GPU node: 6 student workers + 2 teachers (1 replica x 1 worker each). The launch script starts the two teacher HTTP servers, injects XTUNER_OPD_TEACHER_ENDPOINTS_JSON, then runs the RL training. Verified scale for accuracy checks: TOTAL_TRAIN_STEPS=25 (constant LR, no warmup, so steps 0-25 are identical to the 50-step variant; the gsm8k gain shows by step 10). One 25-step run takes ~15 min on 8xH200.

export MODEL_PATH=${CI_SHARE_MODEL}/models--Qwen--Qwen3-VL-2B-Instruct/snapshots/89644892e4d85e24eaac8bacfd4f463576704203
export DATA_PATH=${CI_SHARE_DATA}/on_policy_distillation/gsm8k_geo3k_train.json
export EVAL_DATA_PATH=${CI_SHARE_DATA}/on_policy_distillation/gsm8k_geo3k_eval.json
export GSM8K_TEACHER_MODEL_PATH=${CI_SHARE_MODEL}/Qwen3-4B-Instruct-2507
export GEO3K_TEACHER_MODEL_PATH=${CI_SHARE_MODEL}/Qwen3-VL-4B-Instruct
export OPD_CONFIG_PATH=recipe/on_policy_distillation/config/rl_qwen3_vl_2b_sampled_token_mopd_rollout_teacher.py
export XTUNER_USE_LMDEPLOY=1 XTUNER_USE_FA3=1
export TOTAL_TRAIN_STEPS=25 TRAIN_BATCH_SIZE=128 EVALUATE_STEP=5
export XTUNER_RL_NUM_WORKERS=6 MOPD_TEACHER_NUM_REPLICAS=1 MOPD_TEACHER_REPLICA_NUM_WORKERS=1
export WORK_DIR=<work dir for this run>

# Positional args are required: STUDENT_MODEL_PATH DATA_PATH
bash recipe/on_policy_distillation/scripts/run_sampled_token_mopd.sh "$MODEL_PATH" "$DATA_PATH"

The script partitions the node's GPUs itself (teachers first, students get the rest via STUDENT_CUDA_VISIBLE_DEVICES) and starts/stops the teacher servers and Ray via an EXIT trap — do not set CUDA_VISIBLE_DEVICES yourself.

Baseline (golden values): 3-run mean measured 2026-10-26 on the config above at TOTAL_TRAIN_STEPS=25 (commit 4e7db028, lmdeploy main, ray 2.56, transformers 5.20.1). Metric mapping: distill_kl = distillation/reduced_distillation_kl, abs_loss = distillation/reduced_distillation_abs_loss, rewards = response/rewards/mean, gsm8k_acc = eval/gsm8k/acc, geo3k_acc = eval/geo3k/acc.

Compare at every eval point (steps 0/5/10/15/20/25), not just the final step. The current run's tracker is at $WORK_DIR/<exp>/logs/exp_tracking/tracker.jsonl; eval metrics are on the eval rows (step 0 is the eval-only row before training), training metrics on the RL step-summary rows (response/rewards/mean present; step N = row N).

Thresholds (measured max run-to-run deviation x 1.5 margin, N=3, 25 steps), applied per eval point against the golden table:

In k1 loss mode opd_reverse_kl == reduced_distillation_kl and opd_abs_logprob_loss == reduced_distillation_abs_loss (verified numerically identical) — do not check both.

The config's docstring holds a 50-step reference table (recorded 2026-07-31); steps 0-25 of a 25-step run match it (e.g. step-25 distill_kl 0.3165 vs 0.3167). Steps 30-50 have no calibrated thresholds yet — record a 50-step CI baseline before tightening that range.

Reference comparison snippet:

import json

def series(path, metric, marker="response/rewards/mean"):
    vals = []
    for line in open(path):
        r = json.loads(line)
        if isinstance(r, dict) and marker in r and metric in r:
            vals.append(r[metric])
    return vals

# eval metrics live on eval-point rows (steps 0/5/.../25), keyed separately:
# series(path, "eval/gsm8k/acc", marker="eval/gsm8k/acc")

3. Interpreting results

• A metric drifting past threshold at step 1 / first eval points to initialization or config differences (weight mapping, seed, data order), not the training loop.
• A metric that diverges gradually points to the training loop (loss, advantage, KL, weight sync).
• mismatch/mismatch_k3_kl breach means train/inference inconsistency (logprob path, weight update, rollout backend) — check that first for GRPO failures.
• OPD distill_kl breach with healthy rewards points at the teacher path (teacher endpoints, sampled tokens, distillation loss); rewards-only drift points at rollout/judger, not distillation.

4. Report

Report per run: PASS/FAIL -> metric table (baseline vs current vs error vs threshold) -> tracker and log paths -> commit under test -> suggested next step. Every number must point to a concrete tracker file; mark inference vs measurement explicitly. If anything failed or looks abnormal, report immediately rather than waiting to be asked.

5. Cleanup

ray stop --force if a run died mid-flight, and keep the copied trackers/logs under the project directory for the record.