Learned Value Function + Search Beats Heuristic Augmentation

When a strong hand-crafted heuristic resists every naive ML augmentation (search, behavioral cloning, type-classification), the failure isn't data volume or feature richness — it's the absence of a learned value function. The breakthrough comes from combining (a) a learned value network as a leaf evaluator with (b) sufficient search depth. Neither alone is enough.

This skill captures the hard-won methodology from a multi-week PTCG (Pokémon TCG) AI Battle project where a rank-304 rule-based agent (nursrijan_adv) defeated every naive ML upgrade — until a ReBeL-style value network + 2-ply search finally beat it.

The Anti-Pattern: Strong Baselines Resist Crude Augmentation

A strong heuristic encodes deep domain knowledge (matchup awareness, sequencing, timing). Naive ML augmentation disrupts its internal coherence:

• Search with a hand-crafted eval overrides good heuristic picks with worse ones (the eval is cruder than the heuristic it's trying to improve).
• Behavioral cloning on winners picks the wrong move ~45% of the time, and errors compound across a game. 10× more data doesn't fix this — 437K rows still lost 98%.
• Type-guided policy (predict action TYPE, let heuristic pick within type) breaks the heuristic's coherent sequencing — 6% win rate, worse than random.

Common failure signal: every augmentation approach lands in the same 5-45% range regardless of data volume or feature richness. If you see this, you're not fighting a data problem — you're fighting the coherence problem.

The Breakthrough: V_net + Search Depth, Together

The combination that finally wins:

1. Train a value network V(state) → P(win). This is a classification problem (who's-ahead), not an imitation problem. Use existing game logs with outcome labels. Targets: >70% accuracy, well-calibrated win-probabilities.
2. Use V_net as the leaf evaluator inside search (MCTS / minimax / forward-rollout). The search provides depth (sees the opponent's response); V_net provides the learned judgment at the leaf.
3. Override the heuristic only when V_net is confident (margin ≥ 0.03), so the heuristic's coherence is preserved on uncertain decisions.

Empirical result (PTCG, 200-game offline A/B):

• 2-ply search + hand-crafted eval: 42.5%
• 1-ply search + V_net: 45%
• 2-ply search + V_net: 59.0% (first statistically-significant win over the heuristic)

Both components are necessary. The learned eval fixes the search-augmentation failure; the search depth fixes the value-net's myopia.

Methodology: Phase-Gated with Kill Criteria

Don't commit to the full RL pipeline upfront. Use decision gates:

If Phase 1 fails the gate, enrich features — don't proceed to search. If Phase 2 fails, the value net is too weak — don't add depth. Early-stop on gates prevents sunk-cost on dead approaches (this project spent days on BC before V_net worked).

Diagnostics: Which Failure Mode Are You In?

Before choosing an approach, diagnose:

• V_net calibration plot (predicted win-prob vs actual win-rate buckets): if uncalibrated, features are missing signal. Fix before search.
• Early-game vs late-game V_net accuracy: if early-game << late-game, the model only learned "who has more prizes" — not positional judgment. Enrich features.
• Margin sweep (override threshold 0.02 / 0.03 / 0.05): if looser margin → worse win rate, V_net is noisy on near-eval states — keep the conservative margin.

When NOT to Apply This Skill

• The baseline is itself a learned model (standard tabular/CV/NLP ML). The "strong heuristic" framing doesn't apply — just improve the model directly.
• No simulator / no way to forward-rollout. Search needs a forward model. If you only have offline logs, stick with BC + careful evaluation.
• The baseline is weak (< median). Cheaper improvements (features, ensembling, hyperparameters) dominate — don't jump to RL.

Anti-Patterns (Forbidden)

• "More data will fix BC" — it won't, if the value function is missing. 10× data moved PTCG BC from 5.8% → 1.7% (worse, because the larger model overfit winner bias).
• "Search depth with hand-crafted eval" — depth amplifies eval errors. Always learn the eval first.
• "Augment the heuristic in-place" — coherence conflicts kill performance. Either wrap (heuristic handles fallback, learned model overrides on confidence) or replace entirely; don't interleave their logic.

Key Files (PTCG reference implementation)

• Value net training: train_value.py — HistGradientBoosting, 82 state features.
• Search+V_net agent: 2-ply rollout (heuristic-guided, globals isolated) + V_net leaf.
• Decision-gate metrics: accuracy, AUC, calibration buckets, early/mid/late split.

Source

PTCG AI Battle Challenge (Kaggle, 2026-06). Sessions 5-6: 8 failed augmentation approaches → V_net (89.4% acc) + 2-ply search = 59.0% vs rank-304 heuristic. Full plan in the project's DEEP_RL_PLAN.md (ReBeL architecture, 5 phases).

Session 8 update (2026-06-29): v29 baseline ladder-drifted from 970 → 770 LB (48h, Starmie/Archaludon/Dragapult meta shift). All "meta-aware" patches (v33, v34) regressed by 10-13% vs Archaludon in 200-game tests despite 60-game tests showing "neutral". See [[ladder-drift-meta-aware-regression]] for the meta-aware regression anti-pattern. Final state: v29 stays as best (770.5 LB, plateau).