agentleFS
phase-rsphase-rs/phase/.claude/skills/retrain-ai-weights/SKILL.md

retrain-ai-weights

Skill331 starsChanged 4 months agoApache-2.0View on GitHub

Use when retraining AI evaluation weights from 17Lands replay data, adding new training datasets, updating learned weight values in phase-ai Rust code, or running CMA-ES optimization for AI profiles and evaluation parameters.

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 2 are distinctive, and 3 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Use when the user wants to retrain AI weights from 17Lands data, add new training datasets, update learned weight values in Rust, or run …

    line 8
  2. Very specific:

    All stored in AiConfig. The CMA-ES optimizer tunes one parameter group per run via --group eval|penalties|keywords|archetype:

    line 20
  3. Distinctive:

    Do not mix groups in one run.

    line 28
  4. Very specific:

    Read data/learned-weights.json and update EvalWeightSet::learned() in crates/phase-ai/src/eval.rs:

    line 71
  5. Very specific:

    If Tilt is not running (rtk tilt get uiresource clippy fails), use the project-reference skill before falling back to direct cargo commands.

    line 100

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Retrain AI Evaluation Weights

Use when the user wants to retrain AI weights from 17Lands data, add new training datasets, update learned weight values in Rust, or run CMA-ES optimization.

Architecture Overview

The AI weight system has 4 layers:

  1. Base weights (EvalWeightSet in crates/phase-ai/src/eval.rs) — 9 weights × 3 game phases (early T1-3, mid T4-7, late T8+). Learned from 17Lands replay data.
  2. Archetype multipliers (ArchetypeMultipliers in crates/phase-ai/src/deck_profile.rs) — 5 archetypes × 9 multipliers. Scale base weights per deck type.
  3. Keyword bonuses (KeywordBonuses in crates/phase-ai/src/eval.rs) — 10 params for creature evaluation.
  4. Policy penalties (PolicyPenalties in crates/phase-ai/src/config.rs) — tactical policy score knobs.
  5. AiProfile (crates/phase-ai/src/config.rs) — 3 params (risk_tolerance, interaction_patience, stabilize_bias).

All stored in AiConfig. The CMA-ES optimizer tunes one parameter group per run via --group eval|penalties|keywords|archetype:

  • eval: 9 late-game EvalWeights plus 3 AiProfile values. Early/mid weights are derived from the 17Lands phase ratios.
  • penalties: every field listed in ACTIVE_POLICY_PENALTY_FIELDS.
  • keywords: all KeywordBonuses fields.
  • archetype: 5 archetypes x 9 ArchetypeMultipliers.

Do not mix groups in one run. Compare and validate one group artifact at a time so regressions can be attributed to a specific surface.

Training Data Setup

Data location: data/17lands/ (gitignored)

Required files from 17Lands (https://www.17lands.com/public_datasets):

  • replay_data_public.{SET}.PremierDraft.csv — Per-turn board state snapshots. Premier Draft (Bo1) is best: largest dataset, no sideboard confounds, human-drafted decks.
  • cards.csv — Arena card ID to mana value mapping.

To add new sets: Download CSVs and symlink or copy into data/17lands/:

ln -s ~/Downloads/replay_data_public.FDN.PremierDraft.csv data/17lands/
ln -s ~/Downloads/replay_data_public.DSK.PremierDraft.csv data/17lands/
# cards.csv only needed once (shared across sets)
ln -s ~/Downloads/cards.csv data/17lands/

The script auto-discovers all replay_data_public.*.PremierDraft.csv files.

Retraining Steps

Step 1: Run the training script

rtk python3 scripts/train_eval_weights.py --data-dir data/17lands --output data/learned-weights.json

Dependencies: pip3 install -r scripts/requirements-training.txt (pandas, scikit-learn, numpy)

What it does:

  • Streams all replay CSVs with skill filter (win_rate >= 0.55, games >= 50)
  • Splits samples into 3 turn-phase buckets (early T1-3, mid T4-7, late T8+)
  • Trains separate logistic regression per phase
  • Maps 5 features to EvalWeights fields: life_diff→life, creature_count_diff→board_presence, creature_mv_diff→board_power, hand_diff→hand_size, non_creature_diff→card_advantage
  • Scales so max coefficient = 2.5
  • 4 weights stay hand-tuned: board_toughness=1.0, aggression=0.5, zone_quality=0.3, synergy=0.5

Output: data/learned-weights.json with per-phase weights and accuracy metrics.

Step 2: Update Rust with new values

Read data/learned-weights.json and update EvalWeightSet::learned() in crates/phase-ai/src/eval.rs:

pub fn learned() -> Self {
    EvalWeightSet {
        early: EvalWeights {
            life: /* phases.early.weights.life */,
            aggression: /* phases.early.weights.aggression */,
            board_presence: /* phases.early.weights.board_presence */,
            board_power: /* phases.early.weights.board_power */,
            board_toughness: /* phases.early.weights.board_toughness */,
            hand_size: /* phases.early.weights.hand_size */,
            zone_quality: /* phases.early.weights.zone_quality */,
            card_advantage: /* phases.early.weights.card_advantage */,
            synergy: /* phases.early.weights.synergy */,
        },
        mid: EvalWeights { /* same pattern from phases.mid.weights */ },
        late: EvalWeights { /* same pattern from phases.late.weights */ },
    }
}

Step 3: Verify

rtk cargo fmt --all
rtk ./scripts/tilt-wait.sh --timeout 420 clippy test-ai

If Tilt is not running (rtk tilt get uiresource clippy fails), use the project-reference skill before falling back to direct cargo commands.

Step 4 (optional): CMA-ES optimization

# Smoke test (fast, verifies binary works)
rtk cargo tune-ai data/ --group eval --generations 2 --population 5 --games 3 --seed 42

# Full eval/profile run
rtk cargo tune-ai data/ --group eval --generations 100 --population 50 --games 20 --output data/cma-tuned-eval.json

# Full policy-penalty run
rtk cargo tune-ai data/ --group penalties --generations 100 --population 50 --games 20 --output data/cma-tuned-penalties.json

# Full keyword-bonus run
rtk cargo tune-ai data/ --group keywords --generations 100 --population 50 --games 20 --output data/cma-tuned-keywords.json

# Full archetype-multiplier run
rtk cargo tune-ai data/ --group archetype --generations 100 --population 50 --games 20 --output data/cma-tuned-archetype.json

# Validate a tuned artifact against paired holdout matchups and Easy/Medium/Hard opponents
rtk cargo tune-ai data/ --validate --games 500 --output data/cma-tuned-eval.json

CMA-ES writes the requested artifact plus a sibling *-manifest.json containing the git SHA, seed, group, parameter names, fitness decks, holdout decks, opponent pool, games/eval, paired-seed flag, draw-exclusion flag, and baseline config hash. Do not write CMA output to data/learned-weights.json; that path is reserved for the 17Lands logistic-regression artifact (kind: 17lands_phase_weights). After a full run, review the validation output before manually updating Rust defaults.

Fitness uses registered duel_suite matchup IDs from the fitness split and paired mirrored seeds. Drawn games are excluded from the fitness denominator. Holdout validation uses a separate registered split and compares baseline vs learned on the same seeds against Easy, Medium, and Hard opponent configs.

Commander sanity measurement lives in ai-duel, not ai-tune:

rtk cargo run --release --bin ai-duel -- client/public --commander-suite --games 8 --seed 42 \
  --difficulty Hard --baseline-difficulty Medium \
  --output target/commander-suite-results.json

This runs the candidate seat through four seat rotations against three baseline seats and reports win rate, survival turns, and elimination order.

Key Files

FilePurpose
scripts/train_eval_weights.pyPython training pipeline
scripts/requirements-training.txtPython deps (pandas, scikit-learn, numpy)
data/learned-weights.jsonTrained weight artifact (committed)
data/cma-tuned-weights.jsonCMA-ES tuned artifact (generated, not a 17Lands artifact)
data/cma-tuned-weights-manifest.jsonCMA-ES reproducibility manifest
data/17lands/Raw 17Lands CSVs (gitignored)
crates/phase-ai/src/eval.rsEvalWeights, EvalWeightSet, KeywordBonuses, evaluation functions
crates/phase-ai/src/deck_profile.rsArchetypeMultipliers, deck classification
crates/phase-ai/src/config.rsAiConfig with all tunable params
crates/phase-ai/src/bin/ai_tune.rsCMA-ES optimizer binary
crates/phase-ai/src/bin/ai_duel.rsDuel-suite, compare, and Commander measurement binary

EvalWeights Fields (9 total)

Field17Lands FeatureMeasures
lifelife_diffLife total differential
board_presencecreature_count_diffCreature count differential
board_powercreature_mv_diffTotal mana value of creatures
hand_sizehand_diffCards in hand differential
card_advantagenon_creature_diffNon-creature, non-land permanents
board_toughness—Total toughness (hand-tuned)
aggression—Power bonus when ahead on life (hand-tuned)
zone_quality—Hand quality + graveyard value (hand-tuned)
synergy—Board synergy bonus (hand-tuned)

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.