retrain-ai-weights
Skill331 starsChanged 4 months agoApache-2.0View on GitHub
Use when retraining AI evaluation weights from 17Lands replay data, adding new training datasets, updating learned weight values in phase-ai Rust code, or running CMA-ES optimization for AI profiles and evaluation parameters.
In this repository33 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- add-ai-feature-policySkill
- add-card-data-pipelineSkill
- add-engine-effectSkill
- add-engine-variantSkill
- add-frontend-componentSkill
- add-interactive-effectSkill
- add-keywordSkill
- add-replacement-effectSkill
- address-reviewSkill
- add-static-abilitySkill
- add-triggerSkill
- ai-duelSkill
- audit-card-parsingSkill
- batch-mechanicsSkill
- bug-coverage-classifierSkill
- bug-triageSkill
- card-testSkill
- casting-stack-conditionsSkill
- changelogSkill
- engine-implementerSkill
- engine-plannerSkill
- oracle-parserSkill
- parser-velocitySkill
- pr-contribution-handlerSkill
- pr-flow-auditSkill
- project-referenceSkill
- pr-review-loopSkill
- review-engine-implSkill
- review-engine-planSkill
- ship-commitsSkill
- unlock-setSkill
- validate-cr-annotationsSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 2 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 8
Use when the user wants to retrain AI weights from 17Lands data, add new training datasets, update learned weight values in Rust, or run …
- Very specific: line 20
All stored in AiConfig. The CMA-ES optimizer tunes one parameter group per run via --group eval|penalties|keywords|archetype:
- Distinctive: line 28
Do not mix groups in one run.
- Very specific: line 71
Read data/learned-weights.json and update EvalWeightSet::learned() in crates/phase-ai/src/eval.rs:
- Very specific: line 100
If Tilt is not running (rtk tilt get uiresource clippy fails), use the project-reference skill before falling back to direct cargo commands.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Retrain AI Evaluation Weights
Use when the user wants to retrain AI weights from 17Lands data, add new training datasets, update learned weight values in Rust, or run CMA-ES optimization.
Architecture Overview
The AI weight system has 4 layers:
- Base weights (
EvalWeightSetincrates/phase-ai/src/eval.rs) — 9 weights × 3 game phases (early T1-3, mid T4-7, late T8+). Learned from 17Lands replay data. - Archetype multipliers (
ArchetypeMultipliersincrates/phase-ai/src/deck_profile.rs) — 5 archetypes × 9 multipliers. Scale base weights per deck type. - Keyword bonuses (
KeywordBonusesincrates/phase-ai/src/eval.rs) — 10 params for creature evaluation. - Policy penalties (
PolicyPenaltiesincrates/phase-ai/src/config.rs) — tactical policy score knobs. - AiProfile (
crates/phase-ai/src/config.rs) — 3 params (risk_tolerance, interaction_patience, stabilize_bias).
All stored in AiConfig. The CMA-ES optimizer tunes one parameter group per
run via --group eval|penalties|keywords|archetype:
eval: 9 late-gameEvalWeightsplus 3AiProfilevalues. Early/mid weights are derived from the 17Lands phase ratios.penalties: every field listed inACTIVE_POLICY_PENALTY_FIELDS.keywords: allKeywordBonusesfields.archetype: 5 archetypes x 9ArchetypeMultipliers.
Do not mix groups in one run. Compare and validate one group artifact at a time so regressions can be attributed to a specific surface.
Training Data Setup
Data location: data/17lands/ (gitignored)
Required files from 17Lands (https://www.17lands.com/public_datasets):
replay_data_public.{SET}.PremierDraft.csv— Per-turn board state snapshots. Premier Draft (Bo1) is best: largest dataset, no sideboard confounds, human-drafted decks.cards.csv— Arena card ID to mana value mapping.
To add new sets: Download CSVs and symlink or copy into data/17lands/:
ln -s ~/Downloads/replay_data_public.FDN.PremierDraft.csv data/17lands/
ln -s ~/Downloads/replay_data_public.DSK.PremierDraft.csv data/17lands/
# cards.csv only needed once (shared across sets)
ln -s ~/Downloads/cards.csv data/17lands/The script auto-discovers all replay_data_public.*.PremierDraft.csv files.
Retraining Steps
Step 1: Run the training script
rtk python3 scripts/train_eval_weights.py --data-dir data/17lands --output data/learned-weights.jsonDependencies: pip3 install -r scripts/requirements-training.txt (pandas, scikit-learn, numpy)
What it does:
- Streams all replay CSVs with skill filter (win_rate >= 0.55, games >= 50)
- Splits samples into 3 turn-phase buckets (early T1-3, mid T4-7, late T8+)
- Trains separate logistic regression per phase
- Maps 5 features to EvalWeights fields: life_diff→life, creature_count_diff→board_presence, creature_mv_diff→board_power, hand_diff→hand_size, non_creature_diff→card_advantage
- Scales so max coefficient = 2.5
- 4 weights stay hand-tuned: board_toughness=1.0, aggression=0.5, zone_quality=0.3, synergy=0.5
Output: data/learned-weights.json with per-phase weights and accuracy metrics.
Step 2: Update Rust with new values
Read data/learned-weights.json and update EvalWeightSet::learned() in crates/phase-ai/src/eval.rs:
pub fn learned() -> Self {
EvalWeightSet {
early: EvalWeights {
life: /* phases.early.weights.life */,
aggression: /* phases.early.weights.aggression */,
board_presence: /* phases.early.weights.board_presence */,
board_power: /* phases.early.weights.board_power */,
board_toughness: /* phases.early.weights.board_toughness */,
hand_size: /* phases.early.weights.hand_size */,
zone_quality: /* phases.early.weights.zone_quality */,
card_advantage: /* phases.early.weights.card_advantage */,
synergy: /* phases.early.weights.synergy */,
},
mid: EvalWeights { /* same pattern from phases.mid.weights */ },
late: EvalWeights { /* same pattern from phases.late.weights */ },
}
}Step 3: Verify
rtk cargo fmt --all
rtk ./scripts/tilt-wait.sh --timeout 420 clippy test-aiIf Tilt is not running (rtk tilt get uiresource clippy fails), use the
project-reference skill before falling back to direct cargo commands.
Step 4 (optional): CMA-ES optimization
# Smoke test (fast, verifies binary works)
rtk cargo tune-ai data/ --group eval --generations 2 --population 5 --games 3 --seed 42
# Full eval/profile run
rtk cargo tune-ai data/ --group eval --generations 100 --population 50 --games 20 --output data/cma-tuned-eval.json
# Full policy-penalty run
rtk cargo tune-ai data/ --group penalties --generations 100 --population 50 --games 20 --output data/cma-tuned-penalties.json
# Full keyword-bonus run
rtk cargo tune-ai data/ --group keywords --generations 100 --population 50 --games 20 --output data/cma-tuned-keywords.json
# Full archetype-multiplier run
rtk cargo tune-ai data/ --group archetype --generations 100 --population 50 --games 20 --output data/cma-tuned-archetype.json
# Validate a tuned artifact against paired holdout matchups and Easy/Medium/Hard opponents
rtk cargo tune-ai data/ --validate --games 500 --output data/cma-tuned-eval.jsonCMA-ES writes the requested artifact plus a sibling *-manifest.json
containing the git SHA, seed, group, parameter names, fitness decks, holdout
decks, opponent pool, games/eval, paired-seed flag, draw-exclusion flag, and
baseline config hash. Do not write CMA output to data/learned-weights.json;
that path is reserved for the 17Lands logistic-regression artifact
(kind: 17lands_phase_weights). After a full run, review the validation
output before manually updating Rust defaults.
Fitness uses registered duel_suite matchup IDs from the fitness split and
paired mirrored seeds. Drawn games are excluded from the fitness denominator.
Holdout validation uses a separate registered split and compares baseline vs
learned on the same seeds against Easy, Medium, and Hard opponent configs.
Commander sanity measurement lives in ai-duel, not ai-tune:
rtk cargo run --release --bin ai-duel -- client/public --commander-suite --games 8 --seed 42 \
--difficulty Hard --baseline-difficulty Medium \
--output target/commander-suite-results.jsonThis runs the candidate seat through four seat rotations against three baseline seats and reports win rate, survival turns, and elimination order.
Key Files
| File | Purpose |
scripts/train_eval_weights.py | Python training pipeline |
scripts/requirements-training.txt | Python deps (pandas, scikit-learn, numpy) |
data/learned-weights.json | Trained weight artifact (committed) |
data/cma-tuned-weights.json | CMA-ES tuned artifact (generated, not a 17Lands artifact) |
data/cma-tuned-weights-manifest.json | CMA-ES reproducibility manifest |
data/17lands/ | Raw 17Lands CSVs (gitignored) |
crates/phase-ai/src/eval.rs | EvalWeights, EvalWeightSet, KeywordBonuses, evaluation functions |
crates/phase-ai/src/deck_profile.rs | ArchetypeMultipliers, deck classification |
crates/phase-ai/src/config.rs | AiConfig with all tunable params |
crates/phase-ai/src/bin/ai_tune.rs | CMA-ES optimizer binary |
crates/phase-ai/src/bin/ai_duel.rs | Duel-suite, compare, and Commander measurement binary |
EvalWeights Fields (9 total)
| Field | 17Lands Feature | Measures |
life | life_diff | Life total differential |
board_presence | creature_count_diff | Creature count differential |
board_power | creature_mv_diff | Total mana value of creatures |
hand_size | hand_diff | Cards in hand differential |
card_advantage | non_creature_diff | Non-creature, non-land permanents |
board_toughness | — | Total toughness (hand-tuned) |
aggression | — | Power bonus when ahead on life (hand-tuned) |
zone_quality | — | Hand quality + graveyard value (hand-tuned) |
synergy | — | Board synergy bonus (hand-tuned) |
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

