AILANG Post-Release Tasks

Run post-release tasks for an AILANG release: evaluation baselines, dashboard updates, and documentation.

Current State

• Current version: !'cat std/VERSION'
• Latest tag: !'git describe --tags --abbrev=0 2>/dev/null || echo "no tags"'
• GitHub release: !'gh release list --limit 3 2>/dev/null | head -3 || echo "gh not available"'
• Existing baselines: !'ls eval_results/baselines/ 2>/dev/null | tail -5 || echo "none"'
• Dashboard version: !'jq -r .version docs/static/benchmarks/latest.json 2>/dev/null || echo "not found"'
• Active changelog: !'ls changelogs/ | grep current 2>/dev/null'

Use the data above first. Only re-run these commands manually if the injected context is empty or you need to refresh after making changes.

Quick Start

Most common usage:

# User says: "Run post-release tasks for v0.3.14"
# This skill will:
# 1. Run eval baseline (extended_suite: 7 production models + lang-harness sweep) - ALWAYS USE --full FOR RELEASES
# 2. Update website dashboard (JSON with history preservation)
# 3. Update axiom scorecard KPI (if features affect axiom compliance)
# 4. Extract metrics and UPDATE CHANGELOG.md automatically
# 5. Move design docs from planned/ to implemented/
# 6. Run docs-sync to verify website accuracy (version constants, PLANNED banners, examples)
# 7. Commit all changes to git

🚨 CRITICAL: For releases, ALWAYS use --full flag by default

• Dev models (without --full) are only for quick testing/validation, NOT releases
• Users expect full benchmark results when they say "post-release" or "update dashboard"
• Never start with dev models and then try to add production models later

When to Use This Skill

Invoke this skill when:

• User says "post-release tasks", "update dashboard", "run benchmarks"
• After successful release (once GitHub release is published)
• User asks about eval baselines or benchmark results
• User wants to update documentation after a release

Available Scripts

scripts/run_eval_baseline.sh <version> [--full] [--cross-harness]

Run evaluation baseline for a release version.

🚨 CRITICAL: ALWAYS use --full for releases!

Usage:

# ✅ RECOMMENDED release baseline (standard + agent + 4-language Explorer sweep) — ~$23
.Codex/skills/post-release/scripts/run_eval_baseline.sh 0.15.0 --full --lang-harness

# Standard + agent only (no 4-lang Explorer sweep) — ~$16
.Codex/skills/post-release/scripts/run_eval_baseline.sh v0.15.0 --full

# Major release: includes cross-harness comparison (gpt5-5 + opencode-gpt5-5 etc.) — ~$47
.Codex/skills/post-release/scripts/run_eval_baseline.sh 0.15.0 --full --cross-harness

# ❌ Dev only — 3 cheap models, AILANG lang only (quick testing/validation) — ~$3.50
.Codex/skills/post-release/scripts/run_eval_baseline.sh 0.15.0

Output:

Running eval baseline for 0.3.14...
Mode: FULL (extended_suite: 7 production models)
Expected cost: ~$16 (FULL) or ~$23 (FULL + lang-harness) or ~$47 (FULL + cross-harness)
Expected time: ~30-60 minutes

[Running benchmarks...]

✓ Baseline complete
  Results: eval_results/baselines/0.3.14
  Files: 726 result files

What it does:

• Step 1: Standard eval (0-shot + self-repair)
  • Uses extended_suite (--full, 10 models): gpt5-5, gpt5-4-mini, Codex-opus-4-8 (Jun 2026 flagship), Codex-sonnet-4-6, gemini-3-1-pro, gemini-3-flash, or-glm-5-1, or-minimax-m3, or-deepseek-v4-flash, or-deepseek-v4-pro (modern OS refreshed 2026-06-04)
  • Or dev_models (default): gpt5-4-mini, Codex-haiku-4-5, gemini-3-flash
  • Both AILANG and Python; all benchmarks in selected tier(s)
  • Cloud-vs-OS note: in standard mode the best OS model (glm-5.1, 90% de-flaked) MATCHES the best cloud model (gemini-3-1-pro, 90%) at ~½ the cost; minimax-m3 (87%, $0.30/1M) ties opus/gpt5-5 (87%, $5/1M) at ~1/16 the cost.
• Step 2: Agent eval — AILANG-only (redesigned 2026-07-11). Agent mode measures the agent-loop uplift on WEAK models, so the subjects are the free on-device GPU models; a small lab sample gives reference signal. It does not run the expensive multi-turn cloud fleet (near-ceiling cloud models add little agent signal at high $).
  • agent_suite (7 cloud weak+reference models, run in parallel): gpt5-6-luna (codex — OpenAI weak/fast), Codex-haiku-4-5 (Codex — Anthropic weak), Codex-sonnet-4-6 (Codex — longitudinal anchor), opencode-or-deepseek-v4-pro (OS agent champion), opencode-or-deepseek-v4-flash (OS best-value), opencode-or-glm-5-1 + opencode-or-glm-5-2 (settle whether 5.2 is actually good).
  • On-device GPU models are NOT in this suite — they are covered continuously by the daily rig rotation (dev.ailang.os-rotation-filler → eval_results/rotation/os-rolling, --agent --bank-by-version). For the on-device-vs-cloud agent table, aggregate the rotation's per-version GPU data with this suite's results.
  • motoko- removed 2026-06-04*: the AILANG-native motoko/bun harness hangs on the rig (0 completions, orphans subprocesses). Pending agent-harness-instability diagnosis; re-add when reliable.
  • Tier system (v0.14.0+, frontier added v0.29.0): smoke (23), core (19), stretch (21), frontier (16), vision (9) — counts as of the 2026-07-11 v0.29.2 re-tier (8 stretch→frontier promotions from the ELO/frontier-model-fail audit; demotions deferred to the post-agent audit)
  • 🚫 Never run smoke for cloud models. Smoke is the cheap/fast sanity tier for the local OS-model iteration loop (the nightly rig, Ollama, de-flaking). Cloud/API models (Anthropic, OpenAI, Google, OpenRouter) go straight to core,stretch,frontier — smoke would just spend API budget re-confirming saturated benchmarks every model already passes, with zero added signal. The only time smoke joins a cloud run is an explicit --tier smoke,core,stretch,frontier full audit.
  • Default scope: core,stretch,frontier — Core is the headline metric, Stretch is harder mixed results, Frontier is the top-end discriminator (release baselines are its only routine data source)
  • Expected: core 70%+ for AILANG; vision intentionally low
  • Feeds the ailang-vs-python comparison story in the Model Leaderboard page
• Step 3 (--lang-harness): Language × Harness sweep — cheapest models × 4 languages
  • lang_harness_suite: Codex-haiku-4-5, gemini-3-flash, gpt5-4-mini, opencode-haiku
  • All 4 languages: ailang, python, javascript, go
  • Tier: core only (19 benchmarks) — stretch/frontier are skipped here even if a wider --tier was set globally
  • Note: 4 core benchmarks are AILANG/Python-only (contract_bst_validate, contract_roman_numeral, effect_composition, effect_tracking_io_fs) and auto-skip on JS/Go runs
  • Feeds the Agent Harness Explorer language spread and cross-harness comparison data
  • Cost: ~$7 extra
• --cross-harness: Replaces Step 2 with harness_suite (6 models, paired across harnesses)
  • Codex-sonnet-4-6 + opencode-sonnet-4-6, gemini-3-flash + opencode-gemini-3-flash, gpt5-5 + opencode-gpt5-5
  • Cost: ~$31 extra vs base FULL (3x)
• Saves combined results to eval_results/baselines/vX.X.X/
• Accepts version with or without 'v' prefix

⚠️ Note: gpt5-5-pro is in models.yml but not in any default suite — agent mode is blocked (codex rejects with ChatGPT account, opencode returns 0 tool calls). Don't add it to suites.

scripts/update_dashboard.sh <version>

Update website benchmark dashboard with new release data.

Usage:

.Codex/skills/post-release/scripts/update_dashboard.sh 0.3.14

Output:

Updating dashboard for 0.3.14...

1/5 Generating Docusaurus markdown...
  ✓ Written to docs/docs/benchmarks/performance.md

2/5 Generating dashboard JSON with history...
  ✓ Written to docs/static/benchmarks/latest.json (history preserved)

3/5 Validating JSON...
  ✓ Version: 0.3.14
  ✓ Success rate: 0.627

4/5 Clearing Docusaurus cache...
  ✓ Cache cleared

5/5 Summary
  ✓ Dashboard updated for 0.3.14
  ✓ Markdown: docs/docs/benchmarks/performance.md
  ✓ JSON: docs/static/benchmarks/latest.json

Next steps:
  1. Test locally: cd docs && npm start
  2. Visit: http://localhost:3000/ailang/docs/benchmarks/performance
  3. Verify timeline shows 0.3.14
  4. Commit: git add docs/docs/benchmarks/performance.md docs/static/benchmarks/latest.json
  5. Commit: git commit -m 'Update benchmark dashboard for 0.3.14'
  6. Push: git push

What it does:

• Generates Docusaurus-formatted markdown
• Updates dashboard JSON with history preservation
• Validates JSON structure (version matches input exactly)
• Clears Docusaurus build cache
• Provides next steps for testing and committing
• Accepts version with or without 'v' prefix

scripts/extract_changelog_metrics.sh [json_file]

Extract benchmark metrics from dashboard JSON for CHANGELOG.

Usage:

.Codex/skills/post-release/scripts/extract_changelog_metrics.sh
# Or specify JSON file:
.Codex/skills/post-release/scripts/extract_changelog_metrics.sh docs/static/benchmarks/latest.json

Output:

Extracting metrics from docs/static/benchmarks/latest.json...

=== CHANGELOG.md Template ===

### Benchmark Results (M-EVAL)

**Overall Performance**: 59.1% success rate (399 total runs)

**By Language:**
- **AILANG**: 33.0% - New language, learning curve
- **Python**: 87.0% - Baseline for comparison
- **Gap: 54.0 percentage points (expected for new language)

**Comparison**: -15.2% AILANG regression from 0.3.14 (48.2% → 33.0%)

=== End Template ===

Use this template in CHANGELOG.md for 0.3.15

What it does:

• Parses dashboard JSON for metrics
• Calculates percentages and gap between AILANG/Python
• Auto-compares with previous version from history
• Formats comparison text automatically (+X% improvement or -X% regression)
• Generates ready-to-paste CHANGELOG template with no manual work needed

scripts/cleanup_design_docs.sh <version> [--dry-run] [--force] [--check-only]

Move design docs from planned/ to implemented/ after a release.

Features:

• Detects duplicates (docs already in implemented/)
• Detects misplaced docs (Target: field doesn't match folder version)
• Only moves docs with "Status: Implemented" in their frontmatter
• Docs with other statuses are flagged for review

Usage:

# Check-only: Report issues without making changes
.Codex/skills/post-release/scripts/cleanup_design_docs.sh 0.5.9 --check-only

# Preview what would be moved/deleted/relocated
.Codex/skills/post-release/scripts/cleanup_design_docs.sh 0.5.9 --dry-run

# Execute: Move implemented docs, delete duplicates, relocate misplaced
.Codex/skills/post-release/scripts/cleanup_design_docs.sh 0.5.9

# Force move all docs regardless of status
.Codex/skills/post-release/scripts/cleanup_design_docs.sh 0.5.9 --force

Output:

Design Doc Cleanup for v0.5.9
==================================

Checking 5 design doc(s) in design_docs/planned/v0_5_9/:

Phase 1: Detecting issues...

  [DUPLICATE] m-fix-if-else-let-block.md
              Already exists in design_docs/implemented/v0_5_9/
  [MISPLACED] m-codegen-value-types.md
              Target: v0.5.10 (folder: v0_5_9)
              Should be in: design_docs/planned/v0_5_10/

Issues found:
  - 1 duplicate(s) (can be deleted)
  - 1 misplaced doc(s) (wrong version folder)

Phase 2: Processing docs...

  [DELETED] m-fix-if-else-let-block.md (duplicate - already in implemented/)
  [RELOCATED] m-codegen-value-types.md → design_docs/planned/v0_5_10/
  [MOVED] m-dx11-cycles.md → design_docs/implemented/v0_5_9/
  [NEEDS REVIEW] m-unfinished-feature.md
                 Found: **Status**: Planned

Summary:
  ✓ Deleted 1 duplicate(s)
  ✓ Relocated 1 doc(s) to correct version folder
  ✓ Moved 1 doc(s) to design_docs/implemented/v0_5_9/
  ⚠ 1 doc(s) need review (not marked as Implemented)

What it does:

• Phase 1 (Detection): Identifies duplicates and misplaced docs
• Phase 2 (Processing):
  • Deletes duplicates (same file already exists in implemented/)
  • Relocates misplaced docs (Target: version doesn't match folder)
  • Moves docs with "Status: Implemented" to implemented/
  • Flags docs without Implemented status for review
• Creates target folders if needed
• Removes empty planned folder after cleanup
• Use --check-only to only report issues, --dry-run to preview actions, --force to move all

Post-Release Workflow

1. Verify Release Exists

git tag -l vX.X.X
gh release view vX.X.X

If release doesn't exist, run release-manager skill first.

1a. Refresh Codebase Statistics (REQUIRED — commit, don't rely on CI)

The docs/static/codebase_stats.json file drives the Codebase Statistics page (LOC/token/commit growth chart).

⚠️ The deploy workflow regenerates this file at build time but does NOT commit it back — it only bakes the result into the Pages artifact. The generator (generate_codebase_stats.sh) appends only the version it runs on to the committed history. So if a release is not captured + committed here, that version is permanently skipped from the history chart, producing visible gaps (e.g. the 0.14.1 → 0.25.0 jump that skipped every 0.15–0.24 graduation). You must run + commit this every release.

# Generate stats for THIS release and append to the committed history
AILANG_VERSION=vX.X.X bash tools/generate_codebase_stats.sh

# Sanity check: current == this release, and the new entry is in history
jq -r '.current.version, (.history[-1].version)' docs/static/codebase_stats.json

# Commit (CI will NOT do this for you)
git add docs/static/codebase_stats.json
git commit -m "data(stats): codebase statistics for vX.X.X"
git push

• codebase_stats.json current shows vX.X.X
• history includes a vX.X.X entry (no gap vs. previous release)
• Change is committed and pushed (not just regenerated locally)

Backfilling a missed gap: check out each missed tag in a throwaway git worktree, run the same counting logic, and merge the entries into history sorted by version. See the v0.15→0.24 backfill (June 2026) for the pattern.

1b. Benchmark data provenance snapshot (REQUIRED — this is the ONLY routine commit of this data)

The dashboard fetches the benchmark JSONs at runtime from the GCS bucket via the dashboard's /benchmarks/ route (M-EVAL-DATA-HOSTING-DECOUPLE), so the rig no longer commits them every 45 minutes — W5 retired that churn. The committed copies now serve two purposes only:

1. The in-build fallback the site degrades to when the Cloud Run route is unreachable.
2. Release provenance — an immutable record of what the numbers were at vX.X.X.

Between releases the working tree carries these files as modified-but-uncommitted; that drift is expected ("the bucket has newer data than HEAD"). This step is what sweeps it up. Skip it and the fallback copy silently ages another release, so an outage would serve stale numbers.

# The rig regenerates these continuously; commit the current state as the release snapshot.
git add docs/static/benchmarks/latest.json \
        docs/static/benchmarks/os/latest.json \
        docs/static/benchmarks/os/history.json
git commit -m "data(bench): release provenance snapshot for vX.X.X"
git push

# Verify the runtime path agrees with what you just committed
curl -s https://ailang-dev-dashboard-ejjw6zt3bq-ew.a.run.app/benchmarks/os/latest.json \
  | jq -r '.ailang_version, .version, .generated'

• All 3 JSONs committed and pushed
• latest.json ailang_version shows vX.X.X (not the previous release)
• The runtime route returns the same ailang_version (bucket and git agree at release time)

If the runtime route and the committed copy disagree, the bucket sync is failing — check bucket sync lines in /tmp/ailang-os-filler.log before shipping the release notes.

2. Run Eval Baseline

🚨 CRITICAL: ALWAYS use --full for releases!

Correct workflow:

# ✅ RECOMMENDED for releases — adds 4-language Explorer data for ~$7 more
.Codex/skills/post-release/scripts/run_eval_baseline.sh X.X.X --full --lang-harness

# Minimum acceptable for releases
.Codex/skills/post-release/scripts/run_eval_baseline.sh X.X.X --full

This runs all 7 production models (extended_suite) with both AILANG and Python.

Tier scope for releases (counts re-centered after the v0.29.0 M-EVAL-FRONTIER-TIER re-tier: 7 saturated core benchmarks demoted to stretch, 8 stretch promoted to the new frontier tier):

• Default (release): --tier core,stretch,frontier — 56 benchmarks (19 core + 29 stretch + 8 frontier). This is the tier for every cloud/API model — never smoke. Smoke is the local OS-model iteration tier only (see the 🚫 note above); a cloud model added to a release baseline (e.g. a new Anthropic/OpenAI/Google model) goes straight to core,stretch,frontier.
• frontier (8): the anti-saturation discriminator tier — a frontier benchmark's defining property is that at least one frontier model FAILS it in standard mode; if every frontier model passes it, it demotes back to stretch (CURATION.md §5). Release baselines are the ONLY routine source of frontier-failure data (its authoring-time failure validation was parked as API-billed), so keep it in every release run — that data doubles as the check that the tier still discriminates.
• Dev/fast mode: --tier core — 19 benchmarks (Core is the headline metric)
• Full audit: --tier smoke,core,stretch,frontier — 79 benchmarks; only add smoke when you deliberately want the local sanity tier in the sweep, not for routine cloud baselines
• vision benchmarks are research-grade and excluded by default — opt in explicitly with --tier vision if you want to publish those numbers.

Override tier via the script's --tier flag (see run_eval_baseline.sh --help). If unsure, the default is tuned to produce a release-ready baseline in ~30–60 minutes.

Cost & time (default tier core,stretch,frontier = 56 benchmarks; ~1.5× the old 37-benchmark core,stretch figures — re-center after the first v0.29.0+ baseline):

❌ WRONG workflow (what happened with v0.3.22):

# DON'T do this for releases!
.Codex/skills/post-release/scripts/run_eval_baseline.sh X.X.X  # Missing --full!
# Then try to add production models later with --skip-existing
# Result: Confusion, multiple processes, incomplete baseline

If baseline times out or is interrupted:

# Resume with ALL 6 models (maintains --full semantics)
ailang eval-suite --full --langs python,ailang --parallel 5 \
  --output eval_results/baselines/X.X.X --skip-existing

The --skip-existing flag skips benchmarks that already have result files, allowing resumption of interrupted runs. But ONLY use this for recovery, not as a strategy to "add more models later".

3. Update Website Dashboard

Use the automation script:

.Codex/skills/post-release/scripts/update_dashboard.sh X.X.X

IMPORTANT: This script automatically:

• Generates Docusaurus markdown (docs/docs/benchmarks/performance.md)
• Updates JSON with history preservation (docs/static/benchmarks/latest.json)
• Does NOT overwrite historical data - merges new version into existing history
• Validates JSON structure before writing
• Clears Docusaurus cache to prevent webpack errors

Test locally (optional but recommended):

cd docs && npm start
# Visit: http://localhost:3000/ailang/docs/benchmarks/performance

Verify:

• Timeline shows vX.X.X
• Success rate matches eval results
• No errors or warnings

Commit dashboard updates:

git add docs/docs/benchmarks/performance.md docs/static/benchmarks/latest.json
git commit -m "Update benchmark dashboard for vX.X.X"
git push

3a. Snapshot Local-Rig Longitudinal (M-EVAL-OS-LONGITUDINAL)

The local Ollama rig (opencode + pi + motoko on local qwen, via the os-rotation-filler) measures whether each AILANG release moves the needle for local models.

This step is AUTOMATED since 2026-07-20: the os-rotation-filler's release-pickup step (3b) detects the std/VERSION bump on origin/dev within one cycle (~45 min), pulls, reinstalls the binary, and runs the snapshot/reset itself. Check /tmp/ailang-os-filler.log for a release pickup complete line. Only run the manual command below if the log shows snapshot/reset ... failed or the rig was down at release time:

# MANUAL FALLBACK — normally done by the filler's release pickup automatically.
# Snapshot the rig's current numbers as this release, AND reset the active-model
# accumulator so the rotation re-measures fresh against the NEXT release.
# (Retired models — anything not matching ACTIVE_PATTERN, default qwen3-6 — stay
# frozen at their last version.)
tools/os-release-snapshot.sh vX.X.X --reset

git add docs/static/benchmarks/os/history.json docs/static/benchmarks/os/latest.json
git commit -m "Snapshot local-rig OS leaderboard for vX.X.X (longitudinal)"
git push

This appends a vX.X.X entry to docs/static/benchmarks/os/history.json (deduped by version) and clears the active-model rolling files. The website's local-rig trend chart reads os/history.json; the live table still reads os/latest.json. Run without --reset if you only want to refresh a version's numbers while they're still filling (safe, idempotent).

3b. Publish the UNIFIED Dashboard (cloud + local in one leaderboard)

The whole point of the on-device roster is comparison with cloud. Step 3 above publishes the cloud baseline into the main latest.json; this step folds the local rig's rotation results for the same release into that SAME leaderboard so the on-device models (qwen/gemma) appear alongside the cloud frontier in the main tables (ELO leaderboard, gap-trend, per-model-trend). It re-runs eval-report on the cloud baseline with --merge pointed at this release's rotation dir:

# Regenerates docs/static/benchmarks/latest.json with cloud + local UNIFIED.
# Auto-skips the merge if no rotation dir exists yet for this release
# (then it's just a cloud-only refresh, identical to step 3).
tools/publish-unified-dashboard.sh vX.X.X

git add docs/static/benchmarks/latest.json
git commit -m "Unify local rotation into main leaderboard for vX.X.X"
git push

Under the hood this is: ailang eval-report eval_results/baselines/vX.X.X vX.X.X --merge eval_results/rotation/os-rolling/vX.X.X --format=json (the wrapper adds --merge only when that rotation dir exists). Do not redirect its stdout — eval-report writes latest.json itself and preserves history. Verify ratings.agent.byLang.ailang.models now lists BOTH cloud (Codex-*, opencode-or-*) and local (*-qwen*, *gemma*) models, while ratings.standard still has the cloud roster. Note the daily os-rotation-filler also runs this automatically once a release's cloud baseline exists, so this step mainly guarantees the unify happens at release.

4. Update Axiom Scorecard

Review and update the axiom scorecard:

# View current scorecard
ailang axioms

# The scorecard is at docs/static/benchmarks/axiom_scorecard.json
# Update scores if features were added/improved that affect axiom compliance

When to update scores:

• +1 → +2 if a partial implementation becomes complete
• New feature aligns with an axiom → update evidence
• Gaps were fixed → remove from gaps array
• Add to history array to track KPI over time

Update history entry:

{
  "version": "vX.X.X",
  "date": "YYYY-MM-DD",
  "score": 18,
  "maxScore": 24,
  "percentage": 75.0,
  "notes": "Added capability budgets (A9 +1)"
}

5. Extract Metrics for CHANGELOG

Generate metrics template:

.Codex/skills/post-release/scripts/extract_changelog_metrics.sh X.X.X

This outputs a formatted template with:

• Overall success rate
• Standard eval metrics (0-shot, final with repair, repair effectiveness)
• Agent eval metrics by language
• Automatic comparison with previous version (no manual work!)

Update changelog automatically:

1. Run the script to generate the template
2. Insert the "Benchmark Results (M-EVAL)" section into the active changelog file in changelogs/ (find with ls changelogs/ | grep current)
3. Place it after the feature/fix sections and before the next version
4. Note: Root CHANGELOG.md is an index file — do NOT write entries there

For CHANGELOG template format, see resources/version_notes.md.

5a. Analyze Agent Evaluation Results

v0.8.0+ (chain-based - recommended):

# Find the chain ID from the latest eval run
ailang eval-chains list

# View per-benchmark pass/fail with cost and turns
ailang eval-chains view <chain-id>

# Pass rate breakdown
ailang eval-chains stats <chain-id>

# Show failures with error details
ailang eval-chains failures <chain-id>

# Generate chain-based report
ailang eval-report --from-chain <chain-id> X.X.X --format=json

Legacy (file-based):

# Get KPIs (turns, tokens, cost by language)
.Codex/skills/eval-analyzer/scripts/agent_kpis.sh eval_results/baselines/X.X.X

Target metrics: Avg Turns ≤1.5x gap, Avg Tokens ≤2.0x gap vs Python.

For detailed agent analysis guide, see resources/version_notes.md.

5b. Tag-Based Analysis and Rotation Check (v0.14.0+)

After dashboard + changelog metrics, run the curation analysis primitives. These inform what to keep, demote, or promote for the next release — the suite is curated, not accumulated. See benchmarks/CURATION.md for the full philosophy.

# Tag coverage: which of the 12 canonical tags are thin or over-represented?
ailang eval-matrix eval_results/baselines/vX.X.X vX.X.X --by-tags

# AILANG-only wins: AILANG beats Python by ≥ 10pp — protect these from regressions
ailang eval-matrix eval_results/baselines/vX.X.X vX.X.X --ailang-wins

# Dual-mode saturation audit (RECOMMENDED for demotion decisions):
# Lists every benchmark's standard AND agent pass rate per language.
# Demote candidates require ≥95% on ALL 4 dimensions (std-AI, std-Py, agent-AI, agent-Py).
# Standard-only saturation is misleading: many benchmarks are 100/100 in standard
# but drop to 12-50% in agent mode — those are still valuable signal, KEEP IN CORE.
.Codex/skills/benchmark-manager/scripts/audit_saturation.sh vX.X.X

# Built-in saturated check (uses topline successRate only — less reliable, prefer the script above):
ailang eval-matrix eval_results/baselines/vX.X.X vX.X.X --show-saturated

# Optional: compare this release's tag deltas against the previous baseline
ailang eval-report eval_results/baselines/vX.X.X vX.X.X --format=json

Record three things in the release notes (or the design doc retro):

• Demote candidates — saturated benchmarks that should move to stretch or be retired in the next sprint.
• Keep as value evidence — AILANG-only wins that prove the language's ROI.
• Thin tags — taxonomy gaps (<3 benchmarks in a tag) to target with new benchmarks.

This step is cheap (seconds), has no external dependencies, and produces the input for the next release's benchmark-manager / eval-gap-finder work.

6. Move Design Docs to Implemented

Step 1: Check for issues (duplicates, misplaced docs):

.Codex/skills/post-release/scripts/cleanup_design_docs.sh X.X.X --check-only

Step 2: Preview all changes:

.Codex/skills/post-release/scripts/cleanup_design_docs.sh X.X.X --dry-run

Step 3: Check any flagged docs:

• [DUPLICATE] - These will be deleted (already in implemented/)
• [MISPLACED] - These will be relocated to correct version folder
• [NEEDS REVIEW] - Update **Status**: to Implemented if done, or leave for next version

Step 4: Execute the cleanup:

.Codex/skills/post-release/scripts/cleanup_design_docs.sh X.X.X

Step 5: Commit the changes:

git add design_docs/
git commit -m "docs: cleanup design docs for vX_Y_Z"

The script automatically:

• Deletes duplicates (same file already in implemented/)
• Relocates misplaced docs (Target: version doesn't match folder)
• Moves docs with "Status: Implemented" to implemented/
• Flags remaining docs for manual review

7. Update Public Documentation

• Update prompts/ with latest AILANG syntax (ailang prompt)
• Update prompts/devtools/ with latest toolchain docs (ailang devtools-prompt)
  • New CLI commands or flags should be added to the devtools prompt
  • Verify with: ailang devtools-prompt | grep "new-command"
• Update website docs (docs/) with latest features
• Remove outdated examples or references
• Add new examples to website
• Update docs/guides/evaluation/ if significant benchmark improvements
• Update docs/LIMITATIONS.md:
  • Remove limitations that were fixed in this release
  • Add new known limitations discovered during development/testing
  • Update workarounds if they changed
  • Update version numbers in "Since" and "Fixed in" fields
  • Test examples: Verify that limitations listed still exist and workarounds still work

    # Test examples from LIMITATIONS.md
    # Example: Test Y-combinator still fails (should fail)
    echo 'let Y = \f. (\x. f(x(x)))(\x. f(x(x))) in Y' | ailang repl
    
    # Example: Test named recursion works (should succeed)
    ailang run examples/factorial.ail
    
    # Example: Test polymorphic operator limitation (should panic with floats)
    # Create test file and verify behavior matches documentation

  • Commit changes: git add docs/LIMITATIONS.md && git commit -m "Update LIMITATIONS.md for vX.X.X"

8. Run Documentation Sync Check

Run docs-sync to verify website accuracy:

# Check version constants are correct
.Codex/skills/docs-sync/scripts/check_versions.sh

# Audit design docs vs website claims
.Codex/skills/docs-sync/scripts/audit_design_docs.sh

# Generate full sync report
.Codex/skills/docs-sync/scripts/generate_report.sh

What docs-sync checks:

• Version constants in docs/src/constants/version.js match git tag
• Teaching prompt references point to latest version
• Architecture pages have PLANNED banners for unimplemented features
• Design docs status (planned vs implemented) matches website claims
• Examples referenced in website actually work

If issues found:

1. Update version.js if stale
2. Add PLANNED banners to theoretical feature pages
3. Move implemented features from roadmap to current sections
4. Fix broken example references

Commit docs-sync fixes:

git add docs/
git commit -m "docs: sync website with vX.X.X implementation"

See docs-sync skill for full documentation.

9. Verify μRAG Corpus Reindex (REQUIRED)

release-manager runs make brain-index-syntax-reset immediately after the tag pushes. This step verifies that the reset actually populated the brain with chunks tagged with the new release version — protects against silent indexer failures that would leave Codex pulling stale snippets.

Spot-check ≥5 chunks reference the active version:

EXPECTED_VERSION="$(ailang prompt --version-active)"
ailang cache search --namespace ailang-syntax --limit 5 "string" \
  | grep -c "version:${EXPECTED_VERSION}" \
  || { echo "FAIL: μRAG corpus does not reference $EXPECTED_VERSION"; exit 1; }

Quick stat check:

ailang cache stats | grep -E "ailang-(syntax|builtins|examples)"
# Expect: ailang-syntax >= 50, ailang-builtins >= 250, ailang-examples >= 100

Append a one-line audit to release notes (or eval_results/baselines/<version>/notes.md):

μRAG corpus reindex verified: <ailang-syntax count>, <ailang-builtins count>, <ailang-examples count> chunks @ <version>.

If verification fails:

• Re-run make brain-index-syntax-reset from the project root.
• Check ailang prompt --version-active returns the new release tag.
• If it returns nothing, the prompt directory wasn't published in the release — escalate to release-manager rather than masking the issue.

Resources

Post-Release Checklist

See resources/post_release_checklist.md for complete step-by-step checklist.

Prerequisites

• Release vX.X.X completed successfully
• Git tag vX.X.X exists
• GitHub release published with all binaries
• ailang binary installed (for eval baseline)
• Node.js/npm installed (for dashboard, optional)

Agent CLIs + API keys (for the agent baseline on this rig)

The agent baseline (agent_suite) routes each model to a CLI harness. To run all of them on the Studio rig, these must be installed and authenticated:

API keys live in ~/.config/ailang/secrets.env (sourced from ~/.zshenv). Pull the cloud-managed ones with ~/.config/ailang/pull-secrets.sh (Anthropic/OpenAI/Google from Secret Manager). OpenRouter is the manual key. Gemini uses Vertex ADC, not the API key.

Verify everything resolves before a release run:

for c in opencode Codex codex pi motoko; do which $c >/dev/null && echo "✓ $c" || echo "✗ $c"; done
# Note: NO gemini CLI — the @google/gemini-cli is deprecated/unused. Google agent mode
# goes through managed_agents (Vertex API via `gcloud auth application-default login`).
ailang eval-suite --agent --models agent_suite --benchmarks fizzbuzz --langs ailang --dry-run  # all should route, none "<none>"

Common Issues

Eval Baseline Times Out

Solution: Use --skip-existing flag to resume:

bin/ailang eval-suite --full --skip-existing --output eval_results/baselines/vX.X.X

Dashboard Shows "null" for Aggregates

Cause: Wrong JSON file (performance matrix vs dashboard JSON) Solution: Use update_dashboard.sh script, not manual file copying

Webpack/Cache Errors in Docusaurus

Cause: Stale build cache Solution: Run cd docs && npm run clear && rm -rf .docusaurus build

Dashboard Shows Old Version

Cause: Didn't run update_dashboard.sh with correct version Solution: Re-run update_dashboard.sh X.X.X with correct version

Progressive Disclosure

This skill loads information progressively:

1. Always loaded: This SKILL.md file (YAML frontmatter + workflow overview)
2. Execute as needed: Scripts in scripts/ directory (automation)
3. Load on demand: resources/post_release_checklist.md (detailed checklist)

Scripts execute without loading into context window, saving tokens while providing powerful automation.

Version Notes

For historical improvements and lessons learned, see resources/version_notes.md.

Key points:

• All scripts accept version with or without 'v' prefix
• Use --validate flag to check configuration before running
• Agent eval scopes benchmarks by tier (--tier core,stretch,frontier default) — benchmarks are resolved from benchmarks/*.yml by tier, not a hardcoded list
• Tag-based curation analysis runs via ailang eval-matrix --by-tags/--show-saturated/ --ailang-wins in Step 5b

Notes

• This skill follows Anthropic's Agent Skills specification (Oct 2025)
• Scripts handle 100% of automation (eval baseline, dashboard, metrics extraction, design docs)
• Can be run hours or even days after release
• Dashboard JSON preserves history - never overwrites historical data
• Always use --full flag for release baselines (all production models)
• Design docs cleanup now handles duplicates, misplaced docs, and status-based moves
  • --check-only to report issues without changes
  • --dry-run to preview all actions
  • --force to move all regardless of status