Release Notes Skill

Use this skill when the user wants to cut a release of agentic-coding-harness-benchmarks. It decides the correct next version, gathers every change since the previous release, writes structured release notes in the project format, opens a PR (this repo never commits to main directly), and tags the merge commit after the PR is approved and merged.

This repo is a benchmark harness plus its published results, docs and slides. It is not a deployed service, so there is no Docker Compose / Helm / Terraform upgrade path and no live smoke test. What a release captures instead is: new models benchmarked, new datasets, methodology changes, new harness functionality, and the results / frontier / doc updates that follow.

Versioning scheme (read this first)

Releases follow Semantic Versioning 2.0.0: MAJOR.MINOR.PATCH, bare, with no v prefix (e.g. 0.2.0, not v0.2.0). The one existing non-version tag, media-assets, hosts video/asset release attachments and is not a release: ignore it when finding the latest version.

What each bump means in this repo

The "public API" of a benchmark repo is its dataset, its measured models, and its measurement methodology. Map the change set to a bump with these rules, in priority order (if a change set matches more than one, take the highest):

Because a methodology change makes previously published numbers no longer directly comparable, it is the clearest MAJOR signal: when scores from before and after the change cannot sit in the same table, bump MAJOR. New datasets and new models are additive, so they are MINOR even though they add a lot of data.

Choosing the first version and the 1.0.0 line

There are no version tags yet, so the first release sets the convention:

• In 0.y.z the methodology and results are still settling and nothing is promised to stay comparable across releases. This is the honest place to start: use 0.1.0 for the first release unless the user says the harness and its published frontier are already something people depend on.
• Go to 1.0.0 when you are ready to stand behind a stable methodology and results contract, and to bump MAJOR whenever a methodology change breaks it. 1.0.0 is a commitment, not a maturity trophy earned after 0.9.0 -- there is no rule that the first release must be 0.9.0, and you never need to march through 0.9.0 before 1.0.0.

Always confirm the computed version with the user before writing anything. State the bump you chose and the one-line reason (e.g. "MINOR: adds the minimax-m3 model to the omp/swe3 frontier"), and let them override.

Input

• An optional version, {major}.{minor}.{patch} (e.g. 0.2.0), no v prefix. If the user gives a v-prefixed version, strip the v and confirm.
• If no version is given, compute the recommended bump from the change set (see above) and confirm it with the user.

Output

• docs/release-notes/{version}.md -- the release notes file (e.g. docs/release-notes/0.2.0.md).
• On the first release only: docs/release-notes/README.md -- a small index of releases, and one new row in the root README.md "Documentation map" table pointing at it.
• A git tag {version} on the merge commit, created after the release-notes PR is merged to main.

Workflow

Step 0: Verification gate

A release must not ship code or results that fail the repo's own checks. Before writing notes, confirm the checks pass for every uv project touched since the base version (benchmarks/ and/or self-hosted/vllm/). CI gates on the unittest suites, so those are the hard block.

1. Ask the user, using AskUserQuestion, whether the checks have already been run and passed. Offer:
   • "Yes, they passed" (proceed to Step 1)
   • "No, run them now" (Recommended -- run them, see below)
   • "Skip them" (proceed only after the warning below)
2. If the user asks you to run them, run the AGENTS.md check line from inside each touched uv project. At minimum the test suites must pass:

   # From benchmarks/ (and/or self-hosted/vllm/) -- whichever owns the changed files
   uv run ruff check --fix . && uv run ruff format .
   uv run python -m unittest discover -s tests

   • If any test FAILS, STOP. Do not cut the release. Report which tests failed and help investigate. A release must not be cut with a failing suite.
   • Run the fuller line (bandit, mypy) too when the change is security- or type-sensitive; treat a new failure as a blocker.
3. If the user chooses to skip, warn once that the release is being cut without verification, then proceed only if they confirm.

Only after this gate is resolved continue to Step 1.

Step 1: Determine the new version tag

1. Compute the recommended bump from the change set using the Versioning scheme rules, or take the version the user gave.
2. Normalize to bare semver (strip any leading v).
3. Verify the tag does not already exist: git tag -l {version}.
4. If it exists, ask the user whether to move it or pick a different version.
5. Confirm the version and the one-line bump reason with the user before proceeding.

Step 2: Determine the base version (ask the user to confirm)

Release notes are incremental from the previous release.

1. List existing release notes and version tags (excluding the media-assets asset tag):

   ls docs/release-notes/*.md 2>/dev/null
   git tag --sort=-v:refname | grep -E '^v?[0-9]+\.[0-9]+\.[0-9]+$'

2. First release (no version tags): there is no base. Diff from the repository's first commit and summarize the harness as it stands, rather than an incremental changelog:

   FIRST_COMMIT=$(git rev-list --max-parents=0 HEAD | tail -1)

   Use $FIRST_COMMIT wherever the steps below say {base_tag}.
3. Otherwise: take the most recent version tag as the base and confirm it with the user using AskUserQuestion (present it as the recommended option, with the 2-3 previous tags as alternatives). The user may want to span multiple versions.

Step 3: Gather all changes between base and HEAD

Run these in parallel to gather the change data:

# All commits (including merges) between base and HEAD
git log {base_tag}..HEAD --oneline

# Non-merge commits (for detailed change analysis)
git log {base_tag}..HEAD --oneline --no-merges

# Merge commits (to extract PR numbers)
git log {base_tag}..HEAD --oneline --grep="Merge pull request"

# Contributors -- direct authors of commits on main
# WARNING: this misses co-authors of squash-merged PRs (Step 4 #9 explains).
git log {base_tag}..HEAD --format="%aN" | sort | uniq -c | sort -rn

# Contributors -- per-PR commit authors (catches squash-merge co-authors)
# Squash merges collapse a branch's commits into one commit on main authored by
# the merger, so `git log` above will not show the real code authors.
# `gh pr view --json commits` returns the original branch commits with authors intact.
for pr in $(git log {base_tag}..HEAD --oneline --grep="Merge pull request\|(#[0-9]\+)$" | grep -oE "#[0-9]+" | tr -d '#' | sort -u); do
  gh pr view $pr --json number,author,commits \
    --jq '"PR #\(.number) | opener: \(.author.login) | commit_authors: \([.commits[].authors[].name] | unique | join(", "))"' 2>/dev/null
done

# Config-schema changes (this repo's analog of an env-var diff): the RunnerConfig
# model, the example config, and the pricing table. Any change here is a config
# change users may need to act on; a backward-incompatible one is a MAJOR signal.
git diff {base_tag}..HEAD --stat -- \
  benchmarks/scripts/runner_config.py \
  benchmarks/config/runner.example.yaml \
  self-hosted/vllm/pricing.json

# New models benchmarked (per-model serving guides and the frontier data)
git diff {base_tag}..HEAD --stat -- self-hosted/vllm/models/ docs/metrics/

# New or changed skills
git diff {base_tag}..HEAD --stat -- .claude/skills/

# Dataset changes (new sweN task sets)
git diff {base_tag}..HEAD --stat -- benchmarks/dataset/

# Closed issues since the base tag was cut (floor by the base tag's commit date)
BASE_TAG_DATE=$(git log -1 --format=%cI {base_tag})
gh issue list --state closed --limit 200 --json number,title,closedAt,labels \
  --jq ".[] | select(.closedAt >= \"$BASE_TAG_DATE\") | \"\(.number) | \(.title) | \(.closedAt)\""

# Closed issues referenced by merged PRs (most reliable mapping)
for pr in $(git log {base_tag}..HEAD --oneline --grep="Merge pull request" | grep -oE "#[0-9]+" | tr -d '#' | sort -u); do
  gh pr view $pr --json number,title,closingIssuesReferences \
    --jq '"\(.number) | \(.title) | closes: \(.closingIssuesReferences | map("#\(.number)") | join(","))"' 2>/dev/null
done

Step 4: Categorize changes

Analyze the commits and PRs and sort them. The bump you chose in Step 1 should be justified by what lands in the first few categories.

1. New models benchmarked (MINOR): a model added to the frontier -- new models.json / docs/metrics/ frontier entries, a new per-model guide under self-hosted/vllm/models/, new charts and results docs. Name the model, the harness/skill and dataset it was run on, and the headline number.
2. New datasets (MINOR): a new sweN task set or a new set of tasks under benchmarks/dataset/. Say how many tasks and how it differs from existing datasets (never merge scores across datasets).
3. Methodology changes (MAJOR): anything that changes what a number means -- the judge, scoring, cost-per-task derivation, frontier basis, or a backward-incompatible config-schema change. Spell out why old and new numbers are no longer comparable.
4. New functionality (MAJOR): a new benchmark type, hosting path, or major harness capability.
5. Skills (MINOR for a new skill; PATCH for a fix): new or changed entries under .claude/skills/.
6. Results and frontier updates (PATCH when just regenerated for existing models): recomputed charts, frontier JSON, decks.
7. Config changes: new or changed RunnerConfig knobs, runner.example.yaml, pricing.json (from the diff above).
8. Bug fixes (PATCH): fix:-prefixed commits or PRs labeled bug.
9. Documentation / slides (PATCH): docs: commits, changes under docs/, deck HTML/PDF regen.
10. Dependency updates: pyproject.toml / lockfile bumps in either uv project.
11. Closed issues: build from the closingIssuesReferences of every merged PR (GitHub auto-closes Closes #N / Fixes #N), supplemented by manually-closed issues whose closedAt falls between the base-tag commit date and HEAD. De-duplicate by issue number.

12. Contributors: build the union of TWO sources, because neither alone is complete:

    • Direct authors on main: git log {base_tag}..HEAD --format="%aN".
    • Per-PR commit authors: for every merged PR, gh pr view <num> --json commits --jq '[.commits[].authors[].name] | unique', unioned. This catches co-authors of squash-merged PRs whose branch commits collapse into one commit authored by the merger. Without it, every contributor on a squash-merged branch except the merger is silently dropped.

    Resolve each unique name to a GitHub login from a real PR: .author.login if they opened one, or .commits[].authors[].login if they were a co-author only. Verify uncertain guesses with gh api users/<candidate> (404 means wrong). Never synthesize a username from a display name -- "Amit Arora" is aarora79, not amitarora.

Step 5: Write the release notes

Write docs/release-notes/{version}.md following this structure. Keep every prose section to the writing skill (.claude/skills/writing/SKILL.md): plain words, no em-dashes, no emojis, and every number cited to the file it came from.

# Release {version} - {short title summarizing the headline change}

**{Month} {Year}**

Version bump: **{MAJOR|MINOR|PATCH}** -- {one-line reason, e.g. "adds the minimax-m3 model to the omp/swe3 frontier"}.

---

## Upgrading from {base_version}

This repo is cloned and run, not deployed, so upgrading is a pull plus a dependency sync.

```bash
git pull origin main
git checkout {version}

# Sync the uv project(s) you run. If dependencies changed, sync both:
cd benchmarks && uv sync && cd ..
cd self-hosted/vllm && uv sync && cd ../..

Methodology changes

{For a MAJOR release, state exactly what changed in how a number is produced and why results from {base_version} and {version} must not be compared or merged in the same table. For MINOR / PATCH, write: "None. Numbers remain comparable with {base_version}."}

Config changes

{If no config changed, write: "No configuration changes in this release."}

New models

{Model name} on {harness} / {dataset}

{What it scored and where it sits on the frontier, cited to the frontier JSON / results doc. Link the per-model serving guide.}

{Repeat per model. Omit the whole section if no model was added.}

New datasets

• {dataset name} -- {N tasks, how it was sourced, how it differs from existing sets, and the reminder that its scores never merge into another dataset's table}.

{Omit if no dataset was added.}

New functionality

{Feature name}

{What it does and why it matters. Link the PR.}

PR #{number}

{Omit if none.}

What's changed

{Group remaining changes by category. Only include categories that have changes.}

Skills

• {change} (#{pr})

Results and frontier

• {change} (#{pr})

Documentation and slides

• {change} (#{pr})

Bug fixes

• {fix} (#{pr})

{Omit if none.}

Closed issues

{Sorted by issue number descending. If none: "No issues were closed in this release window."}

Pull requests included

{All merged PRs between base and HEAD, sorted by PR number descending.}

Dependency updates

{Omit if none.}

Contributors

Thank you to everyone who contributed to this release:

• {Full Name} (@{login})

{Union of direct authors and per-PR commit authors (Step 4 #12). Every login resolved from a real PR. Sorted by commit count descending.}

Full Changelog: {base_version}...{version}

For the **first release** there is no `{base_version}` to diff against: drop the "Upgrading from" section's diff framing, describe the harness and its published frontier as it stands at this version, and omit the "Full Changelog" compare link (or point it at the first commit).

### Step 6: Present the draft for review

After writing the file:

1. Tell the user the file is at `docs/release-notes/{version}.md`.
2. Summarize: the chosen bump and why, new models, new datasets, methodology changes, PR count, bug-fix count, closed-issue count, contributor count.
3. Ask the user to review and confirm, or request changes.

### Step 7: Security gate, PR, and tag

This repo **never commits or merges directly to `main`.** Cut the release through a PR, and tag only after it merges.

1. **Run the `security-check` skill** over the pending diff (mandatory gate before any commit or PR). Resolve every blocker before continuing; do not commit while the verdict is NEEDS REVISION.

2. **On the first release only**, also create `docs/release-notes/README.md` as a short index (a table of `| Version | Date | Highlights |` rows, newest first) and add one row to the root `README.md` "Documentation map" table pointing at it. On later releases, prepend a row to that index instead.

3. **Commit on the feature branch** (you should already be on one; if on `main`, create `release/{version}` first):
   ```bash
   git add docs/release-notes/{version}.md
   # first release also:
   git add docs/release-notes/README.md README.md
   git commit -m "docs: add {version} release notes"
   git push -u origin HEAD

4. Open a PR with a professional description that summarizes the release (no assistant attribution anywhere):

   gh pr create --title "Release {version}" --body-file <(printf '...') --base main

5. After the PR is approved and merged to main, tag the merge commit and push the tag:

   git checkout main && git pull origin main
   git tag {version}          # bare semver, no v prefix
   git push origin {version}

6. Optionally publish a GitHub Release from the tag using the notes file:

   gh release create {version} --title "Release {version}" --notes-file docs/release-notes/{version}.md

7. Verify:

   git tag -l {version} --format="%(refname:short) -> %(objectname:short)"

   Tell the user the notes are merged and the tag is created and pushed.

Important rules

• Pick the bump by the repo's rules, not by feel: new dataset or new model is a MINOR; a methodology change or completely new functionality is a MAJOR; everything else is a PATCH. A methodology change is the clearest MAJOR because it breaks comparability of published numbers. Always confirm the computed version with the user.
• Bare semver, no v prefix. Ignore the media-assets tag when finding the latest version.
• Never commit or merge to main directly. Cut every release through a feature branch and a PR; tag the merge commit only after it lands.
• Always run the security-check gate before committing and before opening the PR.
• Run the Step 0 verification gate first. If any test suite fails, STOP and do not cut the release. Only the user's explicit skip may bypass it, warned once.
• Never skip the base-version confirmation in Step 2; the user may span multiple versions.
• Never include emojis in the release notes or commits, and run the writing skill revision pass over the prose before committing.
• Never name the coding assistant anywhere -- not in the notes, commit messages, PR title or body.
• Check every number against its source (a performance-summary.json, a frontier JSON, a results doc) and cite it, per AGENTS.md.
• Always credit squash-merge co-authors via gh pr view <num> --json commits, and never synthesize a GitHub username from a display name ("Amit Arora" is aarora79).

Example usage

User: /release-notes
Assistant: (computes the bump from the change set, e.g. "MINOR -- adds the
minimax-m3 model to the omp/swe3 frontier", confirms 0.2.0 with the user,
runs the verification gate, gathers changes since the base tag, writes
docs/release-notes/0.2.0.md, runs security-check, opens a PR, and tags after merge)