Port PwC evals to Hub leaderboards

The Hub's evaluation results feature builds a leaderboard on each official benchmark dataset from .eval_results/*.yaml files in model repos, including files in open PRs. This skill finds PwC results that belong on those leaderboards, verifies each one against its source, and opens one PR per model repo that a maintainer can review and merge in a minute.

A PR is only worth opening if the reviewer can see the result on the leaderboard and find the number at the cited source. Most of the rules below exist because one of those two failed.

Scripts

All scripts live in .agents/skills/port-evals-to-hub/scripts/ and run from the repository root. Write scripts guard their side effects with --apply; without it they only print.

Run find_candidates.py with uv run --env-file backend/keys.env python …; the Hub-only scripts need just uv run python …. The registry defaults to reports/hf-eval-results-prs.json (local, not committed).

Opening, editing, or closing PRs posts publicly on other people's repos. The auto-mode classifier may block --apply; if it does, show the user the exact command and let them run it, then verify the result.

1. Find candidates

1. Check the official benchmark list at https://huggingface.co/api/datasets?filter=benchmark:official&limit=200. If a benchmark is missing from MAPPING in find_candidates.py, read its eval.yaml, find the matching PwC dataset and metric, and add a row. Make sure the PwC metric is the one the Hub task measures (for example MMMU-Pro's mmmu_pro_vision task is the vision split, not the overall score).
2. Run find_candidates.py and work from the candidate rows. When the user gives a model, paper, or release instead, collect candidates from its sources directly; the PwC rows are a lead, not the evidence.
3. When the user works in batches ("rows 46–65"), keep row numbers stable within a CSV, and report every skipped row with its reason.

Repo eligibility:

• Organizations only. Results in personal-account repos never appear on leaderboards. namespace_kind tells them apart. The user prefers major labs (MAJOR), but any other Hub organization is fine.
• The exact evaluated checkpoint. The repo must hold the weights that produced the number. Skip it when the card says the reported results came from another checkpoint (NorthCUA: scores from SFT checkpoints, repo holds a later post-training step), and don't attach a sibling's or base model's score.
• Not already declared. Skip benchmarks the repo declares on main or in any open PR, including PRs from other people.

2. Verify every value at its primary source

Read the model card, paper, README, or blog yourself. Never rely only on the PwC row: PwC values have been wrong (tool-integrated scores stored for no-tool boards, fractions for percent boards, a typo'd digit). When PwC is wrong, fix PwC with the add-evals or verify-evals conventions as well.

Skip a result, or the whole repo, when:

• The score is third-party. For example a card's † numbers from Artificial Analysis, or scores for the model taken from a benchmark's own paper. The source must be the model's authors.
• The protocol differs from the board. Examples: with tools or code execution on a no-tools board (AIME tool-integrated reasoning), inference tricks such as zoom, cropping, verifiers, or agentic refinement on a direct-inference board (ScreenSpot-Pro), a subset of the benchmark (Terminal-Bench run on 88 of 89 tasks), or a different metric or scale. Use the no-tools number when the source gives both, and say so in notes.
• The benchmark is contaminated or may be. For example a tech report admitting overlap, or training data that visibly contains test items. Raise possible contamination with the user rather than deciding silently.
• The board mixes metrics or protocols. For now that means Video-MME-v2: skip it.
• The checkpoint is unfinished ("still training", "under revision").

Use the number exactly as the source prints it, in the board's units. DeepSWE uses percent, so a PwC 0.627 becomes 62.7. When the source shows the same score at two precisions, the model card's precise value wins (44.94 over the blog's 44.9).

3. Make the number findable

The reviewer must be able to find each number at the cited source. open_eval_prs.py warns when the value string is not in the source text. Then write origin so the reviewer can find it anyway:

• name the table, row, and column (Table 6 … (NorthCUA-Pro row, SSP column)), or quote the sentence (the model card ("On Terminal-Bench 2.0 it scores 9.36%"));
• if it appears only in an image or chart, say which figure and bar;
• if the source hides it behind a control (DeepSWE's Models picker, a chart's benchmark toggle), say how to reveal it.

Cite the most direct surface: model card over paper over blog. Use the source.name the Hub shows, such as Model Card, Paper (Table 4), or TMax v1.1 blog.

4. Check top-50 visibility

The leaderboard UI shows ranks 1–50 of the selected view only. A view is a "Parameters Size" tier (max_params: 1B, 3B, 6B, 12B, 32B, 128B, 500B) plus the "Base only" toggle. That toggle is on by default and hides every model whose card declares base_model, which covers most fine-tunes (GUI agents, RL'd terminal agents). predicted_view returns the broadest view where a new result would rank in the top 50, and the PR link opens that view (leaderboard_max_params=…&leaderboard_base_model=false). Drop a result that is visible nowhere.

Gotchas:

• A model without safetensors num_parameters drops out of every size-filtered view. A wrong count misplaces it: tmax-27b-v1.1's index claims 2.6M parameters, so it lands in the ≤1B view. Mention metadata bugs like this to the user.
• A card that lists itself as its own base_model is hidden in the default view.
• A PR that edits an existing .eval_results file previews main's version until merged, so its result is not on the board yet. That doesn't make it invisible: never close such a PR in a sweep.
• task_id must be a task in the benchmark's eval.yaml (terminalbench_2, swe_bench_%_resolved), not the split name. A wrong id returns an empty board, which looks like rank 1. open_eval_prs.py refuses unknown ids.

5. Open the PRs

Write a spec (see the open_eval_prs.py docstring), with one entry per repo and every eligible benchmark in it. Use notes for the protocol a reviewer needs in order to judge comparability: harness and version, number of tasks and trials, sandbox, no tools, direct inference on the full input. Then:

uv run python .agents/skills/port-evals-to-hub/scripts/open_eval_prs.py /path/spec.json            # dry run
uv run python .agents/skills/port-evals-to-hub/scripts/open_eval_prs.py /path/spec.json --apply

The PR format follows meta-models/Muse-Glimmer-30B discussion 8:

• Title: "Add community evaluation results for <exact benchmark names>".
• Body: one bold link per benchmark to the leaderboard view with the result highlighted (eval_result=<repo>), followed by "These results were extracted from <origin>." and the feature link and automated-PR note.
• Benchmark names are written the way their authors write them: SWE-Bench Pro, olmOCR-Bench, Terminal-Bench 2.0. Add new names to NAMES in hub_leaderboard.py when the eval.yaml name differs.

To change a file in an existing PR (for example pointing .eval_results/terminal_bench_2_1.yaml at the right dataset), commit to refs/pr/<num> with HfApi.create_commit(revision=…). Don't open a second PR.

6. Verify and sweep

Right after opening, confirm each PR's results with visible_view (live rank, which can sit a few places below the prediction), and report a table with PR link, benchmark, value, live rank, and view. Ranks drift as other PRs land, so after each batch run:

uv run python .agents/skills/port-evals-to-hub/scripts/sweep_eval_prs.py            # dry run
uv run python .agents/skills/port-evals-to-hub/scripts/sweep_eval_prs.py --apply

Review the dry run before applying. A sweep may remove a result or close a PR, but only one whose previewable results all fell out of every view. Tell the user which ones it touched.

Report

For each batch, report:

• the PRs opened, with benchmark, value, rank, and view;
• every skipped row with its reason (third-party, protocol, checkpoint, contamination, personal account, already declared, outside the top 50);
• any PwC corrections made;
• metadata problems seen on model repos;
• where the batch stopped, so the next batch starts at the right row.