Process & Upload a Benchmarked Method to TabArena

This skill drives the maintainer workflow that turns a benchmark run's already-present raw results.pkl files into cached, hosted (r2), and registered TabArena artifacts. There is no download and no auto-generation — the raw data is assumed to be on disk already (e.g. you unzipped a submission, or it's a fresh run's output/<run>/data dir).

The authoritative prose for this flow is AGENTS.md → "Processing & uploading method artifacts (maintainers)" and the two scripts' module docstrings (scripts/run_process_method.py, scripts/run_upload_results.py). This skill operationalizes it: Claude executes the flow itself by default, and makes the small deterministic code edits along the way.

What this skill delivers

1. A plan, stated first: which methods, which run dir, which suite, and that the flow ends in a real r2 upload — tell the maintainer before executing (no need to wait for approval unless they asked to review something first).
2. Claude-run execution, in order per method:
   • inspect — read the raw data, print inferred fields + a metadata diff.
   • process — build + cache metadata.yaml + processed/ + results/ locally.
   • upload (dry-run, then --no-dry-run) — push the cached artifacts to r2.
3. Code edits:
   • The model's info.py MethodMetadata — fill in the manual upload fields, fix any raw-data mismatches the inspect diff surfaces.
   • The arena collection registration in methods.py so the method appears in the benchmark.
   • The run's entry in packages/tabflow_slurm/BENCHMARK_LOG.md, if the launch stage never wrote one.
4. Disk cleanup once r2 is verified: the run's raw results, the cache copies made for the upload, its setup and log directories, the pinned worktrees and venv, and the SkyPilot bucket data (Step 8).
5. A fallback command sheet only for steps the environment can't run.

Execution requirements (check before starting; hand off the affected step if missing):

• a venv with tabarena[benchmark] installed (look under ~/.venvs/tabarena_*; run scripts with its python from the repo root),
• R2 credentials in the environment for the real upload (printenv | grep -o '^R2_[A-Z_]*' should show R2_ACCOUNT_ID / R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY; never pass them as flags).

Processing and uploads are long-running: run them in the background (per-method), watch with a monitor (per-method DONE/FAILED lines + a stall watchdog), and for multi-method batches pipeline the uploads — start each method's dry-run + real upload as soon as its processing finishes.

Step 0: Gather inputs

Results from a contributor's own run (a submission hosted instead of a maintainer re-run): verify the files before anything is processed, following references/verify_submission.md (safe download and unpickling, a scan of every results.pkl with scripts/scan_submission.py, split and label hashes against our own run, hyperparameters against the search space, the claimed Elo reproduced with compare, and the impact of any problem the review finds). verified=True needs the maintainer's sign-off on that report.

Parse $ARGUMENTS. Collect (ask only for what's missing or ambiguous):

Step 1: Locate the method's MethodMetadata

Read packages/tabarena/src/tabarena/models/<model>/info.py (or systems/<system>/info.py) and note the exact variable name of its MethodMetadata (e.g. chimeraboost_method_metadata, tabfm_plus_method_metadata). Both scripts take it as a dotted reference:

tabarena.models.<model>.info:<varname>
tabarena.systems.<system>.info:<varname>

• info.py exists (the normal case — the model was added via the add-model skill): it already carries the raw-inferable fields (ag_key, config_default, can_hpo, is_bag, compute, method_type). validation_protocol is inferable too but was not authored before the run: a new run records it, and --process fails until info.py declares the inferred key. Claude fills it together with the manual upload fields in Step 3.
• No info.py (a raw external submission): the method isn't integrated yet. Run inspect (Step 2) to get the copy-paste MethodMetadata.<type>(...) snippet, then author models/<model>/info.py from it (use the add-model skill if the model also needs a wrapper, or add-system if it is a whole pipeline). Processing requires an explicit committed MethodMetadata — it refuses to guess.
• The method is a system: the inspector cannot tell a system from any other baseline, since the runner records both identically. Use MethodMetadata.system(...) rather than .baseline(...), and set tags — see the add-system skill. Getting this wrong types the method as a model on the leaderboard and puts it in the models-only entrant pool, where it does not belong.

Step 2: Inspect the raw data (Claude runs; confirms inference)

<venv>/bin/python scripts/run_process_method.py <run_data_dir>

This prints the fields inferred from the raw data (method_type, ag_key, compute, config_default, can_hpo, is_bag, validation_protocol with a fold histogram, task/problem-type/metric coverage), a suggested MethodMetadata snippet, and — when an info.py metadata is passed — an inferred-vs-provided diff. Use it to confirm info.py matches the raw data before processing. Two gotchas it catches:

• config_default is compared post-rename: configs are renamed to the method's prefix during processing, so a config_default authored with the raw prefix won't match. The snippet/diff shows the post-rename value — use that. The most common real mismatch is the infix, not the prefix: the TabArena-v0.1 bundle names the first config <Method>_c1_default_BAG_L1 (the _default is the preprocessing pipeline name, appended for HPO and default-only models alike), while info.py files authored before a run, and the BeyondArena bundle, use <Method>_c1_BAG_L1. An info.py written by add-model before the run therefore usually needs this one fix. A single-config method (can_hpo=False) can drop the field instead: an undeclared config_default is not checked (the diff marks it not declared) and --process records the lone config in the cached metadata.yaml.
• Only error rows gate processing: the method row differs whenever the raw ag_name carries the TA- prefix (warn-only) and model_key is never checked (shown for context). A NO on either is expected and needs no edit; a NO on config_default, ag_key, compute, can_hpo, is_bag or method_type does.
• method != suite: process fails if they're equal (suite defaults to method when unset).
• validation_protocol must be declared: a run made after the protocol record existed infers a key (8x1 for TabArena, system for a system, the long BeyondArena key); --process fails while info.py leaves it None or declares another value. Legacy raw artifacts infer None and pass. Add --expect-validation-protocol tabarena (or beyondarena) to also assert that the run used the arena's official protocol; a holdout:/outer/custom key there means the run cannot be uploaded as an official result. KNN's 8x1 requested; 1 child via use_child_oof histogram line is expected.

BeyondArena runs: the task directories of a BeyondArena run are named <dataset>-<hash>, not OpenML task ids, so EndToEnd.from_path_raw cannot infer the task metadata from them and filters every result out (No results to process). Add --task-metadata BeyondArena to the run_process_method.py commands so processing uses the committed collection (a driver passes task_metadata=TaskMetadataCollection.from_source("BeyondArena") to process_method / process_raw). Register the method in contexts/beyondarena/methods.py through the model's ModelDescriptor (<x>_descriptor.method_metadata(...) with the file's _common_fm_kwargs / _common_bag_kwargs), adding a descriptor to the model's info.py when it only has MethodMetadata.config(...) entries; the BeyondArena bundle names configs <Method>_c1_BAG_L1 (no _default infix) and method follows the raw ag_name (TA-... for most wrappers, Causilo for Causilo).

Multi-method run dirs: if the run's data/ holds several methods' config dirs side by side, run_process_method.py can't split them — write a small gitignored driver in tmp_scripts/ that discovers the results.pkl paths once, splits them by top-level config-dir prefix, and calls _infer_from_raw / verify_method_metadata / process_raw(file_paths=...) per method (see the multi-method-run-upload pattern; a --c1-only pass over just the _c1_ dirs makes config_default inferable even for HPO methods with hundreds of configs).

Step 3: Edit info.py (Claude does now)

Fill in the manual fields the upload requires. These are not inferable from raw data, so the maintainer normally hand-edits them — Claude does it now. Read the file first, then Edit:

Leave the other raw-inferable fields (ag_key, config_default, can_hpo, is_bag, compute, method_type) as they are — they came from add-model. Only change one if Step 2's diff shows a genuine mismatch (and then to the inferred value; for a single-config method, deleting a mismatched config_default is equally valid).

Example — making chimeraboost upload-ready (the upload fields added; compare nori's info.py, which is already in this shape):

chimeraboost_method_metadata = MethodMetadata.config(
    method="ChimeraBoost",
    suite="tabarena-2026-06-30",                              # added: distinct dated suite
    ag_key="CHIMERA",
    config_default="ChimeraBoost_c1_default_BAG_L1",             # HPO method: post-rename, `_default` = the run's pipeline
    compute="cpu",
    is_bag=False,
    date="2026-06-15",
    reference_url="https://github.com/bbstats/chimeraboost",
    display_name="ChimeraBoost",
    verified=False,                                           # flip to True once verified
    cache_type="r2",                                          # added
    cache_kwargs={"bucket": "tabarena", "prefix": "cache"},   # added
)

Step 4: Execute process → upload (Claude runs, in the background)

Run these with the resolved dotted reference, per method — background + monitor for the slow ones. The same block doubles as the fallback command sheet if a step must be handed to the maintainer. Using chimeraboost as the example:

# 1. Inspect (confirm inferred fields match info.py) — already done in Step 2

# 2. (info.py edited: suite + cache_type/cache_kwargs + date, plus any inspect-diff fixes)

# 3. Process: build + cache metadata.yaml + processed/ + results/ locally
<venv>/bin/python scripts/run_process_method.py <run_data_dir> \
    --method-metadata tabarena.models.chimeraboost.info:chimeraboost_method_metadata --process \
    --expect-validation-protocol tabarena  # the run must record TabArena's official protocol
# (a BeyondArena run: --expect-validation-protocol beyondarena --task-metadata BeyondArena)

# 4. Upload dry-run: verifies every part exists locally and prints what/where (no creds needed)
<venv>/bin/python scripts/run_upload_results.py \
    --method-metadata tabarena.models.chimeraboost.info:chimeraboost_method_metadata

# 5. Real upload (R2 creds via env, NEVER flags):
<venv>/bin/python scripts/run_upload_results.py \
    --method-metadata tabarena.models.chimeraboost.info:chimeraboost_method_metadata --no-dry-run

After the real upload, verify on the bucket (ListObjects via boto3 against the R2 endpoint): each method should have the full object set under cache/artifacts/<suite>/methods/<Method>/ (metadata.yaml, processed.zip, processed/configs_hyperparameters.json, raw.zip, results/*.parquet).

The real upload also records the MD5 of each uploaded results table in packages/tabarena/src/tabarena/models/_artifacts/results_checksums.json. Commit that diff together with the info.py / methods.py edits: download="auto" loaders re-download a cached table that differs from its committed checksum, and tests/tabarena/tools/test_results_checksums.py fails in CI when a registered method has no entry or an entry differs from the hosted object. For a re-upload under an existing suite the diff changes the old entries, which is what makes other machines replace their cached copies. python -m tabarena.tools.results_checksums --check compares the file with the hosted ETags (--refresh rewrites it from them).

Notes:

• process requires the explicit --method-metadata and verifies it against the raw data first; a real mismatch errors (override with --ignore-metadata-mismatch, a method-name mismatch only warns). Raw + HPO trajectories are cached by default (--no-cache-raw / --no-cache-hpo-trajectories).
• The dry-run is the default for the upload script; it needs no credentials and prints the exact --no-dry-run command. Raw uploads by default (--no-upload-raw to skip).
• For the real upload, set R2_ACCOUNT_ID / R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY in the environment (export them or use a .env) — never as CLI flags (they'd leak into shell history / the process table). The dry-run prints how to obtain them if unset (MethodMetadata.r2_credentials_help()).

Step 5: Register in the arena collection (Claude does now)

Add the method to the collection so it appears in the benchmark. For the default tabarena arena, edit packages/tabarena/src/tabarena/contexts/tabarena/methods.py (read it first):

1. Import the model's info.py metadata in the alphabetical import block, matching the file's existing style (most recent additions use the plain name, e.g. from tabarena.models.nori.info import nori_method_metadata; alias only on a name collision).
2. Add the entry to tabarena_method_metadata_collection.method_metadata_lst, under the matching group comment (# Default tabular models (CPU) vs # Neural / GPU / foundation models), placed by compute/type.

It flows into tabarena_method_metadata_complete_collection automatically (no separate edit). Other arenas (e.g. beyondarena) register in their own collection's methods.py.

When the upload is a rerun of an already-registered method (a new library version, a remeasured run), the entry is a swap, not an addition: point the collection at the new metadata and append the one it replaced to methods_superseded, so the predecessor's hosted artifacts stay reachable through the complete collection. Say in the report that the default leaderboard now carries only the new run.

Caveat to surface in the report: the collection entry only resolves to downloadable artifacts once the real upload (Step 4.5) has actually run. The code edit is safe to land now, but the method won't load for others until uploaded.

Step 6: Record the run in the benchmark log (Claude does now)

packages/tabflow_slurm/BENCHMARK_LOG.md is the committed record of every cluster run (the tmp_scripts/run_<model>.py that launched it is gitignored, so this log is the only surviving copy of the setup). The benchmark-model skill merely offers to write the entry at launch time, before any results exist, so in practice runs reach upload unlogged. Check here, where the run is finished:

1. grep -n "^## " packages/tabflow_slurm/BENCHMARK_LOG.md | head — if the run's benchmark_name is already there, nothing to do.
2. Otherwise add an entry at the top of the log (append-only, newest-first), following the template in the file's "Conventions" section: ## YYYY-MM-DD — <benchmark_name>, then Model(s) / Git SHA / Purpose / Notes, then the verbatim plan.

Two fields need care:

• Git SHA is HEAD at setup time, not HEAD now. git reflog --date=iso shows when HEAD sat on which commit; bracket the launch with the run's own timestamps (the earliest results.pkl mtime under the run's data/, or the earliest file in <workspace>/slurm_out/<benchmark_name>/) and pick the commit that was HEAD then.
• The python block is the launch script's setup() body copied verbatim, with its module-level constants (BENCHMARK_NAME, WORKSPACE, PYTHON_PATH, MODEL, NUM_CONFIGS) inlined as literals and _path_setup() expanded, so the snippet stands alone against its SHA. Never refactor a neighbouring entry to match the current API.

The launch script's module docstring usually holds the why (issue link, what changed versus the previous run, partition choice, model-specific caveats) — that is the Purpose/Notes material. Add the processed suite id and the metadata variable name too, so the log ties the run to its artifacts.

Step 7: Lint touched files

ruff check <touched-files>
ruff format --check <touched-files>

Touched files are the model's info.py and contexts/<arena>/methods.py. Fix anything reported (the from __future__ import annotations import is already present in both — don't drop it). BENCHMARK_LOG.md is markdown, so ruff does not apply to it.

Step 8: Free the local disk (Claude does now)

A finished run leaves several copies of its results behind: the raw results.pkl files in the workspace (about 100 GB per method for a full run), the copies --process made in the TabArena cache for the upload (raw plus about 10 GB processed), the setup files and logs, the pinned run environment, and for SkyPilot runs the bucket queue and results. Once the upload is verified, r2 holds everything needed to rebuild them, so remove them before reporting.

Clean up only when all of these hold, and otherwise skip the step and say why in the report:

1. Every method in the run's data dir was uploaded with --no-dry-run, and the bucket listing of Step 4 showed its full object set with non-zero sizes (raw.zip included; a run uploaded with --no-upload-raw keeps its raw files). For a multi-method run that means all of its methods.
2. The info.py, methods.py and results_checksums.json edits are in the working tree, and the BENCHMARK_LOG.md entry exists (Step 6 reads the run's log timestamps and launch script).
3. Nothing about the run is pending: no items to relaunch, no eval the maintainer still wants to redo.

Remove, for the run's <benchmark_name>, <suite> and <Method> (the workspace is the PathSetup workspace of the launch script or the log entry; the cache root is $TABARENA_CACHE, default ~/.cache/tabarena):

Keep metadata.yaml and results/ under <cache root>/artifacts/<suite>/methods/<Method>/ (well under 1 MB, and what update-leaderboard loads next), and leave every shared path alone: the workspace root, other runs' directories, <cache root>/evaluations/ (task data), the bucket's dataset cache and env/ manifests (shared by all runs), and any venv another run or session uses.

Measure each path with du -sh first; on the NFS home that is slow, so run it in the background. State the list with sizes, then delete without waiting for approval unless the maintainer asked to review it. Name each path explicitly (never a glob that can reach another run), and run the large rm -rf in the background too. Report what was removed and how much space it freed.

Step 9: Report

Tell the maintainer:

• What was executed and verified: inspect diff results, processing outcome per method, and the r2 destinations confirmed after the real upload — plus any data-quality warnings from processing (e.g. Not close TEST prediction-fidelity lines, with affected datasets and severity).
• Disk cleanup: the paths removed in Step 8 with the space freed, or why the step was skipped.
• Edits Claude made: the info.py upload fields (suite / cache_type / cache_kwargs / date / verified, plus any inspect-diff fixes), the methods.py import + collection entry (and the methods_superseded append, for a rerun), the results_checksums.json entries the upload recorded, and the BENCHMARK_LOG.md entry (with the SHA it recorded and how it was determined, since that one is inferred).
• Steps handed off (only if the env couldn't run one): the exact command(s) from Step 4.
• Open decisions / TODOs: whether to flip verified to True (only after sign-off), committing the working-tree edits, and — if the method should appear on the website — that update-leaderboard is the next lifecycle step.

Open a PR, comment on one or edit its description only when the maintainer asks: the Report above goes to the chat, and any text meant for a PR is shown to the maintainer first and posted only after they say so, starting with 🤖 Generated with [Claude Code](https://claude.com/claude-code) as its first line unless they say otherwise. When asked to open the PR, use .github/pull_request_template.md and delete its "Model or system submission" section (that one is for external submissions). The summary names the method and what was uploaded; the collapsed Details block carries the method key, suite and date, the r2 destinations, the verified state, and whether update-leaderboard is the next step. Do not paste this Report into the PR body.