CLAUDE.md

The primary agent guide for this repo is AGENTS.md. Read it first — everything in it applies here.

This file only documents Claude-specific extensions.

Claude-Specific Tooling

Skills (.claude/skills/)

• add-model — Use whenever the user asks to add/integrate/wrap a new tabular ML model. The input is usually just a link (repository, paper, PyPI or Hugging Face page) or a name; the skill derives the name, keys, pin, license, organization and the authors' preprocessing recipe from those sources and asks only for what they leave open. It encodes the full change: a per-model folder (model.py wrapper, hpo.py search space, info.py registry entry) plus edits to models/__init__.py, models/utils.py, and the pyproject.toml extra — and points to reference implementations for each model class (foundation, torch, sklearn-like). It also decides the model's untimed environment warmup (classmethod convention in models/warmup.py), asking the user when the library's warm-up/pre-compile entry point is unknown and reporting warm-up's limitation (main process + disk-backed caches only; parallel-fold Ray workers stay cold). The model is auto-discovered from its info.py and fit-tested automatically by tests/tabarena/models/test_all_models.py (tests specific to the model, if any, go into models/<key>/tests/, outside CI); only add a smoke_configs.py override if its toy fit needs faster hyperparameters. Before the report, reviewer agents that did not write the wrapper check it and the pinned library for cheating, leakage and protocol violations along three lenses (references/model_review.md), and each finding is verified against the code before it is reported or fixed.
• add-system — The sibling of add-model, for a whole pipeline rather than a single model: AutoML frameworks (AutoGluon, LightAutoML, FLAML), LLM-driven agents, hosted prediction APIs, or a model run through a heavier self-managing interface (TabFM+). Use it whenever the thing being added does its own model selection, tuning and ensembling inside its own budget, so inventing a TabArena search space for it makes no sense. It creates packages/tabarena/src/tabarena/systems/<key>/ (system.py = the ExternalSystemModel subclass, hpo.py = the SystemConfigGenerator, info.py = SystemInfo + MethodMetadata.system(...)), and its main judgment call is tags: with-llm and closed-source-api decide which leaderboard entrant pools the system competes in, so the skill asks rather than guesses. For a hosted API (Step 1b) it also sets the wrapper contract (key from the environment, nothing that names the task or split in a request, separate fit and predict calls preferred, server version in get_metadata()) and checks the client with python -P -m tabarena.tools.audit_system --offline. Sits before benchmark-model in the lifecycle.
• benchmark-model — Use whenever a maintainer wants to run an already-integrated model on the benchmark (e.g. "benchmark TabM", "run Nori on the cluster", "benchmark <PR link>", "make a setup/eval script for DenseLight"). The input is short: a PR link, a model name, a repository link, or nothing when the conversation names the model; the skill resolves the registry name and takes the defaults for everything else. It first asks whether Claude should drive the run end-to-end here (launch, monitor, evaluate, report) or hand the launch and monitoring to the maintainer, then scaffolds a single tmp_scripts/run_<model>.py at the repo root with smoke, setup and eval subcommands sharing one benchmark_name + PathSetup (so they can't drift). Defaults: the full TabArena-v0.1 task set with all configs (0 for foundation models without a search space), the run venv resolved to the one that imports this checkout, the model's pip extra installed into it, a local smoke fit on whatever compute exists (CPU when there is no GPU, GPU models included), and, mandatory for every GPU model, fake_memory_for_estimates set to the card's VRAM in GB (asked when not inferable). In end-to-end mode it runs setup, applies the printed SkyPilot pool and launches its jobs, reports the percentage of work left at regular intervals via references/sky_progress.sh (checking failed items and their logs, relaunching the missing items after a fix), then runs eval with all figures for the default subsets, reports the leaderboard with the new model's rows highlighted, and publishes a Claude artifact with the figures to paste into the PR. A model that was not integrated in the same session, in particular one from a contributor's PR, first goes through the add-model skill's agent review (Step 1b), and fixes on top of the PR follow the add-model conventions. A SkyPilot job pool draining a GCS claim queue is the default; --scheduler skypilot runs one managed job per spot VM instead, and --scheduler slurm the same plan as a SLURM array (references/slurm_progress.sh watches that). The eval box-labels the new model in every Pareto figure (pareto_focus_new_methods, on by default in TabArenaEvalConfig). Registered systems run through the same plan (generator tuple, system_experiments=True); a hosted API gets references/run_api_system_template.py (audit, smoke, setup, eval): the client is read and the API probed for transduction and label lookup with tabarena.tools.audit_system before anything runs, smoke and eval check that the submitter's self-reported TabArena-Lite errors reproduce on the same splits (compare_reproduced_results), the key stays in the maintainer's shell, and the run goes from one on-demand CPU node (SlurmSingleNodeSetup, num_workers = the provider's concurrency). Sits in the lifecycle between add-model (integrate) and upload-method (publish).
• upload-method — Use whenever a maintainer points at a benchmark run's output dir and wants to process / upload / register a method's results (e.g. "upload this method", "host/publish <model>'s results", "register <model> in the leaderboard"). By default Claude runs the whole flow itself after stating the plan (scripts/run_process_method.py inspect → --process; scripts/run_upload_results.py dry-run → --no-dry-run, background + monitors, r2 verified after upload) and lands the edits: the model's info.py MethodMetadata (suite / date / cache_type / cache_kwargs / verified, plus inspect-diff fixes), the arena-collection registration in contexts/<arena>/methods.py, and the run's BENCHMARK_LOG.md entry when the launch stage never wrote one. Once r2 is verified it frees the disk: the run's raw results, the cache copies made for the upload, its setup_out / slurm_out dirs, the pinned worktrees and venv, and the SkyPilot bucket prefix, keeping only the small metadata.yaml + results/ in the cache. A command sheet is only handed off when the env can't run a step (no tabarena[benchmark] venv / no R2 creds). Results from a contributor's own run are verified first (references/verify_submission.md: restricted unpickling, a scan of every results.pkl, split/label hashes against our own run, hyperparameters against the search space, the claimed Elo reproduced with compare, and the Elo impact of any problem found). Mirrors AGENTS.md → "Processing & uploading method artifacts (maintainers)".
• update-leaderboard — Use whenever a maintainer wants to regenerate the website artifacts and refresh a leaderboard Space with the latest results (e.g. "update the leaderboard", "regenerate the artifacts and refresh leaderboard-testing", "push the new results to the leaderboard"). Takes the path to the leaderboard Space repo (data/ + main.py + its own .venv) as its key input. It (1) pre-checks that newly added models classify correctly in website/website_format.py (Foundation Model / Tree-based / … not ❓ Other) and that every registered organization has a logo in the Space's assets/logos/ + views.py map before generating, (2) runs scripts/run_generate_website_artifacts.py (background + monitor; the trailing ray SIGTERM traces are harmless), (3) refreshes the Space's data/ via delete-then-copy (CSVs, n_datasets.json and the one-file page pack pages.sqlite), (4) bumps the version history in the Space's website_texts.py (new dated entry + current-version line, verified/unverified from the model's info.py), and (5) optionally serves the Space locally (its own .venv, 127.0.0.1:7860). Last stage after upload-method; the maintainer commits/pushes (Git LFS + Xet).

The five skills above are for working inside this repo. The one below is for a different repository consuming TabArena as a dependency:

• adapt-tabarena — Use whenever a user is building their own domain-specific benchmark (spectroscopy, genomics, time series, a vertical-specific tabular task) in a separate repo and wants to reuse TabArena's model zoo, task and experiment runner, and/or bencheval's leaderboard math instead of reimplementing that layer. Covers what to depend on (bencheval alone for leaderboard math vs. tabarena for the registry and runner too), turning the domain's datasets into UserTasks (data contract, split format, registering a domain eval metric), the repeated-k-fold + size-adaptive-repeats + group-aware outer protocol and the inner ValidationProtocol (declared on the arena context and asserted on bagged experiments; the data-foundry extra grouped folds need), a domain bundle subclass with ModelConstraints, registering domain models next to TabArena's registry via register_model_info (or wrapping self-tuning pipelines as systems), layering domain preprocessing without forking any wrapper, ValidationProtocol parity, running locally or on a cluster (CacheConfig, JobBatch, tabflow_slurm), imputation and Elo anchoring in compare, building a full domain arena (an AbstractArenaContext subclass, a TaskMetadataSource catalogue, a methods.py roster, hosted artifacts, website output), the git-dependency PyPI trap, and a step that has the downstream author report the API gaps they hit to the TabArena maintainers. Distilled from a real migration (RamanBench, github.com/ml-lab-htw/RamanBench).

When the user describes work that matches a skill's trigger criteria, invoke the skill via the Skill tool instead of recreating the steps manually.

Working Style in This Repo

• Always run ruff check --fix on touched files before reporting a task complete. The from __future__ import annotations requirement (isort required-imports) is the most common CI failure on new files. CI now enforces ruff check . + ruff format --check ., and a pinned .pre-commit-config.yaml is available (pre-commit install) — see AGENTS.md "Lint & Format".
• Tests live in the top-level tests/, grouped by package — when adding a test for packages/tabarena/src/tabarena/<area>/foo.py, mirror the path under tests/tabarena/<area>/test_foo.py. (tests/bencheval/ and tests/tabflow_slurm/ hold the other two packages; tests/integration/ is for cross-package tests.) The exception is a model's or system's own tests: they go into models/<key>/tests/ next to its code, stay out of the default pytest and CI, and run with pytest packages/tabarena/src/tabarena/models/<key>/tests (AGENTS.md "Testing").
• Model-fit tests are deselected by default and consolidated — tests/tabarena/models/test_all_models.py fits every registered model (marked models, skipped by the default pytest). Run one with pytest -m models -k <Model>; they need tabarena[benchmark], are slow, and skip GPU-only models without CUDA. There is no per-model test file in tests/; tests specific to one model go into models/<key>/tests/. For non-model changes, the default pytest already skips them. tests/tabarena/models/test_warmup_coverage.py (same marker) runs python -P -m tabarena.tools.audit_warmup per model and fails when a timed fit or predict imports a package the warm-up missed.
• Run python from the repo root, with -P — python -m ... or python -c ... from the workspace directory that contains the editable autogluon/ checkout shadows the installed AutoGluon and empties the model registry (AGENTS.md "Testing"). Use cd <repo root> && python -P -m ... for the audit CLI, the run scripts and run_local; the pytest console script is safe from any cwd.
• Optional model deps: when adding code that imports an optional library (e.g., tabpfn, tabicl, xrfm), keep the import inside the wrapper's _fit / class body, never at module top-level — top-level imports break installs without that extra.
• Writing style is enforced — AGENTS.md ends with an "AI Writing Tropes to Avoid" section. It covers docstrings, comments, markdown, commit messages, PR descriptions, and chat replies. The rules most likely to bite here: no em dashes / -- pivots by reflex, no X → Y arrows in docs, no "it's not X, it's Y" reframes, and no bold-first bullets in new docs.
• PR and issue text follows the templates. PR bodies use .github/pull_request_template.md: a two-to-four sentence summary, a collapsed <details><summary>Details</summary> block for the longer description, then Tests (see AGENTS.md "PR & Commit Guidance"). A skill's Report is for the chat and the PR body is for reviewers, so don't paste one into the other. Issues use the forms in .github/ISSUE_TEMPLATE/.
• Mark what Claude posts. Every PR or issue body, comment and review comment Claude posts on the maintainer's behalf starts with the line 🤖 Generated with [Claude Code](https://claude.com/claude-code), so readers know before they read, unless the maintainer says otherwise for that text. Commits carry the Co-Authored-By line instead.
• Docstrings describe current behavior, not refactor history. When you rename, move, or remove code, write docstrings/comments about what the code is now — don't leave Formerly X, used to live in Y, previously did Z, or now delegates here narratives, or dead references to deleted classes/modules. Rename/removal history belongs in git and the PR, not the source. Keep a historical note only when it's genuinely important to understanding the current code (e.g. a non-obvious backwards-compat shim). (previously / used to live are fine when they explain current behavior and reference code that still exists.)