AGENTS.md

Guidance for coding agents working in this repository. Human-facing docs live in README.md.

Project Overview

TabArena is a living benchmark for tabular machine learning. It evaluates ML methods across 51 curated datasets with cross-validated ensembles, HPO simulation, and leaderboard generation. Built on top of AutoGluon.

Repository Layout

This repo is a uv workspace (root pyproject.toml). Its installable packages live under packages/ (workspace members tabarena, bencheval, tabflow_slurm), alongside supporting dirs:

• packages/tabarena/ — Core package. Repository pattern for benchmark data, model wrappers (src/tabarena/models/), system wrappers (src/tabarena/systems/), simulation, evaluation, plotting. Depends on AutoGluon and bencheval.
• packages/bencheval/ — Standalone lightweight metrics/leaderboard package (ELO, win-rates, ranks, improvability). Computes leaderboards from results DataFrames. No dependency on tabarena.
• packages/tabflow_slurm/ — Package (own pyproject.toml, a uv-workspace member) for running experiments on SLURM clusters. Depends on tabarena. See packages/tabflow_slurm/README.md and packages/tabflow_slurm/AGENTS.md.
• examples/ — Usage examples: the model and system quick starts (benchmarking/, beyondarena/), leaderboard generation and the per-split result export (plots/), meta-learning, custom datasets.
• tests/ — All tests, grouped by package (tests/tabarena/, tests/bencheval/, tests/tabflow_slurm/, mirroring each package's src/ layout) plus tests/integration/ for cross-package tests.

Setup Commands

Requires Python 3.11–3.13 and uv. This is a uv virtual workspace (the root pyproject.toml has no [project] table), so install the tabarena package directly from packages/tabarena rather than running uv sync at the root. --prerelease=allow is required for the pre-release AutoGluon dependency. From the repo root, after creating/activating a venv (uv venv --seed --python 3.12 && source .venv/bin/activate):

uv pip install --prerelease=allow -e "./packages/tabarena"               # Minimal: evaluation/leaderboard/metrics only
uv pip install --prerelease=allow -e "./packages/tabarena[plot]"         # + leaderboard/result plotting
uv pip install --prerelease=allow -e "./packages/tabarena[text]"         # + semantic text features (sentence-transformers; pulls torch)
uv pip install --prerelease=allow -e "./packages/tabarena[preprocessing]" # + skrub datetime/statistical-text feature generators
uv pip install --prerelease=allow -e "./packages/tabarena[benchmark]"    # Full install (models + plot + text + preprocessing)

The core install is intentionally minimal (issue #323): it depends on autogluon.tabular (not the full autogluon meta-package) and leaves plotting, text embeddings, and skrub feature generators to the extras above (all imported lazily). [benchmark] is the union.

For editable AutoGluon development (one directory up):

../autogluon/full_install.sh
uv pip install --prerelease=allow -e "./packages/tabarena[benchmark]"

Lint & Format

ruff check .               # Lint (config: ruff.toml)
ruff format .              # Format

Key rules: from __future__ import annotations is required in every file (enforced via isort required-imports). Line length 120 (the formatter is the authority — E501 is not enforced). Google-style docstrings. Run ruff on touched files before finishing a task.

CI runs ruff check . and ruff format --check . (see .github/workflows/pytest-pytest.yml), so lint/format violations fail the build. Optionally install the local pre-commit hook so commits are auto-fixed before they reach CI:

pip install pre-commit && pre-commit install   # one-time, per clone

After that, git commit runs ruff on staged files; if a hook reformats or fixes anything the commit aborts — re-git add and commit again. The ruff version is pinned identically in .pre-commit-config.yaml, the CI workflow, and the lint dependency group in packages/tabarena/pyproject.toml; keep all three in sync.

Testing

Tests live in a single top-level tests/ dir, organized to mirror src/ (tabarena areas as subfolders — metrics/, repository/, benchmark/, models/, … — plus tests/bencheval/ and tests/tabflow_slurm/ for the other two packages). The root pyproject.toml [tool.pytest.ini_options] sets testpaths = ["tests"], so a bare pytest from the repo root runs the whole suite.

The one exception is a model's or system's own tests. They live next to its code in packages/tabarena/src/tabarena/models/<key>/tests/ (or systems/<key>/tests/), with an empty __init__.py. testpaths keeps them out of a bare pytest and of CI, and the wheel excludes them; run them when changing that model: pytest packages/tabarena/src/tabarena/models/<key>/tests. Don't mark them models (the default -m would deselect them on an explicit run too), skip on a missing optional dependency with pytest.importorskip, and write fit artifacts to tmp_path. The registry fit test every model shares (tests/tabarena/models/test_all_models.py with smoke_configs.py) stays in tests/.

pytest                                      # All tests
pytest tests/metrics/test_metrics.py        # Single file
pytest -k test_name -x                      # Single test, stop on failure
pytest tests/bencheval                      # One package's tests

The default pytest deselects two slow/fragile groups via addopts (-m 'not network and not models'):

• network — tests that hit the network (e.g. download a Hugging Face model).
• models — test_all_models.py, which fits every registered model via AutoGluon's FitHelper. It is parametrized over the model registry and skips models whose optional deps aren't installed (ImportError) or that need a GPU (compute='gpu', no CUDA). Run one model with pytest -m models -k TabM, or the whole sweep with pytest -m models (needs tabarena[benchmark]). test_warmup_coverage.py carries the same marker and runs the warm-up audit per model in a fresh python -P subprocess.

Both groups run in the nightly workflow.

Run python -m ... and python -c ... from the repo root, never from the workspace directory that holds the editable autogluon/ checkout: with that directory on sys.path[0], autogluon.tabular resolves to an empty namespace package, every model info.py fails to import, and the registry comes out empty. get_model_registry() raises in that case (assert_autogluon_resolves), registry_or_fail() in tests/tabarena/models/smoke_configs.py fails collection instead of parametrizing a registry-driven test over nothing, and python -P (or PYTHONSAFEPATH=1) makes the interpreter safe from any cwd. The pytest console script is safe from any cwd. setup, sbatch and python -m tabflow_slurm.run_local are run from the repo root too: Ray puts the driver's cwd on every worker's sys.path, which -P does not cover. CI's per-PR job (.github/workflows/pytest-pytest.yml) runs pytest on Python 3.12 against ./packages/tabarena[plot,preprocessing,data-foundry] plus the editable tabflow_slurm package (so its tests and the data_foundry-gated tests run), but not [text]/[benchmark] — so it stays fast (no model fitting, no torch).

Architecture

Core data flow

Raw predictions → EvaluationRepository → Simulation/Portfolio → Results DataFrames → TabArena leaderboard (bencheval)

Key abstractions

• EvaluationRepository (packages/tabarena/src/tabarena/repository/evaluation_repository.py) — Central class combining config metadata/rankings (ZeroshotSimulatorContext), cached val/test predictions (TabularModelPredictions), and GroundTruth. Supports subsetting by datasets/folds/configs/problem_types and ensemble selection via mixins.
• TabularModelPredictions (packages/tabarena/src/tabarena/predictions/) — Abstract base for prediction storage. Implementations: TabularPredictionsInMemory (dict-based) and TabularPredictionsMemmap (disk-based memory-mapped for large benchmarks). Structure: {dataset: {fold: {val/test: {config: predictions}}}}.
• AbstractExecModel (packages/tabarena/src/tabarena/benchmark/exec_models/base.py) — Base for the benchmark execution wrappers (the AutoGluon wrappers live in benchmark/exec_models/autogluon.py). New benchmarked models live in one folder per model at packages/tabarena/src/tabarena/models/<model>/ (model.py = AutoGluon wrapper subclassing AG's AbstractModel/AbstractTorchModel, hpo.py = search-space generator, info.py = ModelInfo/MethodMetadata registry entry), auto-discovered by packages/tabarena/src/tabarena/models/_registry.py::discover_models() (which packages/tabarena/src/tabarena/benchmark/exec_models/registry.py then derives the AG registry from). Use the add-model skill — there is no benchmark/models/ag/<model>/ layout for new models.
• ExternalSystemModel (packages/tabarena/src/tabarena/benchmark/exec_models/external.py) — Base for a benchmarked system: a whole pipeline that picks, tunes and ensembles models inside its own budget (AutoGluon, TabFM+, an agent, a hosted API). Systems live one folder per system at packages/tabarena/src/tabarena/systems/<system>/ (system.py = the ExternalSystemModel subclass, hpo.py = the SystemConfigGenerator, info.py = SystemInfo/MethodMetadata), auto-discovered by systems/_registry.py::discover_systems(). See "Systems" below.
• ExperimentRunner / ExperimentBatchRunner (packages/tabarena/src/tabarena/benchmark/experiment/) — Execute model fitting across tasks. Configured via YAML (experiment_constructor.py).
• ZeroshotSimulatorContext (packages/tabarena/src/tabarena/simulation/) — Manages config rankings for HPO simulation and portfolio generation.
• BenchmarkEvaluator (packages/bencheval/src/bencheval/evaluator.py) — Leaderboard computation from results DataFrames. Independent of the core tabarena package.
• AbstractArenaContext.compare (packages/tabarena/src/tabarena/contexts/abstract_arena_context.py) — Scores an arena's results (plus any registered new method) into the leaderboard and, given an output_dir, writes it there through LeaderboardReporter.eval (packages/tabarena/src/tabarena/evaluation/leaderboard_reporter.py): tabarena_leaderboard.csv, results_per_split.csv, method_info.csv, winrate_matrix.csv, pareto_front_points.csv, leaderboard.tex and the figures. results_per_split.csv is the frame the leaderboard is scored from: one row per dataset, split and method (metric_error, metric_error_val, time_train_s, time_infer_s, imputed, ...), with leaderboard display names and imputed rows flagged; return_results=True returns the input frame instead (registered method names, before imputation). plot=False writes the CSVs and skips the figures; output_dir=None (context.leaderboard()) writes nothing. The per-split results are not hosted anywhere (the leaderboard Space only embeds them in its per-dataset browser), so a request for per-dataset or per-split numbers is answered with examples/plots/run_export_results_per_split.py (issues #262 and #595 were that request).
• EntrantPool (packages/tabarena/src/tabarena/evaluation/entrants.py) — One field of competitors, evaluated together. The leaderboard publishes one artifact tree per pool. See "Entrant pools" below.

Systems

A model is one method run under TabArena's shared tuning protocol. A system manages its own budget, model selection and ensembling; benchmarking it means handing it the data and the constraints and recording what comes back.

Two fields on MethodMetadata carry this, both orthogonal to method_type (which stays a result-shape discriminator: which parquet load_results reads, whether HPO simulation applies):

• method_class — "model" (default) or "system". Not inferable from raw results, which record a system exactly like any other method_type="baseline" run, so a system must declare it. Build the metadata with MethodMetadata.system(...).
• tags — from the MethodTag vocabulary, for what a reader must weigh before trusting a comparison. Keep it small; a tag earns its place only when someone would reasonably exclude a method because of it.
  • with-llm — an LLM is involved somewhere, agents included. Says nothing about whether the system is open-source.
  • closed-source-api — the method runs behind a remote API we cannot inspect and whose behavior can change. Being an API implies closed-source here, so this is one tag rather than two.

Adding a system mirrors add-model: create systems/<key>/ with system.py, hpo.py and info.py, and export a <key>_info: SystemInfo that discover_systems() picks up. Systems stay out of the AutoGluon model registry on purpose (no ag_key, no search space) and run through the experiment bundle's system_experiments=True mode. SystemInfo.__post_init__ rejects metadata that is not method_class="system", so a misdeclared system fails at import rather than misclassifying downstream. Runnable references: examples/benchmarking/run_quickstart_tabarena_system.py and examples/beyondarena/run_quickstart_beyondarena_system.py; their _model siblings are the model quick starts, and the README section "Contributing a Model or System" is the human-facing guide for both.

A hosted API (closed-source-api) receives the training table and the test features in requests to code nobody can inspect, while every TabArena dataset is public. Before it runs, python -P -m tabarena.tools.audit_system --system <name> reads the client (a static scan and every outbound request with its JSON outline and tables; --offline blocks the network, needs no key and sends nothing) and probes the API for blatant cheating: predictions that depend on the other rows of the test batch, predictions that track the true labels after the training labels were relabeled or shuffled, and a collapse under 1% feature noise that a local reference model does not show, judged on the test rows without an exact copy among the training rows (a model may lean on such a copy). --submitted-results compares a submitter's per-split errors with the hosted methods, and compare_reproduced_results (called from the API run template's smoke and eval) checks that our own run reproduces the errors the submitter reported on the same splits. A FAIL stops the submission. The run itself needs no compute on our side: SlurmSingleNodeSetup runs every item from one on-demand CPU node with num_workers requests in flight (the concurrency the provider allows), with the key exported in the submitting shell and never written to a file. compute in info.py is the provider's hardware, so processing only warns when the raw results record the CPU client. Timings are wall-clock at the client, network and provider queue included, and an API that fits and predicts in one call records the whole fit as time_infer_s. Such a system declares fits_at_predict=True in its MethodMetadata: its loaded results then charge the prediction call to time_train_s as well (the hosted tables keep the raw times), and the leaderboard table marks that train time with a †. The add-system skill (Step 1b) has the wrapper contract and the benchmark-model skill the run.

The website types a system from method_class, never from its name. website_format.get_model_family classifies models by name prefix, but a system short-circuits to the System family via the declared set (system_display_names), so a new system never lands in ❓ Other.

Entrant pools

Every headline number is relative to who competed: Elo is pairwise over the participants, improvability is 1 - best_error_in_pool / error, and the ranks are positions in the field. Narrowing the field therefore needs a recompute, not a row filter, which is why the entrant pool is a subset axis alongside imputation and splits rather than a toggle on the published table.

evaluation/entrants.py groups systems into independently selectable SYSTEM_CATEGORIES: open (untagged, i.e. open-source and local), llm (with-llm), and api (closed-source-api). Every combination is published as its own pool, so there are 2 ** len(SYSTEM_CATEGORIES) = 8, keyed models, open, llm, api, open_llm, open_api, llm_api, open_llm_api. Models always compete.

Every pool also has a twin without the methods that carry a leakage warning, keyed <key>_noleak (16 pools in all). MethodMetadata.leakage_evidence (a link to the evidence, set on the ModelDescriptor since it belongs to the weights) and leakage_datasets (the datasets it is proven on) mark a method whose training saw test data; LimiX-2 carries it per autogluon/tabarena#648. The website tables carry them as Leakage / LeakageDatasets, the converter writes the flagged methods into every chart it ships (each marks their names), and the leaderboard's "Include methods with a leakage warning" switch reads the twin. compare(exclude_leakage=True) builds any leaderboard or figure without them; BeyondArena's generator writes its twin under subsets_noleak/ and result_plots_noleak/.

Independent rather than a cumulative ladder because "LLM-based systems but not the plain open-source ones" is a question people actually have, and a ladder cannot express it. A system carrying several tags belongs to several categories and needs all of them selected, so a closed-API LLM system never appears on the strength of a property the reader excluded.

evaluation/subset_grid.py crosses the pools with the other axes into a 1920-cell grid and owns the folder layout (entrants_<key>/imputation_.../splits_.../tasks_.../datasets_...), which the leaderboard Space's Subset.rel_path mirrors segment for segment. Adding a fourth category doubles the artifact count and the generation time; that is the price of independent toggles.

One coupling to know about when you register a system: eval_all.get_pool_reference_lines returns the pool's admitted systems as (baselines, baseline_colors), and baselines is not only what the figures draw as horizontal reference lines. LeaderboardReporter.eval keeps a row only when its method maps to a config framework type or is named in baselines, and a system is neither, so a system missing from that list is deleted from the pool's published numbers with nothing logged. It is derived from method_metadata_info for exactly that reason, and tests/tabarena/evaluation/test_entrants.py guards that the widest pool covers every system in the shipped collection.

Validation protocol

The inner validation protocol is how a bagged model is validated inside one outer split: the bagging fold and repeat counts, an optional tiny-data regime, whether the inner splits follow the task's group / time structure, and whether AutoGluon may shrink the fold count to the minority-class count. It is one object, ValidationProtocol (packages/tabarena/src/tabarena/benchmark/validation_protocol.py), and two constants: TABARENA_V0PT1_VALIDATION_PROTOCOL (8 folds x 1 set, plain stratified splits, key 8x1) and BEYONDARENA_VALIDATION_PROTOCOL (8x1 with 5x5 at or below 500 training group instances, task-specific splits, class-adaptive folds, key 8x1+tiny5x5<=500+task-specific+adapt-classes). Identity is the field-derived key(), never the name label.

The arena context owns it. TabArenaContext and BeyondArenaContext declare theirs as OFFICIAL_VALIDATION_PROTOCOL; build_jobs / run_jobs stamp it onto every bagged, holdout and full-predictor experiment that carries none, refuse a bagged experiment that carries another protocol (the error names both protocols and the paired bundle), refuse a method_kwargs["validation_metadata"] split-structure override, and check that the task collection still carries the official split structure and splits. The official bundles (TabArenaV0pt1ExperimentBundle, BeyondArenaExperimentBundle) set no protocol, so context plus bundle is the one correct pipeline for a submission. The same assertion runs at every later stage: the tabflow head node refuses a cached result fit under another protocol, the JobBatch ships the context's ValidationExpectation (validation_protocol.json) and the compute node re-checks the experiment against it, Experiment.task_cache_scope refuses a bagged or holdout fit without a protocol, and a results cache carries a validation_protocol.json side record so a rerun under another protocol never reuses it.

Only bagged models and systems are official flavours (OFFICIAL_VALIDATION_FLAVOURS); outer, holdout and full-predictor fits run outside the protocol by construction, need no flag, and are recorded as such. To run bagged experiments under another protocol, build the context with official_validation_protocol=False and choose the protocol at one of three levels: the context (validation_protocol=), the bundle or ModelJob.experiment (per model group), or a single experiment / generate_bag_experiments(validation_protocol=...) (per config); examples/advanced/run_custom_validation_protocol.py shows all three. A bare AbstractArenaContext enforces exactly the protocol it is given.

Every result records what it ran under (results["validation_protocol"]: flavour, protocol, composite key, resolved and fitted counts, the data-dependent clamps, the split structure, the bag's child count and use_child_oof). Downstream, the per-result frame carries vp_* columns, MethodMetadata.validation_protocol holds the key ("8x1", the BeyondArena key, "system", "<flavour>:<key>" for non-bagged fits, "mixed", or None for results that predate the record), the rosters declare it for every hosted method, processing fails when a new run's info.py leaves it undeclared, run_process_method.py --expect-validation-protocol asserts the arena's protocol for a submission, and the contexts warn when a registered method ran a custom protocol. KNN is the one declared outlier: its bag is a single child with leave-one-out predictions (use_child_oof), recorded as child_oof=True and listed as such by the inspector, never as a deviation.

"Protocol" is used for five different things in this repo; keep them apart: the outer split scheme of a task collection (folds x repeats of the benchmark), the task-aware inner splitting (group / time structure), the bagging protocol above (the ValidationProtocol), the tuning protocol (search space, time limits, the default / tuned / tuned + ensembled variants), and BeyondArena's core and core2k subsets.

Data caching

TabArena uses five independent caches. Configure them all at once with tabarena.caching.CacheConfig — the single, documented surface (TabArenaContext(cache_config=CacheConfig.from_root(...)); the context applies it on construction and re-applies it inside run_jobs, so distributed workers inherit it). The SLURM path uses the same object: the setup embeds context.cache_config in the JobBatch, and each worker applies it (no --openml_cache_dir wiring). A node without the shared filesystem (a SkyPilot VM) runs the runner with --cache_root <dir> instead, which puts every cache under one local directory, and with --materialize_tasks True, which downloads the job's dataset through the suite the JobBatch recorded in task_source.json (TaskMetadataCollection.preset / with_preset). See the CacheConfig docstring for the authoritative per-cache reference.

Model libraries with a checkpoint cache of their own are not managed here: TabPFN keeps its .ckpt files under ~/.cache/tabpfn (not HF_HOME), and moving that is done with tabpfn's own TABPFN_MODEL_CACHE_DIR variable, set before the first tabpfn import (the SLURM job environment carries it to the compute nodes; the TabPFN wrappers' prefetch_weights fill whatever directory tabpfn resolves). The opt-in weight staging of the SLURM runner copies those files to node-local scratch for one job and points the variable there for that job only.

Processing & uploading method artifacts (maintainers)

Turn a benchmark run's already-present raw results.pkl files into cached, hosted, and registered TabArena artifacts — no download, no auto-generation. Steps 1–2 are single-method CLIs (full flag reference in each script's module docstring); step 3 is a small code change.

1. Author + process — scripts/run_process_method.py (logic in tabarena.tools.process_local_raw_data):
   • Inspect the raw dir for a suggested metadata snippet: python scripts/run_process_method.py <run_dir>/data (recursive; prints the inferred fields + a MethodMetadata.config/.baseline/.portfolio(...) snippet). Paste it into packages/tabarena/src/tabarena/models/<model>/info.py (or systems/<system>/info.py for a system) and fill in suite (required, must differ from method) + the manual fields.
   • Process: append --method-metadata tabarena.models.<model>.info:<x>_method_metadata --process. Processing requires an explicit MethodMetadata, verified against the raw data first (method_type / compute / ag_key / config_default / can_hpo / is_bag / validation_protocol must align, and method != suite); a run whose results record a validation protocol must declare it in info.py (raw artifacts written before the record existed pass), and --expect-validation-protocol tabarena|beyondarena additionally asserts the arena's official protocol for a submission; a method-name mismatch only warns and model_key is not checked, other mismatches error unless --ignore-metadata-mismatch. A declared config_default is compared post-rename and is the usual failure: the TabArena-v0.1 bundle appends the preprocessing pipeline name to every config id, so a run's first config is <Method>_c1_default_BAG_L1 (HPO or not), while info.py files authored before the run often carry the older <Method>_c1_BAG_L1; set it to the inferred value. A single-config method (can_hpo=False) may leave config_default unset: it is not checked, the lone config is its default, and processing records the resolved name in the cached metadata.yaml. HPO methods should declare it. The task metadata is fetched from OpenML by task id, so a run whose task directories are slugs (BeyondArena, UserTasks) also needs --task-metadata BeyondArena (or a *_tasks_metadata.csv path); without it every result is dropped. Caches metadata.yaml + processed/ + results/ (plus raw + HPO trajectories, on by default) under the TabArena cache.
2. Upload to r2 — scripts/run_upload_results.py:
   • Dry-run (default) verifies each part exists locally and prints what/where: python scripts/run_upload_results.py --method-metadata tabarena.models.<model>.info:<x>_method_metadata (or --from-cache METHOD SUITE to load from the local cache).
   • Real upload: add --no-dry-run with R2_ACCOUNT_ID / R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY set in the environment (never as flags). r2 only — the metadata needs cache_type="r2" + cache_kwargs={"bucket", "prefix"}; the dry-run prints the exact --no-dry-run command and, when the creds are unset, how to obtain them (MethodMetadata.r2_credentials_help()). raw uploads by default (--no-upload-raw to skip).
   • The real upload records each uploaded results table's MD5 in packages/tabarena/src/tabarena/models/_artifacts/results_checksums.json; commit it with the registration. download="auto" loaders re-download a cached table that differs from its committed checksum (a re-upload under the same suite), and tests/tabarena/tools/test_results_checksums.py fails when a registered method has no entry or an entry differs from the hosted object (python -m tabarena.tools.results_checksums --check, --refresh to rewrite it from the hosted ETags).
3. Register in the appropriate context's collection — add the method so it appears in the benchmark. For TabArena, import the model's info.py method_metadata and add it to tabarena_method_metadata_collection in packages/tabarena/src/tabarena/contexts/tabarena/methods.py (the collection lists each model's info.py metadata directly, and is itself the paper method set used by TabArenaContext). It flows into tabarena_method_metadata_complete_collection automatically. Other arena contexts register in their own collection.
4. Free the disk — once the bucket listing shows the full object set (raw.zip included), r2 can rebuild every local copy, so remove the run's output/<run>/, setup_out/<run>/ and slurm_out/<run>/ in the workspace, the cache's raw/ and processed/ for the method (keep metadata.yaml + results/), the eval cache artifacts/<run>/, the pinned run worktrees and venv, and a SkyPilot run's bucket prefix. The upload-method skill (Step 8) lists the preconditions and what to leave alone.

Example

# Verify local cache state
python scripts/run_process_method.py /path/to/nori_regression_18062026/data

# Edit metadata in info.py

 python scripts/run_process_method.py /path/to/nori_regression_18062026/data --method-metadata tabarena.models.nori.info:nori_method_metadata --process

python scripts/run_upload_results.py --method-metadata tabarena.models.nori.info:nori_method_metadata

python scripts/run_upload_results.py --method-metadata tabarena.models.nori.info:nori_method_metadata --no-dry-run

# Add model to the collection in `packages/tabarena/src/tabarena/contexts/tabarena/methods.py`

Releasing to PyPI (maintainers)

The git checkout is the recommended install. PyPI carries an experimental copy of bencheval and tabarena so downstream packages can depend on them without git URLs (issue #495). The two packages share one version (version in both packages/*/pyproject.toml must be equal), and every tabarena release pins bencheval==<version>.

scripts/release/build_pypi_dists.py builds both packages from a staged copy, so the checkout is never modified, and then verifies the wheels. Compared to the pyproject files in the repo, the PyPI build sets the version and the bencheval pin, strips requirements with a direct URL (name @ git+..., which PyPI rejects) so that tabarena[tabfm], [sap-rpt-oss] and [exaone_tabular] are empty on PyPI while the model's pip_extra in its info.py keeps the git dependency for the install hint, and copies the root README.md (relative links made absolute), LICENSE and NOTICE next to each pyproject. The verification fails on a leftover direct URL, an empty description, a missing license file, or a git-tracked non-.py file under src/ that the package-data globs missed. The install-from-wheel job in .github/workflows/install-benchmark.yml runs the build on every PR and installs the wheels non-editable.

All three publishing workflows run in the pypi GitHub environment and authenticate with PyPI trusted publishing (OIDC), so no token is stored anywhere:

To cut a stable release, bump version in both pyproject files in a PR, merge it, then git tag v<version> && git push origin v<version>. Pre-releases and dev builds never change the repository. Local dry run: python scripts/release/build_pypi_dists.py --version 0.1.0a1 writes the checked distributions to dist/ (needs uv on PATH).

One-time setup, and who can do it:

• PyPI trusted publishers need the PyPI project owner. For each of tabarena and bencheval, add one publisher per workflow file (release.yml, prerelease.yml, dev-release.yml) with owner autogluon, repository tabarena, environment pypi. bencheval does not exist on PyPI yet, so add it as a pending publisher, which reserves the name until the first upload. Until this is done, uv publish --trusted-publishing always fails with an OIDC error instead of publishing.
• The pypi GitHub environment is created by GitHub on the first workflow run that references it. Protection rules on it (required reviewers, restricting deployments to tags) need repository admin rights, which maintainers with push access do not have; ask an autogluon org admin.

Conventions

• Add a new model: create one folder packages/tabarena/src/tabarena/models/<model>/ (model.py, hpo.py, info.py, __init__.py), then edit models/__init__.py (lazy class entry), packages/tabarena/pyproject.toml (a per-model extra), and website/website_format.py (the leaderboard family); models/utils.py needs no edit (auto-registry). The registry auto-discovers the model from its info.py, and tests/tabarena/models/test_all_models.py then fits it automatically — there is no per-model fit test file (a model may still have a unit-test file for wrapper logic that a fit does not reach). Only add an entry to tests/tabarena/models/smoke_configs.py if the smoke fit needs faster toy hyperparameters or a restricted problem-type set (keyed by the model's MethodMetadata.method). Use the add-model skill, which encodes this and points to reference implementations (foundation / torch / sklearn).
• Warm-up and the timing audit: every timed fit is preceded by an untimed environment warm-up (AbstractExecModel.warmup_fn, layered per model class in tabarena/models/warmup.py: an optional warmup classmethod, torch and the CUDA context, the class's warmup_modules, a dummy fit on synthetic data). A wrapper declares its lazy imports in warmup_modules and its cheapness knobs in cheap_hyperparameters. A foundation model declares shared_weights = SharedWeights(loader=..., key=..., disabled_by=...) (from autogluon.core.models.abstract), naming the library call that builds its network and the inputs that decide which network it is; AutoGluon's AbstractTorchModel then builds that network once per process (the warm-up's dummy fit is the first build), pickles the fitted model without it and takes it back on load, and _fit does not change. When a library stands in the way of the warm-up or the sharing (it loads inside a constructor, loads lazily at the first predict, or flips a global torch flag when imported), the wrapper carries a small workaround whose docstring or comment opens with Developer fix, names the upstream gap, what the library should offer instead and the upstream issue or PR when one exists (exaone_tabular/_estimators.py, the eager predictor build in nori/model.py, the pinned TF32 flag in iltm/model.py); grep for the header to find them, and remove each once the library ships the fix, as TabDPT's constructor replica went when layer6ai-labs/TabDPT-inference#79 merged (tabdpt/model.py declares the upstream _load_model). The add-model skill (Step 3g) has the library contract and the pattern. The fairness contract (data-independent work only) is in the warmup.py docstring. Check a model with python -P -m tabarena.tools.audit_warmup --model <Method> from the repo root (the timed fit and predict should import no new package), a finished run with --results <output dir>; tests/tabarena/models/test_warmup_coverage.py (marker models) asserts it per registered model. Both see the main process only: parallel Ray fold workers start cold. A warm-up that raised or left steps failed aborts the run before the timed fit (ExperimentRunner(require_warmup=True), the default; the SLURM runner's --require_warmup false or TabArenaBenchmarkPlan(require_warmup=False) bypasses it), so a cold timed fit is never recorded silently.
• Result metadata: on top of the timing and memory keys, a results.pkl carries experiment_metadata["warmup_report"], ["timing_audit"], ["cpu_thread_info"] and ["time_warmup_s"], and the method metadata shared_weights, persisted_models, prepared_for_inference and per_child_test_source ("timed_children_pass" when the timed predict ran AutoGluon's predict_proba_children once and derived the bag's output from the arrays, "timed_prediction" for a single-child bag on the predictor's own path, "child_forward_pass" when the children predicted a second time). Results written before a key existed lack it, so consumers read these with .get (the _collect_memory_usage docstring in exec_models/base.py lists the baseline shifts of the memory keys).
• Add a new system: create one folder packages/tabarena/src/tabarena/systems/<system>/ (system.py with the ExternalSystemModel subclass, hpo.py with the SystemConfigGenerator, info.py with the SystemInfo + MethodMetadata.system(...), __init__.py) and add the extra to packages/tabarena/pyproject.toml. discover_systems() picks it up from info.py; pytest tests/tabarena/systems/ checks the registration, and there is no per-system fit test (verify with examples/benchmarking/run_quickstart_tabarena_system.py). tags decide the entrant pools (see "Systems" above). A hosted API also passes tabarena.tools.audit_system (see "Systems"). Use the add-system skill.
• Imports: from __future__ import annotations must be the first import in every .py file. Use absolute imports rooted at the package (e.g., from tabarena.repository import EvaluationRepository).
• Optional dependencies: each model has its own pyproject extra under packages/tabarena/pyproject.toml; the benchmark extra is the union. Heavy/optional libs must never be imported at module top-level in core paths — import inside the model wrapper.
• Scoping jobs: when scoping context.build_jobs / build_and_run_jobs (in examples and code), pass a typed TaskSubset via task_subset= — it is the single source of truth for the available filters (subset, dataset_names, split_indices, problem_types, n_train_samples, ...). Prefer it over the loose dataset_names= / build_kwargs={...} keyword conveniences.
• No new top-level docs files unless the user asks. Edit existing files in place.
• Prose style: everything you write for a human reader (docstrings, comments, markdown, commit messages, PR descriptions, chat replies) follows AI Writing Tropes to Avoid at the bottom of this file.

PR & Commit Guidance

• Keep commits focused; do not bundle unrelated refactors with bug fixes.
• Run ruff check . and pytest on affected paths before opening a PR.
• CI is mandatory on main PRs.
• PR description: a two-to-four sentence summary first, then a <details><summary>Details</summary> block (collapsed by default) with the longer description. Both are written for the humans reviewing the PR; keep the whole body short.
• Use .github/pull_request_template.md. For a model or system submission fill in the "Model or system submission" section; otherwise delete it. Keep the closing contribution line.
• Issues use the forms in .github/ISSUE_TEMPLATE/ (submission questions; leaderboard or dataset questions that touch this code). The contribution process (evaluate on TabArena-Lite first, maintainers verify and re-run, leaderboard updated after merge) is in the README section "Contributing a Model or System" and applies when an agent drafts a submission PR too.

Things to Avoid

• Do not add a tst/ dir or per-package tests/ dirs — all tests live in the single top-level tests/, grouped by package (tests/tabarena/, tests/bencheval/, tests/tabflow_slurm/, tests/integration/). The exception is a model's or system's own tests in models/<key>/tests/ (see Testing); tests specific to one model never go into tests/.
• Do not import optional model dependencies at the top of shared modules; lazy-import inside the wrapper.
• Do not skip from __future__ import annotations — ruff will fail CI.
• Do not change the public API of EvaluationRepository, TabularModelPredictions, or bencheval.evaluator.BenchmarkEvaluator without explicit user direction; they are consumed by external scripts and artifacts.

AI Writing Tropes to Avoid

Applies to everything you write in this repo that a human reads: docstrings, comments, markdown docs, commit messages, PR descriptions, user-facing copy, and your replies in the chat.

Source: tropes.fyi by ossama.is

Word Choice

"Quietly" and Other Magic Adverbs

Overuse of "quietly" and similar adverbs to convey subtle importance or understated power. AI reaches for these adverbs to make mundane descriptions feel significant. Also includes: "deeply", "fundamentally", "remarkably", "arguably".

Avoid patterns like:

• "quietly orchestrating workflows, decisions, and interactions"
• "the one that quietly suffocates everything else"
• "a quiet intelligence behind it"

"Delve" and Friends

Used to be the most infamous AI tell. "Delve" went from an uncommon English word to appearing in a staggering percentage of AI-generated text. Part of a family of overused AI vocabulary including "certainly", "utilize", "leverage" (as a verb), "robust", "streamline", and "harness".

Avoid patterns like:

• "Let's delve into the details..."
• "Delving deeper into this topic..."
• "We certainly need to leverage these robust frameworks..."

"Tapestry" and "Landscape"

Overuse of ornate or grandiose nouns where simpler words would do. "Tapestry" is used to describe anything interconnected. "Landscape" is used to describe any field or domain. Other offenders: "paradigm", "synergy", "ecosystem", "framework".

Avoid patterns like:

• "The rich tapestry of human experience..."
• "Navigating the complex landscape of modern AI..."
• "The ever-evolving landscape of technology..."

The "Serves As" Dodge

Replacing simple "is" or "are" with pompous alternatives like "serves as", "stands as", "marks", or "represents". AI avoids basic copulas because its repetition penalty pushes it toward fancier constructions (I've studied this!).

Avoid patterns like:

• "The building serves as a reminder of the city's heritage."
• "Gallery 825 serves as LAAA's exhibition space for contemporary art."
• "The station marks a pivotal moment in the evolution of regional transit."

Sentence Structure

Negative Parallelism

The "It's not X -- it's Y" pattern, often with an em dash. The single most commonly identified AI writing tell. Man I f*cking hate it. AI uses this to create false profundity by framing everything as a surprising reframe. One in a piece can be effective; ten in a blog post is a genuine insult to the reader. Before LLMs, people simply did not write like this at scale. Includes the causal variant "not because X, but because Y" where every explanation is framed as a surprise reveal, the em-dash dismissal "X -- not Y", and the cross-sentence reframe where the same noun is negated then repositioned: "The question isn't X. The question is Y."

Avoid patterns like:

• "It's not bold. It's backwards."
• "Feeding isn't nutrition. It's dialysis."
• "Half the bugs you chase aren't in your code. They're in your head."

"Not X. Not Y. Just Z."

The dramatic countdown pattern. AI builds tension by negating two or more things before revealing the actual point. Creates a false sense of narrowing down to the truth.

Avoid patterns like:

• "Not a bug. Not a feature. A fundamental design flaw."
• "Not ten. Not fifty. Five hundred and twenty-three lint violations across 67 files."
• "not recklessly, not completely, but enough"

"The X? A Y."

Self-posed rhetorical questions answered immediately in the next sentence or clause. The model asks a question nobody was asking, then answers it for dramatic effect. Thinks this is the epitome of great writing.

Avoid patterns like:

• "The result? Devastating."
• "The worst part? Nobody saw it coming."
• "The scary part? This attack vector is perfect for developers."

Anaphora Abuse

Repeating the same sentence opening multiple times in quick succession.

Avoid patterns like:

• "They assume that users will pay... They assume that developers will build... They assume that ecosystems will emerge... They assume that..."
• "They could expose... They could offer... They could provide... They could create... They could let... They could unlock..."
• "They have built engines, but not vehicles. They have built power, but not leverage. They have built walls, but not doors."

Tricolon Abuse

Overuse of the rule-of-three pattern, often extended to four or five. A single tricolon is elegant; three back-to-back tricolons are a pattern recognition failure.

Avoid patterns like:

• "Products impress people; platforms empower them. Products solve problems; platforms create worlds. Products scale linearly; platforms scale exponentially."
• "identity, payments, compute, distribution"
• "workflows, decisions, and interactions"

"It's Worth Noting"

Filler transitions that signal nothing. AI uses these phrases to introduce new points without actually connecting them to the previous argument. Also includes: "It bears mentioning", "Importantly", "Interestingly", "Notably".

Avoid patterns like:

• "It's worth noting that this approach has limitations."
• "Importantly, we must consider the broader implications."
• "Interestingly, this pattern repeats across industries."

Superficial Analyses

Tacking a present participle ("-ing") phrase onto the end of a sentence to inject shallow analysis that says nothing. The model attaches significance, legacy, or broader meaning to mundane facts using phrases like "highlighting its importance", "reflecting broader trends", or "contributing to the development of...".

Avoid patterns like:

• "contributing to the region's rich cultural heritage"
• "This etymology highlights the enduring legacy of the community's resistance and the transformative power of unity in shaping its identity."
• "underscoring its role as a dynamic hub of activity and culture"

False Ranges

Using "from X to Y" constructions where X and Y aren't on any real scale. In legitimate use, "from X to Y" implies a spectrum with a meaningful middle. AI uses it as a fancy way to list two loosely related things. "From innovation to cultural transformation" -- what's in between???? Nothing!

Avoid patterns like:

• "From innovation to implementation to cultural transformation."
• "From the singularity of the Big Bang to the grand cosmic web."
• "From problem-solving and tool-making to scientific discovery, artistic expression, and technological innovation."

Paragraph Structure

Short Punchy Fragments

Excessive use of very short sentences or sentence fragments as standalone paragraphs for manufactured emphasis. RLHF training has pushed models toward "writing for readability" aimed at the lowest common denominator: one thought per sentence, no mental state-keeping required. It's an inhuman style. No real person writes first drafts this way because it doesn't match how humans think or speak.

Avoid patterns like:

• "He published this. Openly. In a book. As a priest."
• "These weren't just products. And the software side matched. Then it professionalised. But I adapted."
• "Platforms do."

Listicle in a Trench Coat

Numbered or labeled points dressed up as continuous prose. The model writes what is essentially a listicle but wraps each point in a paragraph that starts with "The first... The second... The third..." to disguise the format. Perhaps you told it to stop generating lists and it decided to do this instead... still very common.

Avoid patterns like:

• "The first wall is the absence of a free, scoped API... The second wall is the lack of delegated access... The third wall is the absence of scoped permissions..."
• "The second takeaway is that... The third takeaway is that... The fourth takeaway is that..."

Tone

"Here's the Kicker"

False suspense transitions that promise a revelation but deliver a point that did NOT need the buildup. The model uses these phrases to manufacture drama before an otherwise unremarkable observation LOL. Also includes: "Here's the thing", "Here's where it gets interesting", "Here's what most people miss", "Here's the starting point", "Here's the deal".

Avoid patterns like:

• "Here's the kicker."
• "Here's the thing about AI adoption."
• "Here's where it gets interesting."

"Think of It As..."

The patronizing analogy. AI constantly reaches for "Think of it as..." or "It's like a..." to simplify concepts. The model defaults to teacher mode and assumes the reader needs a metaphor to understand anything. Often produces analogies that are less clear than the original concept.

Avoid patterns like:

• "Think of it like a highway system for data."
• "Think of it as a Swiss Army knife for your workflow."
• "It's like asking someone to buy a car they're only allowed to sit in while it's parked."

"Imagine a World Where..."

The classic AI invitation to futurism. To sell the argument usually begins with "Imagine" followed by a list of wonderful things that will happen if the reader agrees with the premise.

Avoid patterns like:

• "Imagine a world where every tool you use -- your calendar, your inbox, your documents, your CRM, your code editor -- has a quiet intelligence behind it..."
• "In that world, workflows stop being collections of manual steps and start becoming orchestrations."

False Vulnerability

Simulated self-awareness or honesty that reads as performative. The model pretends to break the fourth wall or admit a bias, creating a false sense of authenticity. Real vulnerability is specific and uncomfortable; AI vulnerability is polished and risk-free!!!!

Avoid patterns like:

• "And yes, I'm openly in love with the platform model"
• "And yes, since we're being honest: I'm looking at you, OpenAI, Google, Anthropic, Meta"
• "This is not a rant; it's a diagnosis"

"The Truth Is Simple"

Asserting that something is obvious, clear or simple instead of actually proving it. If you have to tell the reader your point is clear, it very likely isn't. Also includes the dramatic reveal variant: "but none of them is the real story. The real story is..." -- claiming privileged insight while waving away everything before it.

Avoid patterns like:

• "The reality is simpler and less flattering"
• "History is unambiguous on this point"
• "History is clear, the metrics are clear, the examples are clear"

Grandiose Stakes Inflation

Everything is the most important thing ever. AI inflates the stakes of every argument to world-historical significance. A blog post about API pricing becomes a meditation on the fate of civilization.

Avoid patterns like:

• "This will fundamentally reshape how we think about everything."
• "will define the next era of computing"
• "something entirely new"

"Let's Break This Down"

The pedagogical voice that assumes the reader needs hand-holding. AI defaults to a teacher-student dynamic even when writing for expert audiences. Also includes: "Let's unpack this", "Let's explore", "Let's dive in".

Avoid patterns like:

• "Let's break this down step by step."
• "Let's unpack what this really means."
• "Let's explore this idea further."

Vague Attributions

Attributing claims to unnamed authorities instead of being specific. AI loves to invoke "experts", "observers", "industry reports", and "several publications" without naming anyone. It also inflates the quantity of sources -- presenting what one person said as a widely held view, or writing "several publications have cited" when it means two. If you can't name the expert, you don't have a source.

Avoid patterns like:

• "Experts argue that this approach has significant drawbacks."
• "Industry reports suggest that adoption is accelerating."
• "Observers have cited the initiative as a turning point."

Invented Concept Labels

AI clusters invented compound labels that sound analytical without being grounded. It appends abstract problem-nouns (paradox, trap, creep, divide, vacuum, inversion) to domain words -- "supervision paradox", "acceleration trap", "workload creep" -- and uses them as if they're established, rigorously defined terms. They function as rhetorical shorthand: name a thing, skip the argument. Multiple such labels in the same piece is a strong signal of AI slop.

Avoid patterns like:

• "the supervision paradox"
• "the acceleration trap"
• "workload creep"

Formatting

Em-Dash Addiction

Compulsive overuse of em dashes for dramatic pauses, parenthetical asides and pivot points. A human writer might use 2-3 per piece (and naturally); AI will use 20+.

Avoid patterns like:

• "The problem -- and this is the part nobody talks about -- is systemic."
• "The tinkerer spirit didn't die of natural causes -- it was bought out."
• "Not recklessly, not completely -- but enough -- enough to matter."

Double-Hyphen Dash

The em dash wearing a false moustache. Once "em dash means AI" became common knowledge, the character started getting swapped for a double hyphen: sometimes because the text passed through a markdown conversion, sometimes because someone ran a find-and-replace to look more human, sometimes because the model was steered off the character while keeping the habit. Either way the compulsive mid-sentence pivot survives the substitution, which is what actually gives it away. Writers who reach for double hyphens honestly do so once or twice out of typographic laziness, rarely fifteen times in one post. Flagged at five or more per thousand words.

Avoid patterns like:

• "The problem -- and this is the part nobody talks about -- is systemic."
• "It's not a rewrite -- it's a reckoning."
• "We shipped it fast -- maybe too fast -- and paid for it later."

Bold-First Bullets

Every bullet point or list item starts with a bolded phrase or sentence. Extremely common in Claude and ChatGPT markdown output. Almost nobody formats lists this way when writing by hand. It's a telltale sign of AI-generated documentation and blog posts AND README files (especially with emojis).

Avoid patterns like:

• "Every single bullet point begins with a bold keyword."
• "Security: Environment-based configuration with..."
• "Performance: Lazy loading of expensive resources..."

Unicode Decoration

Use of unicode arrows (->), smart/curly quotes, and other special characters that can't be easily typed on a standard keyboard. Real writers typing in a text editor produce straight quotes and -> or =>. Claude in particular loves the -> arrow.

Avoid patterns like:

• "Input → Processing → Output"
• "This leads to better outcomes → which means higher engagement"
• "“Smart quotes” instead of straight "quotes" that you’d actually type"

Composition

Fractal Summaries

"What I'm going to tell you; what I'm telling you; what I just told you" -- applied at every level of the document. Every subsection gets a summary. Every section gets a summary. The document itself gets a summary.

Avoid patterns like:

• "In this section, we'll explore... [3000 words later] ...as we've seen in this section."
• "A conclusion that restates every point already made in the previous 3000 words"
• "And so we return to where we began."

The Dead Metaphor

Latching onto a single metaphor and beating it into the ground across the entire thing. A human writer would introduce a metaphor, use it then move on. AI will repeat the same metaphor 5-10 times.

Avoid patterns like:

• "The ecosystem needs ecosystems to build ecosystem value."
• "Walls and doors used 30+ times in the same article"
• "Every paragraph finds a way to say "primitives" again"

Historical Analogy Stacking

ESPECIALLY COMMON IN TECHNICAL WRITING: Rapid-fire listing of historical companies or tech revolutions to build false authority.

Avoid patterns like:

• "Apple didn't build Uber. Facebook didn't build Spotify. Stripe didn't build Shopify. AWS didn't build Airbnb."
• "Every major technological shift -- the web, mobile, social, cloud -- followed the same pattern."
• "Take Spotify... Or consider Uber... Airbnb followed a similar path... Shopify is another example... Even Discord..."

One-Point Dilution

Making a single argument and restating it in 10 different ways across thousands of words. The model pads a simple thesis to feel "comprehensive" by rephrasing the same idea with different metaphors, examples, and framings. An 800-word argument becomes 4000 words of circular repetition.

Avoid patterns like:

• "The same point, restated eight ways across 4000 words."
• "Each section rephrases the thesis with a different metaphor but adds nothing new"

Content Duplication

Repeating entire sections or paragraphs verbatim within the same piece. This happens when the model loses track of what it has already written, especially in longer pieces. A dead giveaway of unedited AI output. Less common nowadays.

Avoid patterns like:

• "The same section appeared twice, word-for-word identical."
• "Paragraph 3 and paragraph 17 are the same sentence reworded"

The Signposted Conclusion

Explicitly announcing the conclusion with "In conclusion", "To sum up", or "In summary". Competent writing doesn't need to tell you it's concluding. The reader can feel it. AI signals its structural moves because it's following a template, not writing organically.

Avoid patterns like:

• "In conclusion, the future of AI depends on..."
• "To sum up, we've explored three key themes..."
• "In summary, the evidence suggests..."

"Despite Its Challenges..."

The rigid formula where AI acknowledges problems only to immediately dismiss them. Always follows the same beat: "Despite its [positive words], [subject] faces challenges..." then ends with "Despite these challenges, [optimistic conclusion].".

Avoid patterns like:

• "Despite these challenges, the initiative continues to thrive."
• "Despite its industrial and residential prosperity, Korattur faces challenges typical of urban areas."
• "Despite their promising applications, pyroelectric materials face several challenges that must be addressed for broader adoption."

Remember: any of these patterns used once might be fine. The problem is when multiple tropes appear together or when a single trope is used repeatedly. Write like a human: varied, imperfect, specific.