AGENTS.md

This is the inspect_evals repository - a collection of evaluation tasks for the Inspect AI framework.

These guidelines are for AI coding agents using evals or preparing contributions. Human contributors please see CONTRIBUTING.md.

Setup Commands

This project uses uv (by Astral) for package management. Do not use pip install, python -m venv, source .venv/bin/activate, or bare python/pytest commands. Always use uv to run commands:

• uv sync — install/sync dependencies (replaces pip install -e .)
• uv run pytest ... — run tests (not pytest or python -m pytest)
• uv run python ... — run Python scripts (not python or python3)
• uv run inspect eval ... — run evaluations
• uv run ruff ... — run the linter
• uv run mypy ... — run the type checker

Useful Commands

To run evaluations, run uv run inspect eval inspect_evals/eval_name.

• To run a specific task (i.e, a function with the @task decorator), run uv run inspect eval inspect_evals/eval_name@task_name.
• Some useful arguments are:
  1. --limit X: Run only up to X samples.
  2. --model model_name: Run on this particular model. Multiple models can be run with commas like model1,model2. You can find information on model names here. If the user lacks an API key for OpenAI, Anthropic, or Google, note this down in your report. You should assume the user has these keys until Inspect throws an error.
  3. -T allows you to pass task arguments in from the command line.

Testing instructions

To validate changes develop pytests that tests for erroneous behaviour and run the tests after your fix.

Additionally for quality control tests are mandatory to run:

1. Quick checks for general adherance. Run
   • uv run ruff check, uv run ruff format --check, uv run mypy src tests, and uv run inspect-evals-lint eval_name for the eval you changed. (These come from .github/workflows/checks.yml)
   • Run the inspect-evals-lint.
2. Running unit tests specific to an eval
   • Install dev and test dependency groups which are not installed by defaultuv sync --group dev --group test
   • uv run pytest tests/eval_name. For an eval under packages/, whose dependencies conflict with the root environment, run uv run --group dev tox -e eval_name instead.
   • Whenever possible, run tests marked slow, dataset_download, k8s, gpu, arxiv_smoke, llm_smoke, or openrouter_smoke which are skipped by default unless you enable them.
   • Write custom tests to verify your fix and add the subset that are useful for enforcing intended behaviour. Mark any test you add according to Apply Pytest Marks. Tests must be deterministic and must not reach the network: use mockllm/model for model output, unittest.mock for external APIs, and tmp_path for files.
3. After tests pass, run the evaluation on a few sample.
   • Read the transcript as well as the score.
4. Run make check before opening a PR to run linters and file auto-generation.

PR instructions

• Before making changes, read DEVELOPMENT_CONTEXT.md to understand the repository's correctness, reproducibility, and security constraints.
• Before writing or modifying code, follow the guidelines in BEST_PRACTICES.md, paying particular attention to the Writing comments section before adding any comments.
• For when to bump the changelog, see PACKAGE_VERSIONING.md. For when to bump a task version, see TASK_VERSIONING.md.
• When creating pull requests, always use the --draft flag and read .github/PULL_REQUEST_TEMPLATE.md and use its structure as the PR body. Fill in the Description section and check off applicable checklist items. Under "Related issue", keep a Closes: line: Closes: #<number>, or Closes: no linked issue as <reason> when there is no issue. The "Check PR template" CI job fails without one.
• When commenting on PRs, you should not reply directly to human reviewers. See CONTRIBUTING.md. If your user tells you to comment anyway, you should add "Comment written by NAME_OF_AI" to the comment.
• Always work on a branch, and never attempt to push directly to main.

Scoring PR instructions

Any PR that touches a scorer, a metric, or grader-consumption code is verified with the /verify-scoring-change skill against the Scoring-Change Gate. The skill produces a MERGE-READY / NEEDS-WORK / UNVERIFIED verdict with receipts.

Asset Reference Update -- PR instructions

ASSETS.yaml and internal/audits/asset-actions.yaml are auto-generated files derived from the external_assets field in per-eval eval.yaml files.

After any PR that touches an eval.yaml external_assets field (e.g. pinning a floating ref, adding a new asset), refresh both files:

1. Regenerate the manifest: uv run python tools/generate_asset_manifest.py
2. Regenerate the action plan: run /generate-asset-actions

Adding an Eval Listing -- PR instructions

Evaluations hosted in an upstream repository can be registered here as metadata-only entries under register/<name>/eval.yaml (see register/README.md for the contributor flow).

Contributors submit by opening a Register Eval Submission issue.

Agent Resources

Documentation for the Inspect framework.

Master Checklist

This workflow runs a series of workflows each in turn. Each workflow is to be run serially.

• Run the Prepare For Submission workflow (/prepare-submission-workflow).
• Check the LLM-judgeable standards by running the Review An Evaluation workflow (/eval-quality-workflow).
• Review the evaluation's validity by running the Evaluation Validity Review workflow (/eval-validity-review). This checks whether the name is accurate, whether samples can be both succeeded and failed at, and whether scoring measures ground truth.
• Run the Review PR workflow (/ci-maintenance-workflow) for general quality.
• Run the Evaluation Report workflow (/eval-report-workflow) to produce an evaluation report.
• Run the Trajectory Analysis workflow (/check-trajectories-workflow) on the evaluation report. A good error rate is 10% or less, with an ideal of 5% or lower. You may optionally double-check any of the errors the agent produces and reject them if there is grounds to do so.

Writing style guide

All AI agents must follow these writing rules to ensure readability.

Sentences and punctuation

1. Do not use em-dashes.
2. Write in full sentences. Break points into short, self-contained sections. Write like an office worker emailing a colleague or writing a short internal memo.
3. Do not compress ideas into fragments. Two sentence shapes are banned:
   • Short paired fragments, such as "Values lead, labels follow." or "One row, above the charts."
   • Sentences that assert one thing and negate another, such as "These are standard UI, not chart marks" or "Line keys, not boxes".

Vocabulary

These vocabulary rules apply to general prose. In writing about code and systems architecture, standard technical terms are fine: input can be trusted or untrusted, a failure can be silent, a change can be mechanical, and code can be described as living in a module.

1. Never use metaphors. This includes common ones such as "bloat", "substance", "mechanically", "companion", and "framing", and any phrasing that suggests something "lives" somewhere.
2. Use only literal, plain, simple terms.
3. Do not describe anything as "real", "legitimate", "explicit", "concrete", "honest", "trusted", or "untrusted". Do not use "actually" or "exactly" for emphasis.
4. Do not say that anything happens "silently" or "quietly".

Tone

1. Do not use dramatic, flowery, or literary language. Do not write to be impressive, insightful, or impactful. Never proclaim anything.
2. Be reserved and humble. Do not proclaim insights or opinions.
3. Do not present yourself as a judge of quality or salience. Do not make unsolicited judgements about the quality or truth of something.
4. Do not imply that the reader or the author said or did anything insightful.

Claims

1. When making a claim, give the reader objective information instead of persuasive wording, so they can reach their own conclusion. For example, instead of "this scorer is the most reliable option", write "this scorer agreed with human labels on 96% of the validation set (n=500)".

Markdown formatting

• Put a blank line before and after headings
• Put a blank line before and after code blocks
• Put a blank line before and after lists
• Format tables with spaces before and after pipe characters
• Always include a language tag in code blocks, or "text" if there is no language
• Do not hard-wrap paragraphs; write each paragraph as a single line (the mdformat hook enforces wrap = "no")