AGENTS.md — config

Declarative eval configs. Protected, and executable: a config is untrusted input, not data.

Every file here defines what an evaluation measures, which is why the whole directory is a protected path. README.md lists each config and what consumes it; this file is the security and journey contract an agent has to satisfy to add or change one.

Map

Diagram

flowchart LR
  accTitle: the two gates a config must pass through
  accDescr: A config names a callable target and filesystem paths, which the callable allowlist and the data and output roots confine before the engine runs anything.

  subgraph C["config/ (you are here)"]
    CFG["a config YAML"]
  end

  ALLOW["EVAL_HARNESS_CALLABLE_TARGET_ALLOWLIST<br/>unset means deny"]
  ROOTS["DATA_ROOT / OUTPUT_ROOT"]
  TGT["target: import and call"]
  DS["dataset read"]
  SINK["sink write"]
  ENG["engine run"]
  JRN["pipeline e2e journey test"]

  CFG --> ALLOW
  CFG --> ROOTS
  ALLOW --> TGT
  ROOTS --> DS
  ROOTS --> SINK
  TGT --> ENG
  DS --> ENG
  ENG --> SINK
  CFG --> JRN

  classDef here fill:#e8f0fe,stroke:#1a73e8,stroke-width:2px
  class C here

Rules that bite here

• The whole directory is protected. Changes need the eval-change-approved label and a code owner, because a threshold or a dataset path edited here moves every score without touching a scorer.
• A config is executable input. The callable target turns params.path into an import and a call, gated by EVAL_HARNESS_CALLABLE_TARGET_ALLOWLIST. Unset means deny, and matching is on module boundaries, never a string prefix (ADR 0039). Do not add a new config-driven import or filesystem path without routing it through that gate or through DATA_ROOT / OUTPUT_ROOT.
• A shipped config must be journeyed or explicitly excluded. Every file here is either run end to end by tests/integration/test_pipeline_e2e.py or named in that module's exclusion set with a reason. A config that is neither fails the suite, which is how requirements_eval.yaml once shipped naming an empty evidence store.
• No credentials, ever. Endpoints and keys arrive as environment variables with ${VAR:-default} expansion; ../.env.example is the canonical set.
• No bare numeric literal at a call site. A threshold belongs in a validated config field with the default documented on the field, so it is reviewable here rather than in code.

Verify

python -m pytest tests/integration/test_pipeline_e2e.py -q

Subagents

See also