AGENTS.md — scripts/ (DataPulse MY pipeline)

Working agreement for AI agents editing the 133 Python scripts that drive the datapulse-my pipeline.

What this is

scripts/ contains the build pipeline. Each script does one thing:

• gen_*.py regenerate artifacts from upstream sources (timer-tick runs them)
• verify_*.py enforce invariants (release-blocking)
• validate_*.py check schema/content
• embed_*.py render the dashboard HTML
• check*.py operator smoke-test entry points

Scripts are read-and-execute: every commit triggers the deterministic-safety-net gate which runs all *tests*.py. A bad script breaks the deploy pipeline within 5 minutes.

Hard rules

1. One script = one job. If a script does two unrelated things, split it. gen_health_history.py regenerates health/history.jsonl; it does not also regenerate changelog.json. (That's gen_changelog.py's job.)
2. Every artifact is a generator output; nothing is hand-authored. Generated, and never hand-edited: health/, record-evidence/, .attestations/, data/, docs/index.html, datapulse.json, changelog.json, mcp.json, catalog-snapshot.json. .attestations/chain_head.json is generated too — gen_attestations.py is its only writer (rule 5), and an earlier revision of this rule listed it as hand-authored in the same document that names its generator, which is the contradiction this rule now forbids. Sources — scripts/, config/, templates/, the two schema files — are authored once and then read by generators and validators; they are not artifacts and nothing regenerates them in place. Where a file has no generator yet (datapulse.schema.json, health.schema.json, and the manifest's dataset rows), that is a gap to close with a generator, not a licence to edit it by hand.
3. Never bypass --quick-test. gen_health_history.py and similar long-running scripts have a --quick-test flag for fast iteration. The flag is mandatory during dispatch — a 60-min full run wastes Codex quota.
4. The 10-status taxonomy is hard-coded across multiple files. If you change it, you MUST update: embed_dashboard_data.py (status display), gen_changelog.py (status sections), gen_dashboard_filters.py, gen_dashboard_sections.py, health.schema.json, and mcp/server.py (status enums). Use a shared constant — don't repeat the tuple.
5. Attestation chain integrity is non-negotiable. gen_attestations.py is the only writer to .attestations/chain_head.json. Re-running with different inputs produces a different chain head — that's expected, but never edit chain_head.json directly. The release pipeline (.github/workflows/release-please.yml) asserts the chain head is stamped on each release; manual edits will break release attestation.
6. git add order matters for the safety-net gate. The gate compares pre/post SHAs to determine diff scope. If you regenerate artifacts and then commit them in the same dispatch, the diff scope includes your source change + regenerated artifacts. This is fine for the timer; for operator commits, separate the source change from the regen into two commits.
7. Determinism is load-bearing. Two runs of any gen_*.py on the same inputs must produce byte-identical output. If your script introduces non-determinism (random IDs, timestamps, sort instability), the deterministic-safety-net fails and the release is blocked.
8. Output paths go to a small allowlist (scripts/_artifact_paths.py or equivalent). New output paths must be added there before the safety-net accepts them.

Style conventions

• Shebang: #!/usr/bin/env python3 on every file.
• Imports: from __future__ import annotations then stdlib/third-party/local in 3 groups, alphabetised within each group. Absolute imports preferred.
• Type hints: required on all public functions; mypy --strict clean for new code. Internal helpers can use less strict typing.
• CLI: argparse with --quick-test, --out <path>, --in <path> consistently. No sys.argv slicing.
• Logging: stdlib logging with module-level logger named after the script (logging.getLogger(__name__)). Never print() in pipeline scripts — operators want log levels.
• Errors: raise typed exceptions (ConfigError, SourceError, SchemaError) with descriptive messages. The deterministic-safety-net reports failures via test output, not Python tracebacks.
• Atomic writes: every generator writes to <path>.tmp then os.rename(<path>.tmp, <path>). Partial writes must never be visible at <path>.
• Schema validation: every generator validates input JSON against health.schema.json or datapulse.schema.json before reading. Failed validation = explicit error, not silent coercion.

Script-by-script taxonomy

(Full file list is in ls scripts/*.py; this table covers the load-bearing ones.)

What is NOT in scripts/

• The installed pipeline entrypoint — scripts/datapulse-pipeline.sh is maintained here and deployed to /usr/local/sbin/datapulse-pipeline.sh by scripts/install-pipeline.sh. The service must not execute it from the checkout because each cycle checks out artifact commits in that same tree.
• The deploy flow — .github/workflows/deploy-cloudflare-pages.yml (Cloudflare Pages; separate concern)
• The public MCP server code — that's mcp/server.py, not scripts

Out of scope

• Anything that touches upstream data (the probe scripts are read-only by design)
• Anything that writes outside the health/, record-evidence/, .attestations/, data/, docs/, changelog.json, mcp.json, catalog-snapshot.json allowlist
• New long-running scripts without a --quick-test flag — the deterministic-safety-net will fail if you can prove determinism in <5 minutes

Pre-flight checklist (for dispatch briefs)

When writing a codex brief that touches any scripts/*.py:

• Which script(s) change? file + line range + which inputs/outputs are affected
• What does the regeneration cascade look like? list downstream consumers (e.g. embed_dashboard_data.py reads gen_dashboard_*.json)
• Does the change touch the 10-status taxonomy? STOP and surface to operator first
• Is the script deterministic? brief must explicitly state "two runs on same input → byte-identical output"
• What's the --quick-test plan? brief must include the command
• What does the deterministic-safety-net gate look like after the change? brief must include the expected pytest scripts/tests/ output
• Workdir: absolute path to this repo