AGENTS.md — r3dz4r/datapulse-my

Working agreement for AI agents (Hermes, Codex, Claude Code) operating inside this repository. Read once, then follow on every change.

What this repo is

r3dz4r/datapulse-my — open-source trust layer for Malaysian public data. 447 official datasets from data.gov.my, BNM, DOSM, DOE, KKM, KPDN, MET Malaysia continuously probed and classified into a 10-status taxonomy (fresh | aging | stale | discontinued | degraded | browser_dependent | unreachable | unknown | unknown_freshness | reference). Live MCP server with 19 read-only tools (canonical count: jq '.tools | length' mcp.json — do NOT hardcode this number in prose). Hosted on https://www.data-pulse.my and https://mcp.data-pulse.my.

One-line constraint: every change here is publicly visible. The dashboard auto-deploys on push to main (modulo [skip deploy] trailers). Bad data leaks fast.

Hard rules

1. Do not write/edit/refactor code. Dispatch codex-run (with a tight brief). You may edit .gitignore, commit messages, and markdown files under docs/ — those are operator-owned. A dispatched worker is the exception and must write the scoped code itself: this rule addresses the Hermes orchestrator, not the worker, and it does not authorise a worker to dispatch another agent. A worker that stops because it read this rule as applying to itself has misread it — report any other conflict, but not this one.
2. No --yolo, no bypassing preflight gates. The deterministic-safety-net job in .github/workflows/ci.yml must stay green. If it fails, fix the gate (or the brief that fed into it) — don't silence the failure.
3. Push requires explicit "push". "Proceed" / "go" / "dispatch codex" authorises only the immediately preceding, clearly scoped action. "Push" must be literal.
   • Exception — the health-automation publish lane merges itself (operator decision 2026-10-23). .github/workflows/health-automation-merge.yml merges health-automation into main on green, using the datapulse-release App, with autoMerge=true. Measured on PR #2629: merged 2026-10-23T13:29:37Z by app/datapulse-release before any operator word was given. The gate on that path is the required checks — deterministic-safety-net and the isolated preview — not a human message, so no agent should ask for "push" to release a health-automation publish; asking after the fact is how a completed merge gets reported as pending. The literal word still governs every other push: operator commits, source changes, release-please, attestation appends, and anything merged by hand.
4. Never silently rotate credentials, edit firewall rules, or stop systemd services on the public-facing VPS (https://data-pulse.my). These are always-ask.
5. No upstream mutations. Datapulse MY is read-only by design. New code must NOT write to data.gov.my, BNM, DOSM, or any other upstream. If a use case requires writes, that's a different product — not this repo.
6. The 10-status taxonomy is stable. Adding a status requires changing gen_health_methodology.py, embed_dashboard_data.py, gen_changelog.py, the dashboard hero, mcp/server.py, AND the public-facing methodology page. If you think you need an 11th status, surface it to the operator first.
7. All 4 MCP tool annotations are mandatory on every tool — see mcp/AGENTS.md for the canonical list. OpenAI's directory and M8ven both reject tools missing hints.
8. Tests live next to code. mcp/tests/ for the server, scripts/tests/ for the generators. New code needs new tests; coverage floor is enforced by the deterministic-safety-net gate.
9. Every committed artifact is a generator output — nothing here is maintained by hand. A change to any artifact is a change to the generator that emits it, or to the canonical source that generator reads, followed by regeneration; a hand-edit is not a path, and hand-edits get overwritten on the next timer tick. Artifacts (generated, never hand-edited): datapulse.json, health/, record-evidence/, .attestations/, data/, docs/index.html, changelog.json, mcp.json, catalog-snapshot.json. Sources (authored, then consumed by generators or validators): the scripts/ themselves, config/, templates/, and the two schema files. A source is not an artifact: it is read, never regenerated in place. Two gaps against this rule are open and recorded rather than hidden — datapulse.schema.json and health.schema.json have no generator, and the manifest's dataset rows come from no generator either. Do not describe either as hand-authored; describe them as gaps.
10. Deploy posture is read-only + fail-closed. If a probe fails, mark the dataset stale or unreachable — do not silently retry until it works.

Repo map

datapulse-my/
├── README.md                  # public-facing; badges, agent-quickstart, value prop
├── datapulse.json             # 447-dataset manifest (generated)
├── datapulse.schema.json      # manifest JSON Schema (source; no generator — see rule 9)
├── health.schema.json         # health JSON Schema (source; no generator — see rule 9)
├── agent.json                 # machine-readable agent capability manifest
├── mcp.json                   # MCP server advertisement
├── llms.txt                   # LLM-friendly index of this repo
├── catalog-snapshot.json      # point-in-time snapshot (generated)
├── changelog.json             # release-by-release summary (generated)
│
├── mcp/                       # read-only FastMCP server
│   ├── server.py              # 19 tools (see mcp.json — canonical)
│   ├── tests/                 # pytest integration tests
│   ├── AGENTS.md              # <-- per-subdir working agreement
│   └── README.md
│
├── docs/                      # public website (auto-deployed)
│   ├── index.html             # dashboard (rendered by embed_dashboard_data.py)
│   ├── npra.html              # NPRA-specific page
│   ├── health-methodology.md  # methodology docs (rendered as HTML)
│   ├── mcp-reference.md       # MCP server docs (generated)
│   ├── trust-layer-notebook.ipynb  # Colab tutorial (12 cells)
│   ├── trust-layer-notebook.AGENTS.md
│   └── ...
│
├── scripts/                   # 133 Python files driving the pipeline
│   ├── gen_*.py               # generators (regenerate artifacts from sources)
│   ├── verify_*.py            # invariants (release-blocking)
│   ├── validate_*.py          # schema + content validators
│   ├── embed_dashboard_data.py  # writes the docs/ index.html
│   └── check.py               # operator smoke-test entry point
│
├── health/                    # MIXED - mostly TRACKED, partly ignored
│   │                          # TRACKED (7): latest.json, drift.json, evidence-coverage.json,
│   │                          #   history_daily.json, probe_counts.json, reconciliation.json,
│   │                          #   trends.json - and latest.json is what gen_readme.py reads
│   │                          # IGNORED: history.jsonl, as_of/, staging-signed-health/
│   ├── latest.json
│   ├── history.jsonl          # ignored
│   ├── trends.json
│   ├── drift.json
│   └── reconciliation.json
│
├── record-evidence/           # per-tool evidence logs (generated)
│
├── .attestations/             # signed probe attestation chain
│   ├── chain_head.json        # current head pointer
│   └── latest/                # daily digest envelopes
│
├── samples/                   # frozen snapshot of one row per upstream dataset
│
├── data/                      # per-dataset metadata pages (generated)
│
├── .github/workflows/         # CI: ci.yml (deterministic-safety-net),
│                              #     deploy-cloudflare-pages.yml, health-check.yml,
│                              #     release-please.yml, attest.yml
├── openwiki/                  # generated docs (do not hand-edit)
└── archive/                   # historical datasets (deprecated, kept for diff)

Script taxonomy — what each script class does

Operational commands

# Run a single pipeline stage
python3 scripts/gen_health_history.py
python3 scripts/embed_dashboard_data.py

# Run the full invariant suite (release-blocking)
python3 scripts/verify_release_reproducible.py
python3 scripts/verify_repository_contract.py
python3 scripts/verify_mcp_deployment.py

# Run the deterministic-safety-net test gate locally
python3 -m pytest scripts/tests/ mcp/tests/ -v

# Operator smoke tests
python3 scripts/check.py           # health snapshot
python3 scripts/check_heartbeat.py # recent pipeline activity

Style conventions

• Python: from __future__ import annotations on every file. Type hints required on all public functions. Absolute imports preferred. Stdlib + requests + pydantic + cryptography only — small dep tree.
• Markdown under docs/: public-facing. Date-stamped changelog entries. Citations to upstream sources where claims are made.
• Commit messages: conventional-commit-ish. The [skip deploy] trailer on chore(health) commits is a convention, not a gate: the Pages health-only fast path comes from scripts/classify_change.py classifying the changed paths, and no workflow or config reads the trailer. Without the trailer on operator commits (full release path).
• Inline comments: explain why, not what. The next agent reading should learn the operator's reasoning.

What is NOT in this repo

• The deploy VPS state — ~/.config/systemd/user/datapulse-health.timer, ~/.config/systemd/system/datapulse-health.service, env vars in /etc/systemd/system/datapulse-health.service
• Honcho observations — query via the Honcho API at http://127.0.0.1:8000/v3/workspaces/redza-prod/...
• The MCP deployment — separate code path through the deploy-cloudflare-pages.yml workflow + the headroom-proxy service
• datapulse.my Cloudflare config — DNS, TLS, cache rules (if any); owner is the operator's Cloudflare account
• Upstream data sources — data.gov.my, BNM, DOSM etc. are read but never owned

Out of scope

• Authenticated writes to the data layer (paid-product wedge lives elsewhere)
• State that requires operator secrets beyond ~/.hermes/.env and DATAPULSE_ATTESTATION_PRIVATE_KEY_FILE (the GitHub Actions secret)
• New dependencies beyond the approved list (stdlib, requests, pydantic, cryptography, fastmcp, httpx)
• Anything that disables or weakens the deterministic-safety-net gate

Pre-flight checklist (for dispatch briefs)

When writing a codex brief that touches this repo:

• Which file(s) change? path + line range
• Which scripts regenerate outputs? list them in the brief
• Does this touch docs/? add python3 scripts/embed_dashboard_data.py to the verify step
• Does this touch mcp/server.py? add python3 -m pytest mcp/tests/ -v and python3 scripts/gen_mcp_reference.py to the verify step
• Does this touch the status taxonomy? STOP and surface to operator first
• What's the deploy posture? full deploy (no trailer) or skip ([skip deploy])
• Workdir: absolute path to this repo (most common cross-repo bug — see dotfiles AGENTS.md rule 7)

OpenWiki

This repository has a generated openwiki/ evidence index. It is optional just-in-time context, not required startup reading.

• Treat source code and tests as authoritative. A brief's unknowns and review items are verification gaps, not automatic requirements.
• Prefer the narrowest quiet validation that proves the changed behavior. Preserve complete failure output.

The scheduled OpenWiki GitHub Actions workflow refreshes the repository wiki. Do not hand-edit generated OpenWiki pages unless explicitly asked; prefer updating source code/docs and letting OpenWiki regenerate.