Modal — run it detached, bounded in the cloud, and paid for on purpose

You are operating paid cloud GPUs on someone else's money. The local client (you, your shell, your laptop) can die at any moment; every guarantee must therefore live on Modal's side or in a ledger on disk, never in your memory.

0 · Resolve the skill directory

Everything this skill references sits next to this file: references/*.md and scripts/ (ledger.py, modal_task.py, selftest.py). If you need an absolute path:

MS="${ZCODE_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-${CODEX_PLUGIN_ROOT:-${MODAL_SKILL_ROOT:-}}}}"
[ -n "$MS" ] && MS="$MS/skills/modal"   # else: the directory containing this SKILL.md

Install check (CPU only, no account, no spend): python3 "$MS/scripts/selftest.py" prints ok.

1 · Golden rules (never break these)

1. Spend needs a human yes and a dollar cap. Before the first paid job of a session, get an explicit authorization with a $ cap from the account owner and record it in the ledger (ledger.py init --authorization "...").
2. Reserve before you spend, hold until the bill. Every paid job gets a worst-case reservation first (ledger.py reserve). Release it only against the job's completed-hour per-app bill (ledger.py reconcile), never on elapsed time.
3. The local client is only a launcher. Anything over a few minutes: modal run --detach under nohup, with server-side bounds: function timeout, an in-container deadline, retries=0, no warm pool. A local .get(timeout=) stops nothing in the cloud.
4. Canary first. One job alone; check setup, numerics, speed and its first bill (≤ 1.25× estimate) before fan-out.
5. Never auto-resubmit, never reuse a job id. A retry is a new id with a new reservation, resuming from state saved on the Volume.
6. Receipts come local, checkpoints stay on the Volume. Never download weights to "complete" a job.
7. Sweep before you sleep. modal app list shows nothing of yours running when you stop; stop apps by exact id, never other people's apps, never a blanket pkill modal.
8. Exactly one driver per campaign. Two agents or schedulers dispatching the same work will double-spend.
9. Check the clock before stating a time (date). Bills are hourly; you will need the real time.
10. Never put credentials in payloads, logs, commits or chat. Tokens live in env vars or ~/.modal.toml.

2 · Route by task

Read the reference fully before acting; do not work from this summary.

3 · The standard job loop

First copy scripts/ledger.py and scripts/modal_task.py from this skill into the project's scripts/ (the template finds the project root one level above itself), then:

python3 scripts/ledger.py init --ledger runs/ledger_s1.json --cap 10 --workspace-left 25 \
        --authorization "owner OK 2026-09-30, cap \$10"                       # once per authorized session
python3 scripts/ledger.py reserve    --ledger runs/ledger_s1.json --job train_01 --usd 1.90
nohup modal run --detach scripts/modal_task.py --job train_01 > runs/logs/train_01.log 2>&1 &
python3 scripts/ledger.py dispatched --ledger runs/ledger_s1.json --job train_01 --app-id ap-...   # id from the log
modal volume ls <volume> train_01                                             # done when status.json exists
modal app stop -y ap-...                                                      # confirm stopped, tasks 0
# fetch receipts only (no .pt / adapters / checkpoints), then:
python3 scripts/ledger.py complete   --ledger runs/ledger_s1.json --job train_01    # or --failed "why"
python3 scripts/ledger.py reconcile  --ledger runs/ledger_s1.json --job train_01    # after the hourly bill lands

ledger.py refuses (at $0) on: cap exceeded, workspace headroom exceeded, job-id reuse, an unreconciled failed job, too many jobs in flight, or a month rollover. A refusal is the guard working; do not route around it (never raise a cap, relabel a ledger or reuse a job id to fit). Ask the owner instead.

4 · Reporting

Report four money numbers separately: estimate, reservation, metered (workspace meter delta) and final bill (reconciled). Say which jobs still hold reservations and when their bills land. Before ending a session: 0 live apps of yours, every job reconciled or listed with its bill time, watchdog stopped, ledger closed.

5 · When you learn something new

A new failure mode → add a row to references/lessons.md (symptom → fix, with the date and what it cost). If you are in a plugin-cache copy, tell the user what to upstream to github.com/nagisanzenin/modal-skill instead of editing the cache.