Orch — Agent & Developer Notes

You are an autonomous orch agent. You should look for ways to make yourself better, make the workflow better for your agents, and learn every day.

Upgrading — OPERATOR ONLY, NEVER RUN AS AN AGENT

Agents must NEVER run any brew command. Not brew update, not brew upgrade, not brew install <anything>, not brew services restart — nothing. This applies whether or not ORCH_AGENT / TASK_ID is set, whether or not the user "seems to want" a new version, and whether or not CI just released a new tag.

Brew is a shared system-state mutation: it can pull unrelated package updates, restart the running orch service mid-tick, invalidate other tools the operator is depending on, and leave the machine in a state the operator did not consent to. Only the human operator decides when to upgrade, and the operator already knows the command.

If a new orch version is needed, say so in the issue/PR body ("this fix ships in the next release; operator to upgrade when ready") and stop there. Do not run it "just to verify". Do not run it as part of a post-push workflow. Do not run it because a previous session's notes said to. Do not print the upgrade command as a suggestion — the operator does not need to be reminded.

Do NOT file issues, PR comments, or status reports observing that the running version is behind main, that a fix "isn't live yet", that orch -V reports an older tag than HEAD, or that the operator should upgrade. The operator can see the version themselves and will upgrade when they want to. The gap between main and the installed binary is expected, deliberate, and none of the agent's business. Reports along the lines of "the service is running vX.Y.Z but main is at vX.Y.(Z+1), please upgrade" will be closed as invalid.

Restarting the service

orch service restart

Or equivalently:

brew services restart orch

Cooldowns and unblocking

orch cooldown list                  # list all active cooldowns
orch cooldown clear <key>           # clear a cooldown and that agent's degraded rate-limit window
orch cooldown clear --all           # clear all cooldowns and mark every configured agent as cleared
orch task unblock all               # unblock blocked tasks

Filing issues — NEVER mention a project orch works on internally

When filing a new issue in this repo (gabrielkoerich/orch), describe orch's behavior in generic, repo-agnostic terms. Never name any of the downstream projects orch dispatches to — not in the title, body, suggested fix, evidence section, SQL output, log snippets, or anywhere else.

This applies to:

• Owner/repo slugs of downstream projects (owner/project-name, …)
• Bare project names of downstream projects (project-name, …)
• Any absolute path — never paste raw paths into an issue. Home-directory paths (/Users/<name>/..., /home/<name>/...) leak the operator's username, and even non-home paths often embed a downstream project name or other operational detail. Use a tilde-rooted placeholder form (~/.orch/worktrees/<project>/<branch>/) or describe the location in words ("the agent's worktree", "the orch state dir").
• Evidence tables / SQL output that leak a repo column (or any column that names a project / embeds a path) — redact before pasting (use <repo>, <path>, or drop the column)
• Log lines that name a downstream project or contain absolute paths — paraphrase before quoting

If you need to describe a reproduction, abstract it: "a downstream project", "the repo that owns the failing task", "the affected repo".

Filing an issue triggered by something you observed in a downstream project is fine — the rule is about the name, not the source. Strip the identifying name, keep the behavior. "Tasks for a downstream repo block at merge time when GitHub Actions billing fails — is the per-task block working as intended?" is a fine issue. "Tasks for owner/project block at merge time…" is not.

Why: orch is a generic orchestrator. Issues are the long-term spec of how it should behave. Naming a specific downstream project couples the bug report to that project's transient state, leaks the operator's internal project list into a shared tracker, and produces issues that future agents misinterpret as "this bug only happens for project X" when the underlying behavior is generic.

Cautionary tale — issue #3339 named a specific downstream project throughout the body and SQL output while describing behavior that was actually working as intended per the settled architecture (per-task block at merge time on GitHub Actions billing failure). It was deleted.

Commit only the change

Never commit a report, notes or summary file about your own work: no REVIEW.md, CHANGES.md, NOTES.md, SUMMARY.md or the like, at the root or anywhere else. The summary of a change goes in the PR body, and lasting decisions go in docs/. Two such files sat at the repository root for months after the tasks that wrote them were done. Before committing, check git status for files you created only to describe your work, and delete them.

Manual issue closure cleanup

IMPORTANT: When you manually close a GitHub issue, agents may still be working on it — tmux sessions and worktrees must be cleaned up or they'll interfere with future work.

Stale sessions to check for

When closing issue #NNNN, look for:

• Agent sessions: orch-orch-NNNN-* (e.g., orch-orch-1455-review)
• Chat sessions: orch-chat-* (leftover from debugging)
• Review sessions: orch-orch-*-review

Cleanup steps

# 1. List all orch tmux sessions
tmux list-sessions | grep orch

# 2. Kill the specific session(s)
tmux kill-session -t orch-orch-NNNN-review
tmux kill-session -t orch-chat-default

# 3. Remove associated worktrees
ls ~/.orch/worktrees/orch/ | grep -i "NNNN\|issue"
rm -rf ~/.orch/worktrees/orch/gh-issue-NNNN-*

Do not leave stale sessions/worktrees — they consume resources and can interfere with new work on the same issue.

Logs

• Service log: ~/.orch/state/orch.log
• Brew stdout: /opt/homebrew/var/log/orch.log (startup messages only)
• Brew stderr: /opt/homebrew/var/log/orch.error.log
• If you find any errors on /opt/homebrew/var/log/orch.error.log, first run ls -lh /opt/homebrew/var/log/orch.error.log to check the file modification date. This file is truncated on every service restart, so any content in it is from the current run only. DO NOT refile issues based on this file if its mtime predates the current session or is more than a few minutes old.

Live Session Streaming

orch stream              # stream ALL running sessions (auto-discovers new ones)
orch stream <task_id>    # stream a single task

Implementation: src/channels/capture.rs — diffs tmux pane output every 2 s and broadcasts chunks to subscribers (CLI, Telegram, Discord). Diffing prevents duplicates across multiple clients on the same session.

Discord Gateway (Websocket)

The Discord channel uses the Discord Gateway websocket (wss://gateway.discord.gg) for real-time MESSAGE_CREATE events. Implementation: src/channels/discord/.

Bot token must have GUILDS, GUILD_MESSAGES, and the privileged MESSAGE_CONTENT intent (enable under Privileged Gateway Intents in the Discord Developer Portal).

channels:
  discord:
    bot_token: "${DISCORD_BOT_TOKEN}"
    channel_id: "..."          # optional: restrict to a single channel
    shard_id: 0                # only needed for bots in 2 500+ guilds
    shard_count: 1

Reconnect uses resume_gateway_url + session_id + last seq; falls back to re-identify on invalid session. Backoff: 1 s → 60 s.

Webhooks & Polling Fallback

Polling runs sync_tick() every engine.sync_interval (45 s) in every mode. A webhook delivery or an event from the relay client starts a tick and a background sync at once. The local webhook server's /health is checked every engine.webhook_health_check_interval seconds, and a change logs webhook health check failed or webhook health restored. The sync interval does not change when the webhook is down.

webhook:
  enabled: true
  port: 8080
  secret: "${WEBHOOK_SECRET}"

engine:
  tick_interval: 10
  sync_interval: 45
  webhook_health_check_interval: 60

See also ## DO NOT TOUCH → No external endpoints. In practice polling is the default — webhooks rarely work without a public endpoint.

The relay client (src/engine/relay.rs) connects out to orch-relay at <relay.url>/ws, which receives the GitHub App's webhooks, so it needs no inbound endpoint. It keeps a cursor in the KV store, reconnects with a backoff up to 5 minutes that only a session of a minute or more resets, and is off when relay.url is unset. relay.token takes a passbox:<name> reference.

PR Review Integration

When a PR review on an in_review task returns CHANGES_REQUESTED, the engine stores feedback in pr_review_context, increments review_cycles, re-routes the task back to Routed (reuses existing agent/model/worktree/branch — skips LLM re-classification), and the agent commits fixes to the same PR.

A review starts only after CI has run on the PR head (src/engine/review_gate.rs). While checks run, the task waits in needs_review. Red CI sends it back to routed with the failed checks in last_error and counts toward ci_merge_failures, once per head SHA. The same red head stays in needs_review. With no CI configured, or no check started after 15 minutes, the review starts at once. When no check started and the repository uses shipcheck, orch runs shipcheck run in the task worktree first and holds the review until it ends (src/engine/review_shipcheck.rs), so a long suite runs outside the review's time limit (#3943). Before the CI gate, orch reads the PR mergeability (prepare_branch_for_review in src/engine/auto_merge.rs). A branch that is behind or conflicting is rebased in the task worktree and the review waits for CI on the new head. A conflict the rebase cannot solve routes the task back to its author, who has the context of the change, and counts toward merge_conflict_retries (#3973). The reviewer never rebases. The review-gate job and the orch/review check never hold a review. An approving review moves the task to approved, in comment and app review mode alike. Task becomes blocked when review_cycles >= max_review_cycles, and done only when the PR is actually merged.

workflow:
  max_review_cycles: 2

Complexity-based model routing

The router assigns complexity: simple|medium|complex instead of specific model names. The actual model is resolved per agent from config.yml:

model_map:
  simple:
    claude: haiku
    codex: gpt-5.1-codex-mini
  medium:
    claude: sonnet
    codex: gpt-5.2
  complex:
    claude: opus
    codex: gpt-5.3-codex
  review:
    claude: sonnet
    codex: gpt-5.2

See model_for_complexity() in the router module.

Router Module (Rust)

Implementation: src/engine/router/ (selects agent + model per task). Prompt: prompts/route.md. Structs: RouteResult, AgentProfile in source.

Priority order: label override → round-robin mode → LLM classification → router.fallback_executor. Configured under the router: key in ~/.orch/config.yml.

Label-Based Routing

Environment Variables

The runner passes routing results to the agent via ORCH_AGENT (selected agent) and ORCH_MODEL (specific model).

Directory layout

~/.orch/
  config.yml             # global config
  orch.db                # SQLite database (tasks, metrics, KV store, rate limits)
  projects/              # bare clones added via `orch project add`
    owner/repo.git       #   each has .orch.yml inside
  worktrees/             # agent worktrees (all projects)
    repo/branch/         #   created by the runner, one per task
  state/                 # runtime state (logs, prompts, per-task artifacts)
    control/{pid}/       #   control session temp files (per-process isolation)
  skills/                # cloned skill repositories

• User-managed projects (e.g. ~/Projects/foo): user clones, runs orch init. Project dir stays where the user put it.
• Orch-managed projects (orch project add owner/repo): bare clone at ~/.orch/projects/<owner>/<repo>.git.
• Worktrees: always at ~/.orch/worktrees/<project>/<branch>/ regardless of project type.
• ORCH_WORKTREES env var overrides the worktrees base directory.

Required checks before every commit

Always run these before committing — CI enforces them and will fail otherwise:

cargo fmt -- --check                    # formatting
cargo clippy --all-targets -- -D warnings # lints (warnings are errors, incl. test code)
cargo nextest run                       # unit tests (faster, matches CI)
DYLINT_RUSTFLAGS="-D warnings" cargo dylint --all -- --all-targets # use_in_fn lint

Or all at once:

cargo fmt && cargo clippy --all-targets -- -D warnings && cargo nextest run

Installing dylint locally: cargo +stable install cargo-dylint dylint-link --version 6.3.2 --locked. The +stable matters, the pinned 1.94.1 is too old to build cargo-dylint. The first run installs the nightly that the lint pins in its repository.

Installing nextest locally: CI uses prebuilt binaries via taiki-e/install-action. For local use, install with cargo binstall cargo-nextest (requires cargo-binstall), or fall back to cargo test if nextest is unavailable.

Imports at the top of each module

Every import is a use at the top of its module, after the mod declarations. Lints enforce it, and CI fails on a break:

When two imports want the same name, import the parent module (fmt::Result, io::Error) or rename one (use std::time as std_time). Import an item used only under a cfg with the same cfg, or the other target warns that the import is unused.

Release pipeline

1. Push to main
2. CI runs tests, picks the semver bump from conventional commits and builds the macOS and Linux binaries
3. The release job creates the tag, builds the changelog, creates the GitHub release, publishes the docs, updates the Homebrew tap formula and publishes to crates.io. It pushes the version bump commit ([skip ci]) to main last
4. brew upgrade orch picks up the new version
5. orch service restart to load new code

Do NOT manually edit the tap formula — the CI pipeline handles it. The Formula/orch.rb in this repo is a local reference copy, not the real tap.

Post-push workflow

After pushing to main, agents do these steps and stop:

git push                                    # 1. push
gh run watch --exit-status                  # 2. watch CI (tests → release → deploy)

Do NOT run brew update, brew upgrade orch, brew services restart, or orch -V afterwards. The operator handles the install + service restart on their own schedule (see "Upgrading — OPERATOR ONLY"). The service runs from the Homebrew cellar, not the repo, so your changes are not live until the operator upgrades — that is intentional and not your problem to fix.

Graceful Shutdown

On SIGTERM (e.g. orch service restart or brew services restart orch), the engine:

1. Resets all in_progress tasks → routed so they re-dispatch into existing worktrees on next start
2. Resets all in_review tasks → needs_review so review agents are re-spawned on next start (review agent tmux sessions are killed when the process exits)
3. Kills all orch-* tmux sessions to release resources
4. Waits up to engine.graceful_shutdown_timeout seconds (default: 600) for in-flight work to settle

No work is lost — tasks resume from their worktrees on restart.

Task status semantics

• new — just created, awaiting routing
• routed — agent and model selected, awaiting dispatch
• in_progress — agent is working on the task in a tmux session
• needs_review — agent completed, awaiting review agent dispatch (automatic, not human)
• in_review — review agent is actively reviewing the PR
• approved — the review approved the PR. orch waits for CI and merges with no agent or session, merged → done, conflict or red CI → routed, PR closed unmerged → blocked. A branch that falls behind is rebased onto its base in the task worktree (GitHub merges the base in when that fails), and the approval moves to the new head because only the base changed. Restarts and stuck-review recovery never reset it
• done — PR merged, worktree cleaned up. Terminal: nothing moves a task out of done. retry, unblock and the /retry, /reroute, /unblock channel commands refuse a done task, doctor --fix reports a suspicious done task and leaves its status alone, and orch task reopen no longer exists. A revived task re-runs the body it was created with, which may be months stale — follow-up work gets a new issue or task (#3780)
• blocked — requires human attention (max review cycles, max attempts, agent failures, unrecoverable errors)
• mark_needs_review() sets needs_review, NOT blocked
• Only parent tasks waiting on children should be blocked
• Engine auto-unblocks parent tasks when all children are done (Phase 4 of tick)

DO NOT TOUCH — Settled architecture decisions

These areas have been deliberately designed and must not be changed without an explicit human decision. Do not file issues, refactor, or "improve" them.

Brew / upgrades / installed-version reports are operator-only

The rules in ## Upgrading — OPERATOR ONLY, NEVER RUN AS AN AGENT and ### Post-push workflow are a settled architectural decision, not a preference to be revisited. Do not modify, soften, or "clarify" those sections. Do not add exceptions ("unless the operator asked", "unless in a fresh shell", "unless in CI"). Do not re-introduce the upgrade command, the orch -V verification step, or the multi-step post-push cycle. Do not file issues proposing an "auto-upgrade after push" feature, a "version drift warning", a check_installed_version() helper, or any mechanism that observes / reports / acts on the gap between main and the installed binary.

The operator sees the version themselves and upgrades on their own schedule. brew mutates shared system state that is out of orch's scope. Any PR that touches these two sections or adds version-drift tooling will be reverted.

GitHub token resolution (src/github/token.rs)

The auth flow is: GH_TOKEN env → GITHUB_TOKEN env → gh.auth.token config → gh auth token CLI.

• try_gh_command() uses std::process::Command (blocking). This is intentional. The result is cached until a 401 clears it, so this runs once at startup and again after each 401. It is not a Tokio antipattern in this context.
• TokenResolver is a process-wide singleton via shared(). Do not add a second resolver, re-export the token, or set env vars.
• Do not add spawn_blocking, tokio::process::Command, or orch auth check wiring here.
• Issues #418 and #421 were filed by agents making this exact mistake and were closed as invalid.

If you see std::process::Command in try_gh_command — leave it alone.

PTY runner (src/engine/runner/)

The agent runner uses tmux as the PTY (tmux already provides a PTY to whatever runs inside it). There is no portable_pty crate and no external PTY process. Do not reintroduce PTY-based runners that spawn the agent outside tmux and forward output via send-keys. Issue #416 explains why this was removed.

Config files are off-limits

Agents must NEVER modify ~/.orch/config.yml, config.example.yml, or any .orch.yml project config. These files control the running service and model routing — changes have immediate, global impact.

If a config change is needed, describe the required change in the issue body or PR description. The human operator will apply it. Do not write config changes directly, do not include config file edits in PRs, and do not suggest "just update config.yml" as a fix.

Issues #1261 and #1172 were caused by agents modifying config files without permission.

Cooldown and failure recovery is generic — do not special-case models or agents

The cooldown system (src/engine/cooldown.rs) and failure recovery (src/engine/sync.rs) are designed to handle all agent and model failures generically.

All cooldowns are persisted to SQLite KV (cooldown:{key}) so they survive service restarts. Failure counts are stored separately (failure_count:{key}) to drive exponential backoff.

• Agent failures → exponential backoff starting at 5 min, capping at 4h → router picks a different agent
• Model failures → exponential backoff starting at 5 min, capping at 4h → router picks a different model
• One-off model failures (empty or cut off output, an unparseable review, an opaque provider error) → record_transient_model_failure(), 5 min then 15 min. A stopped session looks the same, so the backoff stays short
• A review that times out → no cooldown, the task goes back to needs_review and the next review picks an agent as usual
• Rate limits with "try again at" → cooldown set to that vendor-specified timestamp (always authoritative)
• Silence detection → model cooldown + short 120s agent cooldown → re-route
• Credit exhaustion (out_of_credits) → exponential from 1h, capping at 8h
• Org-level disabling (org_level_disabled) → exponential from 2h, capping at 8h
• Billing cycle exhaustion → when the failing model is known: persistent model-level cooldown (4h base → 7d cap); when no model is identified: escalating agent-wide cooldown from 24h, capping at 7 days. Model-scoped exhaustion (e.g. a provider sub-model like github-copilot/gpt-5-mini) must not block unrelated models on the same agent.
• On successful completion → failure counts reset via record_agent_success() so next failure starts from base again

Exception for pre-emptive routability checks: The router performs proactive checks to skip agents/models that are likely to fail based on routing weight decay and cooldown states. This is not considered special-casing because it uses the same generic cooldown system and weight decay mechanisms that feed into the routing decision process.

Do not file issues to add special handling for specific models or agents (e.g., "copilot models need longer cooldowns", "add fallback for kimi rate limits"). If a model silently fails, the existing silence detection + cooldown handles it. If a model is rate-limited, parse_retry_at handles it. If all models for an agent are cooled, the router picks a different agent.

Issue #1286 was closed as invalid — it proposed special-casing copilot model failures when the generic system already handles them.

If you believe the generic system has a bug (e.g., cooldowns not being applied, silence not detected), file an issue about the generic mechanism, not about a specific model.

The exponential backoff approach is settled — do not replace it with flat cooldowns or per-agent special cases. The formula min(base * 3^(attempt-1), max) is in compute_backoff(). Failure counts are persisted in failure_count:{key} KV keys. Incremental improvements (e.g. tuning constants, adding decay windows, improving reset logic) are welcome as proposals, but the base mechanism must not be reverted.

NEVER hardcode model names or aliases in router code

There must be zero literal model-identifier strings in the routing/dispatch code paths ("github-copilot/gpt-5.3", "gpt-5.3-codex", "opencode/...", etc.). No match arms keyed on specific models, no "known unavailable" allow/deny lists, no canonicalization tables that rewrite alias A → alias B, no per-model if agent == "X" && model == "Y" branches anywhere in src/engine/router/, src/engine/runner/, or src/engine/cooldown.rs.

When a model fails (including Model not found / ProviderModelNotFoundError / "model unavailable"), the only correct response is:

1. Runner classifies the error as AgentError::ModelUnavailable.
2. cooldown::record_model_unavailable(agent, model, message) cools that specific model. A message that says the model is gone ("not found", "not supported", "not available in your country" or "not available in your region") gets the persistent cooldown (4h base → 7d max) through record_persistent_model_failure. Any other message gets the generic record_model_failure backoff (5m base → 4h max). The runner and the review agent both use this function.
3. The router's next selection skips that model via is_model_in_cooldown() and picks the next entry — or fails over to a different agent if none remain.
4. The agent itself is not penalized — its other models still route normally.

There is no list to update, no alias to add, no is_known_unavailable_model() to extend. If the cooldown clears too early, fix the duration. If the wrong error variant comes back, fix the parser. If the model keeps coming back from config, that's a config problem (see "Config files are off-limits").

Cautionary tales — PRs #3020 and #3040 added canonicalize_model_alias() / is_known_unavailable_model() with hardcoded "github-copilot/gpt-5.3" → "gpt-5.3-codex" rewrites. Both reverted. Issue #3051 proposed adding yet another model name to the same hardcoded list — closed as invalid.

If you are tempted to add a model name to a match or Vec<&str> of "bad models", stop: the generic per-model cooldown handles it the moment dispatch fails, and any list rots the next time a model is renamed. If record_persistent_model_failure isn't firing for your scenario, file an issue about the classifier or the cooldown duration — never about adding a specific model name to a filter.

Pre-emptive Routability and Circuit-Breaker Behavior

The router implements two optimizations to prevent unnecessary agent invocations:

1. Pre-emptive routability check — before routing, evaluate each agent/model's routing weight; skip combinations whose weight has decayed below a configurable threshold.
2. Circuit-breaker behavior — out_of_credits and org_level_disabled trigger extended cooldowns (typically 2-4x standard durations) to prevent repeated futile attempts.

Both reuse the generic cooldown system as the single source of truth.

Integration points (src/engine/router/mod.rs):

• router.refresh_health(&store) — each tick; delegates to cooldown::refresh_degraded_agents() (queries rate_limits, updates degraded set).
• router.agent_is_routable(agent, complexity) — guards routing; false when is_agent_in_cooldown / is_agent_degraded; with weighted_round_robin, also skips agents whose AgentWeights::get_weight is below the fixed threshold SKIP_LIMITED_THRESHOLD (0.3, not configurable).
• router.available_agents_for_complexity(complexity) — filters through agent_is_routable; used by round-robin, weighted, and LLM paths.
• cooldown::record_credit_exhaustion(agent, reason) — exponential agent-wide cooldown (1h→8h for out_of_credits, 2h→8h for org_level_disabled, 24h→7d for billing_cycle_exhausted).
• cooldown::record_agent_success(agent, model) — runner calls on success; resets failure_count:* so next failure restarts from base.

These checks happen before LLM-based routing.

Migrations are immutable — NEVER modify existing migration files

Once a migration file has been applied to any database, its checksum is locked by SQLx. Modifying an existing migration file changes the checksum, causing sqlx::migrate!() to fail on every startup — breaking the service completely.

Always create a NEW migration file (e.g., 014_new_column.sql) instead of editing an existing one. Issue d02cdda modified 012_auto_unblock.sql and broke the service for 70+ minutes.

No external endpoints

Orch is an internal tool running on a local machine with no external network access. There are no publicly reachable HTTP/webhook endpoints. The webhook receiver in the config exists but only works when the machine happens to be reachable (rare). GitHub polling fallback is the default mode.

External consumers (CLI, local debugging tools) connect via localhost-only websocket (127.0.0.1). Do not add externally-reachable servers, do not assume inbound connections from GitHub or other services will work, and do not design features that depend on webhook delivery.

Routing concurrency is controlled by max_tasks_per_tick — no semaphore, no wall-clock budget

LLM routing concurrency is governed by exactly one knob:

• router.max_tasks_per_tick (default: 1) — caps how many tasks enter the routing loop per tick. With the default of 1, only one LLM classification call can ever be in flight at a time.

Per-call timeouts are already enforced by router.timeout_seconds. If a route call hangs longer than that, it errors and the cascade moves on. There is no separate wall-clock "budget" wrapped around the cascade.

Do not add a semaphore, worker pool, ORCH_ROUTER_MAX_PARALLEL_LLMS env var, or llm_budget_secs field. Issue #2676 / PR #2677 introduced an llm_semaphore field on Router that was redundant with max_tasks_per_tick=1 and added no protection that the per-call timeout doesn't already provide. It was removed as dead complexity. A subsequent llm_budget_secs wall-clock wrapper was removed for the same reason (see "No 'budget' features" below).

If routing is too slow: lower router.timeout_seconds, or reduce router.max_tasks_per_tick. Do not add a third concurrency mechanism.

No per-task routing defer — AllAgentsCooledError must retry next tick, not write a timer

There must be zero per-task deferral state in the routing path. No route_defer_until:* KV keys, no route_defer_* helpers, no per-task timer under any other name (route_backoff_until, next_route_attempt_at, cooled_skip_until, etc.).

When router.route() returns AllAgentsCooledError, the only correct response in tick_route_tasks is:

1. tracing::error!(...) — loud, every tick.
2. continue — task stays in new.
3. Next tick (~10s later), routing tries again.

src/engine/cooldown.rs is the single source of truth for "is this agent/model available right now." available_agents_for_complexity() filters cooled agents before any LLM call, so when everything is cooled, router.route() returns AllAgentsCooledError in microseconds — no tokens, no network. There is nothing to "defer" away from.

Why a parallel timer is wrong: the defer key was a wall-clock value (now + remaining_secs / 2) written into SQLite. It counted down independently of the real cooldowns. Long cooldowns (billing-cycle / out_of_credits at 24h → 7d) produced per-task defers of 12h → 3.5d. When agents became available again, tasks stayed stranded in new because the unrelated future timestamp hadn't passed. PR #3243 removed it after a live incident — orch cooldown list reported No active cooldowns. while 9 route_defer_until:* KV keys held tasks for hours.

If the ERROR log is noisy: the volume is the feature. It tells the operator that all agents are unavailable right now, every 10s, until the situation resolves. Suppressing it with a per-task timer is how the bug came back. If you genuinely need to throttle: tune engine.tick_interval, lower router.max_tasks_per_tick, or fix the upstream cooldown duration — never write a parallel timer.

Do not file issues like "add backoff for AllAgentsCooled", "rate-limit the route_attempts log spam", "remember the last-cooled time per task", or "skip routing for N seconds after an all-cooled event." Closed as invalid.

GitHub Actions billing failures block the single task, never a whole repo

When a PR's CI cannot run because of a GitHub Actions billing problem (payment failure, spending limit, plan downgrade), the only correct response is to block the current task at merge time with block_reason = "GitHub Actions billing failure …". The agent still routes, dispatches, does the work, opens the PR; the review agent still reviews. Merging is what's blocked, not the work.

There must be zero "skip the whole repo" mechanism: no repo_billing_failure:* KV markers, no set_repo_billing_failure() / is_repo_billing_blocked() helpers, no pre-dispatch check in tick_route_tasks, no TTL on a repo-wide flag, no per-repo allow/deny list under any name (repo_ci_blocked, actions_disabled_until, etc.).

Why: blocking pre-dispatch confuses three things:

1. Discovering the billing failure is cheap — is_billing_failure() (src/engine/auto_merge.rs) already catches it at merge time.
2. Doing the work is still valuable — the diff, the review, the agent's reasoning are artifacts the operator wants regardless of whether CI is currently paid up.
3. A repo-wide marker has no way to know when CI is fixed — orch can't poll GitHub billing. A TTL is arbitrary; clearing on orch task unblock couples unrelated concerns. The marker either over-blocks or under-blocks. The natural signal is the merge attempt — let it fire per-task, block per-task.

Correct path: is_billing_failure() returns true during auto_merge_pr → mark that one task Blocked with the billing block_reason and return Ok(()). Other tasks for the same repo keep routing/dispatching/opening PRs/getting reviewed. Operator pays the bill → orch task unblock all → blocked tasks return to new → next merge attempt succeeds.

Cautionary tale — PR #3334 introduced repo_billing_failure:{repo} markers with a 24h TTL and a tick_route_tasks pre-check that transitioned new tasks straight to Blocked without running the router. The agent never wrote code, the review agent never reviewed, and the operator lost a full day of throughput per billing hiccup — for a class of failure already caught one layer down at the right granularity. Reverted.

Do not file issues like "skip dispatch when CI is broken", "save agent compute when GitHub billing fails", "add a 24h TTL marker", or "preemptively block tasks for repos whose last merge failed on infra". Closed as invalid. The per-task block at merge time is the boundary; do not move it earlier.

No 'budget' features — agents run on subscription / fixed pricing

There must be zero "budget" tracking, accounting, or enforcement anywhere in orch. No per-task token budgets, no per-PR budget warnings, no routing wall-clock budgets, no budget_warning / budget_exceeded columns, no check_token_budget(), no TokenBudgetExceeded failure category, no BudgetCheckOutcome enum, no "budget exhausted" cooldown reason, no llm_budget_secs / llm_bypass_* knobs.

Why: all agents orch dispatches to (claude, codex, opencode, kimi, minimax, copilot, …) run on subscription or fixed-price plans. There is no per-token billing to throttle against. Counting tokens to "save money" produces no real signal and just adds heavy bookkeeping (extra SQLite columns, extra runner branches, extra PR comments) that breaks routinely.

The correct mechanisms already exist:

• Hung / silent agents → silence detection + agent cooldown
• Slow router calls → router.timeout_seconds (per-call timeout)
• Tick stalls → watchdog
• Runaway retries → max_route_attempts, max_review_cycles, per-task attempt counters
• Rate limits / out of credits → existing cooldown variants (record_rate_limit, record_credit_exhaustion, etc.)

Migration 027_drop_budget_columns.sql removed the budget columns. They stay removed. Do not re-add any of the above under a new name (max_token_spend, cost_cap, route_deadline, etc.) — if you're reaching for a knob that counts tokens or wall-clock seconds against a quota, stop: the existing cooldown / timeout / watchdog mechanisms already cover the legitimate cases.

Do not file issues like "add cost tracking", "warn when task exceeds N tokens", "fall back to round-robin after Ns of LLM time", or "block tasks that would burn too many tokens". Closed as invalid. This is not a per-call-priced system.

If a comment says "retry budget" / "escalation budget", it refers to a count of attempts (a retry quota), not money or tokens. Prefer "quota" or "limit" to avoid tempting future agents into re-introducing the feature.

Model names and prices live in the config, not in code

orch's code names no models and holds no price table. A model name in code goes stale the day a provider renames it. A price table is always behind, and it prices subscription runs at list API rates.

• Model names live in the config: models:, model_map: and router.model. With no entry, orch passes no --model and the agent's CLI uses its own default. The control session's default model is the first entry under models.<agent>, and /model <name> picks the agent whose models: entry lists that name.
• Cost is what the agent reports for its own run, or nothing. The parsers read it into AgentResult.cost_usd and ParsedResponse.cost_usd (total_cost_usd for Claude Code and grok, step_finish.cost for opencode). store_tokens adds it to total_cost_usd. A run that reports no cost adds its tokens only, and the CLI shows "not reported". orch never estimates a cost.
• input_cost_usd and output_cost_usd stay in the schema unused, because no agent reports the split.

tests/no_model_names.rs scans src/ outside #[cfg(test)] items and test files for string literals that hold a model name from config.example.yml, and fails with the file and line. Do not add an allow list to make it pass. Move the name to the config. Tests may name models.

#3997 removed src/store/pricing.rs. Do not bring back a price table, in code or in a data file.

Security leak detection — ALL patterns block GitHub posting

has_leaks() in src/security.rs checks all LEAK_PATTERNS regardless of the high_confidence flag. Nothing is posted to GitHub if any pattern matches — low-confidence patterns included.

Do not change has_leaks() to filter by high_confidence. Issue #2645 / PR #2648 made this mistake: it changed has_leaks() to skip low-confidence patterns to reduce "false positive redactions", but the correct policy is to err on the side of caution — if anything looks like a secret, don't post it. A false positive (overly cautious) is far preferable to a false negative (leaking credentials).

The high_confidence flag exists only to let scan() / redact() distinguish severity for display purposes — it has no effect on whether posting is blocked.

Never delete the only copy of a commit

delete_branches() in src/engine/cleanup.rs is the one place that deletes task branches, and it refuses to destroy work. The remote branch goes only when the default branch already contains every commit on it. The local branch goes only when origin or the default branch holds every commit on it. A squash-merged branch stays, and GitHub's "automatically delete head branches" removes it.

Do not add a git push origin --delete, a git branch -D or a forced git worktree remove anywhere else, and do not weaken that check. Cleanup runs from many callers (startup reconciliation, closed-issue reconciliation, orphan sweeps, re-dispatch after store errors). Any of them can hold unmerged work, and the caller's view of the task status is not enough to know. On 2026-10-26, startup cleanup treated a blocked task as finished and deleted its remote branch, the only copy of pushed work (#3781). The tests delete_branches_keeps_an_unmerged_remote_branch and delete_branches_keeps_a_local_branch_with_unpushed_commits guard this rule.

Blocked and waiting are not terminal

blocked means a person has to look, and waiting means a dependency is open. Both resume from their worktree and branch. Only done is terminal. Cleanup must never treat Blocked or Waiting as finished: startup reconciliation keeps their worktrees, and periodic cleanup collects only done tasks (#3793). cleanup_skips_blocked_internal_tasks_and_cleans_done_ones guards this.

Age stuck tasks from their status entry, not updated_at

Many writes touch updated_at without changing the status. review_open_prs writes pr_number on every pass, so an in_review task with a PR never looked older than a minute, and a task with no reviewer sat for 10 hours (#3791). Stuck and stale checks must age a task from the task_activity row where it entered its current status. Do not move them back to updated_at. stale_in_review_recovery_resets_orphaned_task_without_session guards this.

An in_progress task with a live session ages from its in_progress entry, and its limit is engine.stuck_timeout raised to the run's own timeout (workflow.timeout_by_complexity) plus RUN_TIMEOUT_GRACE_SECS. A fixed 30 minute limit reset complex runs that had an hour, while the agent still worked and committed (#3978). A dead session still ages from updated_at, which the session reaper touches when it sees the session end. Recovery kills a live session before the reset. recover_stuck_tasks_keeps_a_live_complex_run_inside_its_timeout and recover_stuck_tasks_kills_a_live_complex_run_past_its_timeout guard this.

One periodic owner for an orphaned review

tick_recover_stuck_tasks is the only periodic path that resets an in_review task whose review session is gone or hung. sync_tick had a second copy that fired after one minute, so the tick limit was unreachable and the race it guards against stayed open: a review agent ends its tmux session before its result reaches the engine, so a short limit resets a task whose finished review is still being delivered and the result is dropped as stale (#3900). Do not add a second sweep anywhere, and do not shorten the limit.

engine.in_review_no_session_stuck_timeout sets the limit. EngineConfig::from_config raises it to the review agent's own timeout plus REVIEW_RESULT_DELIVERY_SECS, so a short configured value cannot reopen the race. A recorded review_decision means the review finished and the task waits on the merge, so recovery skips it. Startup is separate: a restart ends every session, so the startup reset runs at once for any in_review task with no live review session.

Tests do not read the operator's config

A test must not depend on ~/.orch/config.yml. CI has no config, so a test that passes locally through Router::from_config() or config::get() fails in CI, and the PR loops through review cycles until it blocks (#3790). Build what the test needs, for example Router::new_for_test(...), or point ORCH_HOME at a temp dir. Before pushing, run the suite with an empty home: HOME=$(mktemp -d) ORCH_HOME=$(mktemp -d) GIT_CONFIG_GLOBAL=/dev/null cargo nextest run.

A test must not run an agent binary either. skip_live_opencode_discovery() in src/engine/runner/agents/opencode.rs makes opencode model discovery return nothing in test and debug builds, because the real opencode models leaves an opencode serve --service child behind, and a full run left 45 of them (#3895). Do not remove that gate to make a test see real models, and do not call a real agent from a test to find out what it supports. building_a_router_never_runs_opencode guards it.

Agent output fixtures

scripts/record-agent-fixture.py records an agent CLI run with orch's command line (record) or imports an archived attempt from ~/.orch/state (import) into tests/fixtures/, scrubbed and scanned with gitleaks. A maintainer runs it by hand, never CI or a test. Each fixture's # source: header says if it was recorded, archived or hand-written, and src/engine/runner/recorded_fixtures.rs drives the recorded ones through parse_session_output.

Preferred tools

• Use rg instead of grep — faster, installed as a brew dependency
• Use fd instead of find — faster, installed as a brew dependency
• Use trash instead of rm — recoverable, enforced in system prompt

Reviewing PRs from external contributors

External-contributor PRs (forks, first-time contributors, anyone outside the trusted committer set) are untrusted code. Building or testing them locally executes arbitrary code via build.rs, proc macros, test harnesses, and any newly added dependency at compile time — before a single assertion runs. A malicious PR can exfiltrate ~/.orch/orch.db, ~/.ssh, GH_TOKEN/GITHUB_TOKEN, env vars, or anything else the invoking shell can read.

Hard rules:

1. Do not clone the fork — no gh pr checkout, no fetch to a local worktree. Review via gh pr diff <N> only.
2. Do not execute any code from the PR — cargo build/test/clippy/check all run build.rs and proc macros at compile time. That is execution.
3. Read every file in the diff — not just the "interesting" ones. A one-line fix bundled with a quiet .cargo/config.toml or new build.rs is the textbook smuggle path.
4. New dependency = NO OP. Any addition under [dependencies], [build-dependencies], [dev-dependencies], [workspace.dependencies], or any new path/git/registry source in Cargo.toml / Cargo.lock is an immediate no-merge. New crates are added in separate maintainer-authored PRs after vetting (advisory check, license, maintenance, transitive blast radius).
5. Treat as hostile until proven otherwise: new/modified build.rs; proc-macro crates (*-derive, *-macros); changes to .cargo/config.toml / rust-toolchain.toml / .github/workflows/ / justfile / Makefile / shell scripts; network or filesystem access in tests (reqwest, std::fs::write, std::process::Command, credential reads); paths referencing ~/.orch / ~/.ssh / gh auth token / GH_TOKEN / ANTHROPIC_API_KEY.

Use just review-pr <N> for the standard workflow. It runs scripts/review/hooks-check.sh against the diff (refuses if any tripwire fires — Cargo manifest, build.rs, hooks, CI workflows, shell scripts), then spins up two short-lived Docker containers:

• Stage A (fetch) — network ON, no compilation. git clone --depth=50 --branch <ref> into a volume, cargo fetch --locked. build.rs does not run.
• Stage B (run) — --network=none, no host bind mounts, no host credentials. cargo nextest run --offline. Untrusted code first executes here.

just hermetic-tests runs the current worktree through the same stage-B container — catches tests that silently depend on network, tmux on PATH, non-root file-mode semantics.

Fallback paths when just review-pr is not an option:

1. Re-implement in a maintainer-authored branch. Recreate the diff yourself, push to this repo, let CI run there. The fork's bytes never touch a trusted machine.

2. Clone and execute inside Docker. Fresh container from a clean base (e.g. rust:slim). No bind mounts of host paths. No host credentials in env (no -e GH_TOKEN, no --env-file). --network=none once Cargo fetch is done. Clone the fork inside the container, then cargo nextest run. docker rm -f after. Skeleton:

   docker run --rm -it --network=none --read-only --tmpfs /tmp --tmpfs /work rust:slim bash
   # inside: install git, clone fork, checkout PR branch, cargo nextest run

   Network must be on briefly for the clone + Cargo fetch, then drop it before build. The container is the trust boundary — if you mount ~/.orch so a test can find the DB, you don't have a sandbox.

Sandboxing is not a substitute for static review. A clean cargo nextest run proves nothing — by then build.rs has already run. Read every file in the diff first.

CI does not run on PRs by design. All workflows in .github/workflows/ trigger on push: only — no pull_request: (and never pull_request_target:, which runs the workflow definition from the base branch but checks out the PR head with full secrets access — the classic supply-chain attack vector). PRs from forks do not auto-execute Actions. If CI verification on a fork is needed, a maintainer pulls the change into a branch on this repo after static review.

Do not add pull_request: back, not even to get GitHub's "Approve and run" button for first-time contributors. Commit 4cb7b176 did that and was undone for these reasons:

• Orch's PRs come from branches in this repo. The push: run on the branch already reports the required checks (check, test, secrets) on the head commit, so the PR shows them without a PR trigger.
• With both triggers, every push to a PR branch starts two CI runs. They share one concurrency group with cancel-in-progress, so the PR run cancels the push run and the Actions list fills with cancelled runs.
• The duplicate runs double the macOS jobs. GitHub limits concurrent macOS runners per account, so extra macOS jobs queue and get dropped.
• Fork PRs are rare and go through just review-pr. The owner can merge them without the checks.

The copy-on-write tests have no jobs of their own. They run in test, whose whole suite uses an XFS reflink TMPDIR. Orch's cow code only wraps git-cow, and git-cow's own CI tests clonefile on macOS, so orch runs no cow tests on macOS. build-macos is the only macOS job. GitHub has few macOS arm64 runners, and when a push asked for three, they queued until GitHub dropped them ("not acquired by Runner"). Do not split build-macos into a matrix or add another macOS job.

Agent sandbox

Agents run in worktrees, NOT the main project directory. Orch enforces this:

1. Prompt-level: system prompt tells agents the main project dir is read-only
2. Tool-level: dynamic --disallowedTools blocks Read/Write/Edit/Bash targeting the main project dir
3. Config: workflow.sandbox: false to disable (not recommended)

Codex sandbox config

Codex runs under exec with --sandbox workspace-write -c 'approval_policy="never"' and network access enabled by default. --full-auto is deprecated (Codex 0.128+) and must not be reintroduced.

The sandbox level comes from workflow.permissions.sandbox (workspace-write | full-access) and the approval mode from workflow.permissions.mode (autonomous | supervised). See src/engine/runner/agents/codex.rs for the exact flag combinations.

Network access is enabled via -c 'sandbox_workspace_write.network_access=true' and the shell environment is inherited via -c 'shell_environment_policy.inherit=all'. Both flags must precede exec — placing them after silently fails.

Control Session (orch chat)

Interactive conversational control plane — natural-language interface to orch state. Implementation: src/control.rs, src/cli/chat.rs, prompt at prompts/control_system.md. Persistence in orch.db (control_messages table + kv keys control:model, control:agent, control:memory:{session}:*).

orch chat                           # interactive REPL
orch chat "what's running?"         # single message
orch chat --session ops             # named session (isolated history + memories)
orch chat history [--search QUERY]  # show / search past messages

Inside the REPL: /model <name> or /model <agent>:<model>, /agent <name>. Changing model/agent resets the stored session UUID so the next message starts fresh. Selection is validated by a test invocation before being saved (catches rate limits, missing API keys, expired credits). Sessions isolate history + memories; model/agent selection is global.

Agent invocations run one-shot via bash -c with a 45 s timeout, not in tmux.

Landing the Plane (Session Completion)

Task agents dispatched by orch: If ORCH_AGENT or TASK_ID env vars are set, SKIP this section entirely. The engine handles pushing, PR creation, and cleanup. You should only commit — never push.

MANDATORY WORKFLOW:

1. File issues for remaining work - Create issues for anything that needs follow-up
2. Run quality gates (if code changed) - Tests, linters, builds
3. Update issue status - Close finished work, update in-progress items
4. Commit your changes
5. Clean up - Clear stashes, prune remote branches
6. Verify - All changes committed
7. Hand off - Provide context for next session