agentleFS
Omni-CraftOmni-Craft/OmniCraft/.claude/skills/fucho-e2e-dev/SKILL.md

fucho-e2e-dev

Skill1 starsChanged 4 months agoApache-2.0View on GitHub

End-to-end test the fucho multi-agent coding orchestrator's critical user journeys (CUJs). Two halves — a deterministic mock-LLM driver (fucho_cuj.py) that boots a throwaway local server + mock LLM and asserts the substrate (boot, bridged sys_* tool dispatch, the blast_radius / spawn_bounds / headless_subagent_purpose_guard guardrails, fan-out delegation), and a live real-CLI recipe (real claude/codex/pi, real worktrees/PRs) for fucho's actual judgment. Load when developing, testing, or debugging examples/fucho — its config.yaml, the claude_code/codex/pi sub-agents, the investigate/fanout/cross-review skills, or the omnicraft.inner.nessie.policies guardrails — or reproducing a fucho orchestration bug.

  • Reads credentials
  • Deletes or force-pushes
  • Commits and pushes

The file

We split this skill into the rules it gives an agent

4 rules. About 0 are common, 1 is distinctive, and 3 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Read the result with … | grep '^SUMMARY' | python -m json.tool.

    line 67
  2. Very specific:

    Run the same scenario → the check flips to true.

    line 86
  3. Very specific:

    Always pass --server "$SERVER".

    line 128
  4. Distinctive:

    If a worker CLI, credential, or egress isn't available, say the live CUJ was

    line 227

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

fucho orchestrator: end-to-end CUJ dev & testing

fucho (examples/fucho/) is a multi-agent coding orchestrator: a claude-sdk "brain" that writes no code itself and delegates everything to three coding sub-agents — claude_code (claude-native), codex (codex-native), and pi (headless, multi-model). Its critical user journeys are orchestration behaviors, not single-turn answers:

  • roster preflight — first turn runs command -v claude codex pi, routes only to workers whose CLI resolved.
  • investigate — read-only work fanned to explore/search sub-agents; synthesize from their reports.
  • fanout — independent tasks, each in its own git worktree + sub-agent, each opening its own PR.
  • cross-review — an implementer's diff is verified by a different-vendor sub-agent (diff + contract only); blocking issues become fix-tasks.
  • plan gate / inbox — pull the human in at the plan gate; supervise via the inbox + autowake, never busy-poll.
  • guardrails (omnicraft.inner.nessie.policies) — blast_radius (deny force-push / rm -rf /), spawn_bounds (cap dispatches per turn), headless_subagent_purpose_guard (every dispatch needs args.purpose).

This skill tests those CUJs two ways. Use both — they cover different things:

HalfWhat it provesNeeds
Mock loop (fucho_cuj.py)The substrate/mechanics — the brain is scripted, so this proves bundle load, server-side policy resolution, bridged sys_* tool dispatch, the guardrail DENYs, and fan-out — deterministically, with no credsnothing (mock LLM)
Live recipefucho's judgment — does the real brain preflight, decompose, delegate, cross-review, and pull in the human correctlyreal claude/codex/pi + model creds + network

Like the sibling harness skills, turns run from your current checkout (omni run <bundle> --server <url> = local runner + remote server), so testing exercises exactly the code you're on.

Interpreter

The driver and CLI need the repo's Python ≥3.12 env. If .venv/ is missing, create it once from the checkout:

uv run --frozen python -c "import omnicraft; print('ok')"   # builds .venv

Then use .venv/bin/python / .venv/bin/omni below.


Part A — the deterministic mock loop (fucho_cuj.py)

The driver boots a throwaway local OmniCraft server (which carries omnicraft.inner.nessie.policies — the module fucho's guardrails resolve) plus the repo's mock-LLM server, rewrites the fucho bundle to the openai-agents harness wired to the mock, then runs omnicraft run turns where the brain is scripted (text or tool calls). It prints one SUMMARY {json} per scenario and exits non-zero if any check failed.

.venv/bin/python .claude/skills/fucho-e2e-dev/fucho_cuj.py --list-scenarios
.venv/bin/python .claude/skills/fucho-e2e-dev/fucho_cuj.py --scenario all
.venv/bin/python .claude/skills/fucho-e2e-dev/fucho_cuj.py --scenario guardrail_purpose --keep

Read the result with … | grep '^SUMMARY' | python -m json.tool. Each run takes ~45–55s for all five scenarios; no credentials or egress are required.

Scenario catalog

ScenarioScripts the brain to…Hard check
bootreply with textexit 0 + non-trivial reply (bundle load, server-side policy resolve, turn completes)
tool_dispatchcall sys_os_shell to write a sentinelthe file appears on disk (bridged sys_* dispatch works; blast_radius ALLOWs benign shell)
guardrail_purposesys_session_send with no args.purposetool output carries Denied by policy: … must declare what kind of work it is (headless_subagent_purpose_guard)
guardrail_blast_radiussys_os_shell("git push --force …")tool output carries Denied by policy: … blast-radius policy
fanout_dispatchemit 6 sys_session_send in one turn≥2 sub-agent dispatch handles created (fan-out substrate). Finding: reports whether the spawn_bounds cap fired (see Known sharp edges)

The verifiable before→after loop

The driver exists for a loop, not a one-shot. To prove a fix:

  1. On the unfixed code, run the scenario → a check is false (baseline).
  2. Make the change.
  3. Run the same scenario → the check flips to true.

A fix is "verifiable" only if a check flips. If it doesn't flip, you can't prove the change did anything — keep working. To cover a new mechanism, add a scenario_* function + a row in _SCENARIOS (each builds a bundle, scripts the mock, runs a turn, and asserts an observable effect — a session item, a deny sentinel, a file on disk).

What the mock loop can and can't prove

It tests mechanics because the brain is scripted: tool dispatch, the guardrail gate, session persistence, fan-out plumbing. It does not test fucho's judgment (whether the real brain preflights, decomposes, picks the right vendor, cross-reviews). That is the live recipe.


Part B — the live recipe (real claude/codex/pi)

Prereqs (check first)

  1. You're on the branch you want to test.
  2. A Claude provider for the brain (omni setup, or ANTHROPIC_API_KEY, or a Databricks default). Verify booleans only — never print keys.
  3. Worker CLIs on PATH — this is the roster preflight:
    command -v claude codex pi || true
    A worker is launchable only if its binary resolved. Cross-review needs two different vendors available.
  4. Network egress to the model backends; gh authed if you want real PRs.

Run a live turn

.venv/bin/omni server start && .venv/bin/omni server status   # prints $SERVER, e.g. http://127.0.0.1:6767
SERVER=http://127.0.0.1:6767
timeout 280 .venv/bin/omni run examples/fucho \
  -p "Investigate how the runner enforces tool-call policies and report file:line evidence." \
  --server "$SERVER" 2>&1

Always pass --server "$SERVER"; omitting it routes to the configured remote deploy, which may be stale and reject parts of the bundle.

Observe CUJs (CLI + HTTP API + filesystem)

Grab the session id, then read the transcript and the side effects:

SID=$(curl -s "$SERVER/v1/sessions?kind=default&order=desc&limit=1" | python -c "import sys,json;print(json.load(sys.stdin)['data'][0]['id'])")
curl -s "$SERVER/v1/sessions/$SID/items"          | python -m json.tool | tail -60   # brain transcript + tool calls
curl -s "$SERVER/v1/sessions/$SID/child_sessions" | python -m json.tool             # dispatched sub-agents
git worktree list                                  # fanout: one per task
cat .fucho/registry.json 2>/dev/null               # fucho's task list
gh pr list --author "@me"                          # each implementer opens its own PR

Per-CUJ live playbook

CUJDrive itLook for
roster preflightfirst live turn on a box missing a CLIfucho tells you which worker is unavailable; routes around it
investigateprompt a read-only question ("explain/audit/why does X…")child_sessions with purpose: explore/search; answer cites their reports, not fucho's own deep reads
fanoutprompt 2–3 independent changesone worktree + one sub-agent + one PR per task
cross-reviewlet an implementer finisha different-vendor reviewer child with purpose: review; blocking issues sent back to the same implementer session
plan gate / inboxa multi-step taskfucho pauses for human approval at the plan gate; ends its turn after dispatch and is autowoken by the inbox (no busy-poll)
guardrails (ASK)a task that pushes/mergesthe runner surfaces an approval card; ask_timeout: 86400 keeps it open

For the guardrail DENY set (force-push, rm -rf /, unmarked dispatch, fan-out cap), prefer the mock loop — it's deterministic and creates no real side effects.


CUJ coverage map

CUJMock loopLive recipe
boot / turn completesbootany live turn
bridged sys_* dispatchtool_dispatchtool calls in …/items
headless_subagent_purpose_guardguardrail_purpose ✅(deny — prefer mock)
blast_radiusguardrail_blast_radius ✅ASK card on push/merge
spawn_boundsfanout_dispatch (finding) ⚠️verify cap live
fanout delegationfanout_dispatch (handles)child_sessions + worktrees + PRs
investigate / cross-review / plan gate / inbox— (needs judgment)live playbook above

Known sharp edges (found while building this skill — verify, may change)

  • spawn_bounds per-turn cap does not trip in the local server-side path. The cap is a stateful per-turn counter, but the server rebuilds the policy engine per tools/call (_build_policy_engine_from_spec, sessions.py), so the counter resets every call. Stateless policies (purpose_guard, blast_radius) are unaffected. fanout_dispatch reports this as a finding rather than failing. Verify the cap live, where a persistent per-turn engine applies.
  • Two deny formats. Bridged sys_* tools surface a denial as {"error": "Denied by policy: <reason>"}; SDK function tools use [Denied by policy: <name>] {json}. Both share the Denied by policy: marker — match on that plus a policy-specific reason fragment (the driver does).
  • Live fan-out needs the worker CLIs. In the mock loop, sub-agents are rewritten to openai-agents so a dispatch needs no binary. Live, a missing claude/codex/pi makes that worker fail to boot — treat it as UNAVAILABLE.
  • Default server gotcha. config.yaml's server: points at a remote deploy; always pass --server "$SERVER" for local testing.

Code & tests

  • Bundle / prompt / guardrails: examples/fucho/config.yaml
  • Sub-agents: examples/fucho/agents/{claude_code,codex,pi}/config.yaml
  • Orchestration skills: examples/fucho/skills/{investigate,fanout,cross-review}/SKILL.md
  • Guardrail policies: omnicraft/inner/nessie/policies.py
  • Runner-side gate: omnicraft/runner/policy.py; server-side tool-call enforcement: omnicraft/server/routes/sessions.py
  • Mock LLM server: tests/server/integration/mock_llm_server.py
# Existing pytest e2e for fucho (mock-LLM) — complementary to this skill:
uv run --frozen --extra dev python -m pytest \
  tests/e2e/test_fucho_e2e.py \
  tests/e2e/test_fucho_cost_advisor_e2e.py \
  tests/e2e/test_fucho_subagent_model_e2e.py -q

Teardown — non-negotiable

The driver reaps everything it starts, including the per-conversation omnicraft.host._daemon_entry / runner._entry / harnesses._runner subprocesses an omni run turn spawns (a plain server SIGTERM leaves these orphaned). The sweep is scoped to this interpreter, so it never touches another worktree. After a live session, sweep manually:

.venv/bin/omni server stop
pgrep -af "$(pwd)/.venv/bin/python -m omnicraft" | grep -E "_entry|_runner|_daemon" || echo clean

Honesty

If a worker CLI, credential, or egress isn't available, say the live CUJ was skipped — don't claim it passed. The strongest evidence is a reproduced baseline plus the flipped check (mock loop) or the observed round trip in …/items + …/child_sessions (live). Report the real SUMMARY lines, not a summary of a summary.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.