fucho-e2e-dev
Skill1 starsChanged 4 months agoApache-2.0View on GitHub
End-to-end test the fucho multi-agent coding orchestrator's critical user journeys (CUJs). Two halves — a deterministic mock-LLM driver (fucho_cuj.py) that boots a throwaway local server + mock LLM and asserts the substrate (boot, bridged sys_* tool dispatch, the blast_radius / spawn_bounds / headless_subagent_purpose_guard guardrails, fan-out delegation), and a live real-CLI recipe (real claude/codex/pi, real worktrees/PRs) for fucho's actual judgment. Load when developing, testing, or debugging examples/fucho — its config.yaml, the claude_code/codex/pi sub-agents, the investigate/fanout/cross-review skills, or the omnicraft.inner.nessie.policies guardrails — or reproducing a fucho orchestration bug.
- Reads credentials
- Deletes or force-pushes
- Commits and pushes
In this repository9 other files it gives its agents.
- AGENTS.mdAGENTS.md
- .github/copilot-instructions.mdCopilot instructions
- antigravity-native-e2e-devSkill
- antigravity-sdk-e2e-devSkill
- cli-setup-verifySkill
- copilot-sdk-e2e-devSkill
- cursor-sdk-e2e-devSkill
- harness-integration-guideSkill
- pi-native-e2e-devSkill
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 1 is distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 67
Read the result with … | grep '^SUMMARY' | python -m json.tool.
- Very specific: line 86
Run the same scenario → the check flips to true.
- Very specific: line 128
Always pass --server "$SERVER".
- Distinctive: line 227
If a worker CLI, credential, or egress isn't available, say the live CUJ was
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
fucho orchestrator: end-to-end CUJ dev & testing
fucho (examples/fucho/) is a multi-agent coding orchestrator: a
claude-sdk "brain" that writes no code itself and delegates everything to three
coding sub-agents — claude_code (claude-native), codex (codex-native), and
pi (headless, multi-model). Its critical user journeys are orchestration
behaviors, not single-turn answers:
- roster preflight — first turn runs
command -v claude codex pi, routes only to workers whose CLI resolved. - investigate — read-only work fanned to
explore/searchsub-agents; synthesize from their reports. - fanout — independent tasks, each in its own git worktree + sub-agent, each opening its own PR.
- cross-review — an implementer's diff is verified by a different-vendor sub-agent (diff + contract only); blocking issues become fix-tasks.
- plan gate / inbox — pull the human in at the plan gate; supervise via the inbox + autowake, never busy-poll.
- guardrails (
omnicraft.inner.nessie.policies) —blast_radius(deny force-push /rm -rf /),spawn_bounds(cap dispatches per turn),headless_subagent_purpose_guard(every dispatch needsargs.purpose).
This skill tests those CUJs two ways. Use both — they cover different things:
| Half | What it proves | Needs |
Mock loop (fucho_cuj.py) | The substrate/mechanics — the brain is scripted, so this proves bundle load, server-side policy resolution, bridged sys_* tool dispatch, the guardrail DENYs, and fan-out — deterministically, with no creds | nothing (mock LLM) |
| Live recipe | fucho's judgment — does the real brain preflight, decompose, delegate, cross-review, and pull in the human correctly | real claude/codex/pi + model creds + network |
Like the sibling harness skills, turns run from your current checkout (
omni run <bundle> --server <url>= local runner + remote server), so testing exercises exactly the code you're on.
Interpreter
The driver and CLI need the repo's Python ≥3.12 env. If .venv/ is missing,
create it once from the checkout:
uv run --frozen python -c "import omnicraft; print('ok')" # builds .venvThen use .venv/bin/python / .venv/bin/omni below.
Part A — the deterministic mock loop (fucho_cuj.py)
The driver boots a throwaway local OmniCraft server (which carries
omnicraft.inner.nessie.policies — the module fucho's guardrails resolve) plus
the repo's mock-LLM server, rewrites the fucho bundle to the openai-agents
harness wired to the mock, then runs omnicraft run turns where the brain is
scripted (text or tool calls). It prints one SUMMARY {json} per scenario and
exits non-zero if any check failed.
.venv/bin/python .claude/skills/fucho-e2e-dev/fucho_cuj.py --list-scenarios
.venv/bin/python .claude/skills/fucho-e2e-dev/fucho_cuj.py --scenario all
.venv/bin/python .claude/skills/fucho-e2e-dev/fucho_cuj.py --scenario guardrail_purpose --keepRead the result with … | grep '^SUMMARY' | python -m json.tool. Each run takes
~45–55s for all five scenarios; no credentials or egress are required.
Scenario catalog
| Scenario | Scripts the brain to… | Hard check |
boot | reply with text | exit 0 + non-trivial reply (bundle load, server-side policy resolve, turn completes) |
tool_dispatch | call sys_os_shell to write a sentinel | the file appears on disk (bridged sys_* dispatch works; blast_radius ALLOWs benign shell) |
guardrail_purpose | sys_session_send with no args.purpose | tool output carries Denied by policy: … must declare what kind of work it is (headless_subagent_purpose_guard) |
guardrail_blast_radius | sys_os_shell("git push --force …") | tool output carries Denied by policy: … blast-radius policy |
fanout_dispatch | emit 6 sys_session_send in one turn | ≥2 sub-agent dispatch handles created (fan-out substrate). Finding: reports whether the spawn_bounds cap fired (see Known sharp edges) |
The verifiable before→after loop
The driver exists for a loop, not a one-shot. To prove a fix:
- On the unfixed code, run the scenario → a check is
false(baseline). - Make the change.
- Run the same scenario → the check flips to
true.
A fix is "verifiable" only if a check flips. If it doesn't flip, you can't prove
the change did anything — keep working. To cover a new mechanism, add a
scenario_* function + a row in _SCENARIOS (each builds a bundle, scripts the
mock, runs a turn, and asserts an observable effect — a session item, a deny
sentinel, a file on disk).
What the mock loop can and can't prove
It tests mechanics because the brain is scripted: tool dispatch, the guardrail gate, session persistence, fan-out plumbing. It does not test fucho's judgment (whether the real brain preflights, decomposes, picks the right vendor, cross-reviews). That is the live recipe.
Part B — the live recipe (real claude/codex/pi)
Prereqs (check first)
- You're on the branch you want to test.
- A Claude provider for the brain (
omni setup, orANTHROPIC_API_KEY, or a Databricks default). Verify booleans only — never print keys. - Worker CLIs on PATH — this is the roster preflight:
A worker is launchable only if its binary resolved. Cross-review needs two different vendors available.command -v claude codex pi || true - Network egress to the model backends;
ghauthed if you want real PRs.
Run a live turn
.venv/bin/omni server start && .venv/bin/omni server status # prints $SERVER, e.g. http://127.0.0.1:6767
SERVER=http://127.0.0.1:6767
timeout 280 .venv/bin/omni run examples/fucho \
-p "Investigate how the runner enforces tool-call policies and report file:line evidence." \
--server "$SERVER" 2>&1Always pass --server "$SERVER"; omitting it routes to the configured remote
deploy, which may be stale and reject parts of the bundle.
Observe CUJs (CLI + HTTP API + filesystem)
Grab the session id, then read the transcript and the side effects:
SID=$(curl -s "$SERVER/v1/sessions?kind=default&order=desc&limit=1" | python -c "import sys,json;print(json.load(sys.stdin)['data'][0]['id'])")
curl -s "$SERVER/v1/sessions/$SID/items" | python -m json.tool | tail -60 # brain transcript + tool calls
curl -s "$SERVER/v1/sessions/$SID/child_sessions" | python -m json.tool # dispatched sub-agents
git worktree list # fanout: one per task
cat .fucho/registry.json 2>/dev/null # fucho's task list
gh pr list --author "@me" # each implementer opens its own PRPer-CUJ live playbook
| CUJ | Drive it | Look for |
| roster preflight | first live turn on a box missing a CLI | fucho tells you which worker is unavailable; routes around it |
| investigate | prompt a read-only question ("explain/audit/why does X…") | child_sessions with purpose: explore/search; answer cites their reports, not fucho's own deep reads |
| fanout | prompt 2–3 independent changes | one worktree + one sub-agent + one PR per task |
| cross-review | let an implementer finish | a different-vendor reviewer child with purpose: review; blocking issues sent back to the same implementer session |
| plan gate / inbox | a multi-step task | fucho pauses for human approval at the plan gate; ends its turn after dispatch and is autowoken by the inbox (no busy-poll) |
| guardrails (ASK) | a task that pushes/merges | the runner surfaces an approval card; ask_timeout: 86400 keeps it open |
For the guardrail DENY set (force-push, rm -rf /, unmarked dispatch,
fan-out cap), prefer the mock loop — it's deterministic and creates no real
side effects.
CUJ coverage map
| CUJ | Mock loop | Live recipe |
| boot / turn completes | boot | any live turn |
bridged sys_* dispatch | tool_dispatch | tool calls in …/items |
headless_subagent_purpose_guard | guardrail_purpose ✅ | (deny — prefer mock) |
blast_radius | guardrail_blast_radius ✅ | ASK card on push/merge |
spawn_bounds | fanout_dispatch (finding) ⚠️ | verify cap live |
| fanout delegation | fanout_dispatch (handles) | child_sessions + worktrees + PRs |
| investigate / cross-review / plan gate / inbox | — (needs judgment) | live playbook above |
Known sharp edges (found while building this skill — verify, may change)
spawn_boundsper-turn cap does not trip in the local server-side path. The cap is a stateful per-turn counter, but the server rebuilds the policy engine pertools/call(_build_policy_engine_from_spec,sessions.py), so the counter resets every call. Stateless policies (purpose_guard,blast_radius) are unaffected.fanout_dispatchreports this as a finding rather than failing. Verify the cap live, where a persistent per-turn engine applies.- Two deny formats. Bridged
sys_*tools surface a denial as{"error": "Denied by policy: <reason>"}; SDK function tools use[Denied by policy: <name>] {json}. Both share theDenied by policy:marker — match on that plus a policy-specific reason fragment (the driver does). - Live fan-out needs the worker CLIs. In the mock loop, sub-agents are
rewritten to
openai-agentsso a dispatch needs no binary. Live, a missingclaude/codex/pimakes that worker fail to boot — treat it as UNAVAILABLE. - Default server gotcha.
config.yaml'sserver:points at a remote deploy; always pass--server "$SERVER"for local testing.
Code & tests
- Bundle / prompt / guardrails:
examples/fucho/config.yaml - Sub-agents:
examples/fucho/agents/{claude_code,codex,pi}/config.yaml - Orchestration skills:
examples/fucho/skills/{investigate,fanout,cross-review}/SKILL.md - Guardrail policies:
omnicraft/inner/nessie/policies.py - Runner-side gate:
omnicraft/runner/policy.py; server-side tool-call enforcement:omnicraft/server/routes/sessions.py - Mock LLM server:
tests/server/integration/mock_llm_server.py
# Existing pytest e2e for fucho (mock-LLM) — complementary to this skill:
uv run --frozen --extra dev python -m pytest \
tests/e2e/test_fucho_e2e.py \
tests/e2e/test_fucho_cost_advisor_e2e.py \
tests/e2e/test_fucho_subagent_model_e2e.py -qTeardown — non-negotiable
The driver reaps everything it starts, including the per-conversation
omnicraft.host._daemon_entry / runner._entry / harnesses._runner
subprocesses an omni run turn spawns (a plain server SIGTERM leaves these
orphaned). The sweep is scoped to this interpreter, so it never touches another
worktree. After a live session, sweep manually:
.venv/bin/omni server stop
pgrep -af "$(pwd)/.venv/bin/python -m omnicraft" | grep -E "_entry|_runner|_daemon" || echo cleanHonesty
If a worker CLI, credential, or egress isn't available, say the live CUJ was
skipped — don't claim it passed. The strongest evidence is a reproduced
baseline plus the flipped check (mock loop) or the observed round trip in
…/items + …/child_sessions (live). Report the real SUMMARY lines, not a
summary of a summary.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

