agentleFS
colinhackscolinhacks/frizz/.agents/skills/frizz-artifact-e2e/SKILL.md

frizz-artifact-e2e

Skill29 starsChanged 5 days agoMITView on GitHub

End-to-end verification of a frizz change on a REAL PROMOTED ARTIFACT (not dev source), driven in a real browser. Use this whenever a change could behave differently in a promoted build than in the dev stack — anything touching packaging/bundling, detached daemons (codex app-server, the Claude broker), spawn/exec paths, worker-environment resolution, or "it works on the dev stack but does it ship?" The `frizz-stack` skill runs `startServer({dev:true})` = SOURCE and will pass while the promoted artifact is broken. This skill launches the actual artifact. Do this before claiming a dispatch/runtime change is done.

  • Reads credentials
  • Deletes or force-pushes

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 0 are distinctive, and 3 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Use a throwaway git repo as the project (isolated ~/.frizz/projects/<id>), real HOME (so Keychain claude auth works — a sandbox HOME fails …

    line 31
  2. Very specific:

    Run it with Bash run_in_background: true.

    line 54
  3. Very specific:

    Use scripts/lib/rpc-client.mjs (never hand-rolled fetch — queries are GET /rpc/<name>?input=, mutations POST, every response is …

    line 61

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

frizz-artifact-e2e — verify on the promoted artifact, not the dev stack

Frizz serves a PROMOTED ARTIFACT from ~/.frizz/builds/<digest>/runtime/, never the source checkout (see the frizz-servers-run-promoted-artifacts memory). The frizz-stack isolated stack runs startServer({dev:true}) — SOURCE. So a green frizz-stack run proves the dev path, NOT the shipped build: bundling, detached-daemon emission (build-runtime.mjs writes each DETACHED_DAEMON_ENTRIES file beside index.js), and executable/PATH resolution all differ. A real regression class (e.g. the Claude broker's daemon needing an ABSOLUTE claude path — a bare "claude" crashes it before it publishes its record, so every dispatch times out "did not become ready") is invisible to dev-source gates and to static checks (node loads the file, the daemon publishes standalone). Only launching the artifact + dispatching catches it.

The recipe

1. Launch a real promoted-artifact instance, isolated

frizz-dev (= nub … src/index.ts) builds/promotes an artifact from CURRENT SOURCE and runs it. Two gotchas:

  • You are probably running inside a frizz worker, so your env has FRIZZ_DEV_CHILD=1 and FRIZZ_LAUNCH_*. Left set, frizz-dev takes the CHILD path and dies "Frizz control-plane child has no live matching project launch owner". Strip them.
  • Use a throwaway git repo as the project (isolated ~/.frizz/projects/<id>), real HOME (so Keychain claude auth works — a sandbox HOME fails AUTH_REQUIRED:claude), and force the feature on.
REPO=$(mktemp -d /tmp/fae-XXXX); (cd "$REPO" && git init -q . && git commit -q --allow-empty -m init)
cd "$REPO" && exec env \
  -u FRIZZ_DEV_CHILD -u FRIZZ_DEV_PORT -u FRIZZ_DIRECT_SUPERVISOR -u FRIZZ_LAUNCH_OWNER_TOKEN \
  -u FRIZZ_LAUNCH_PROJECT_DIR -u FRIZZ_LAUNCH_PROJECT_ID -u FRIZZ_LAUNCH_STATE_DIR -u FRIZZ_LAUNCH_IDENTITY_SCOPE \
  -u FRIZZ_PERM_DIR -u FRIZZ_SCRIPTS_DIR \
  -u FRIZZ_SOURCE_COMMAND -u FRIZZ_STABLE_ARTIFACT -u FRIZZ_STABLE_WEB_DIST -u FRIZZ_THREAD -u FRIZZ_WORKER_PLUGIN_DIR \
  FRIZZ_CLAUDE_BROKER_BRIDGE=1 frizz-dev

REAL HOME MEANS REAL SHARED STATE, and the project registry is the part that bites. ~/.frizz/registry.json is global to HOME, not per-server — so the throwaway repo you launch against is written straight into the maintainer's real project list, and it shows up on their board. This is not hypothetical: it happened on 2026-08-19, during the run that added this warning. There is no flag that prevents it, because the isolation this recipe gives you is a throwaway PROJECT, never a throwaway HOME.

So treat the registry as something you borrowed and must put back:

nub -e 'const r=JSON.parse(require("fs").readFileSync(process.env.HOME+"/.frizz/registry.json","utf8"));console.log(r.projects.length);for(const p of r.projects)console.log(p.slug,p.path)'

Run that BEFORE you launch and again after you clean up, and diff the two. Any project of yours that survives is one the maintainer now has to delete by hand. Remove the registry entry and its ~/.frizz/projects/<id> state dir, then re-run the check and confirm the list matches what you started with.

Run it with Bash run_in_background: true. First launch BUILDS the artifact (esbuild runtime + web build, ~30-90s). Wait for requested Frizz in your default browser — http://127.0.0.1:<PROXY> (the proxy port; the child API logs server on http://127.0.0.1:<CHILD> (prod) — (prod) confirms it's the artifact, not dev). Drive the PROXY url.

2. Dispatch through the real RPC surface

Use scripts/lib/rpc-client.mjs (never hand-rolled fetch — queries are GET /rpc/<name>?input=, mutations POST, every response is {result}/{error}):

import { createRpcClient } from ".../scripts/lib/rpc-client.mjs"
const api = createRpcClient(`http://127.0.0.1:${PROXY}/`); await api.waitForHealth()
const { slug, sessionId } = await api.mutate("dispatch", { prompt: "Reply with exactly HELLO-OK then stop." })
// poll for the agent's reply — this is the real end-to-end signal, not a status field
for (let i=0;i<40;i++){ await sleep(3000)
  const tr = await api.query("threadTranscript",{slug}); if (JSON.stringify(tr).includes("HELLO-OK")) break }

For a worker-environment change, dispatch a prompt that uses the real surface — a frizz:haiku sub-agent, the frizz MCP (mcp__frizz__spawn_thread), the cc-worker hooks — and assert the sub-agent's marker token routes back. A plain "write a file" proves almost nothing. Frizz mounts no browser into a worker (since 2026-08-26 the only server it injects is frizz), so a dispatched worker has chrome-devtools tools only when the project it runs in brings them — this repo does, through its own .mcp.json; a throwaway repo does not. If your change is about the worker environment, assert which of the two you expect.

2b. A HAPPY-PATH PROMPT IS NOT A TEST — drive the ugly shapes

This is the step that has actually failed. 2026-07-26 a spike shipped with a promoted-artifact e2e, a real-Chrome screenshot, a sub-agent round-trip, a kill-the-server survival test and 2150 green unit tests. Every dispatched prompt was a toy: "reply with exactly HELLO-OK", "write a marker file", "count to 5". Within a day a real orchestrator thread and every one of its sub-agents were destroyed by a real agent running printf '\033[31m…' — an ANSI escape hit a strict protocol validator, the validator threw, and the daemon's pump read that as the session ending.

A toy prompt cannot find that, and no amount of MORE toy prompts would have. The risk in a provider integration is not in the control flow you wrote; it is in the values the provider sends back, and a clean prompt only ever produces clean values. So the workload has to be hostile ON PURPOSE:

  • control bytes and ANSI escapes in a tool argument (printf '\033[31mX\033[0m') — the shape that actually killed a thread;
  • output far past the size bounds in claude-agent-sdk-protocol.ts (128 KB event text, 64 KB JSON) — head -c 400000 /dev/zero | tr '\0' 'x';
  • non-UTF-8 / binary bytes (head -c 2048 /dev/urandom | base64);
  • unicode the validators single out — bidi overrides, zero-width joiners, astral plane, combining marks;
  • a long tool chain and a live sub-agent, so turn bracketing and liveness are exercised together.

packages/server/src/backend/_live_broker_hostile.mts is exactly this workload against a real session; run it (or extend it) rather than re-deriving the list. When you add a bound to the protocol, add a case.

The assertion that matters is not "the reply was correct" — it is "the session is still alive and still answering afterwards", checked against the daemon's own diagnostics log for a lifecycle:crashed record. A thread that dies quietly is the failure mode; a wrong answer is a much smaller problem.

3. Screenshot in a real browser tab

The chrome-devtools MCP is the reliable path here (scripts/shot.mjs's puppeteer Chrome tends to hang in a worker env). It is YOUR tool, from this repo's .mcp.json — not something Frizz put in the worker you dispatched. new_page the proxy url → wait_for the thread text → take_screenshot. Write the shot under a chrome workspace root (e.g. the frizz repo dir), NOT /tmp (access denied); then move it to a trusted embed root (~/Screenshots, the project dir, or os.tmpdir()) for the handoff and clean the repo. The board only renders threads when the project has a .frizz/ dir — mkdir -p $REPO/.frizz/threads if it's otherwise empty.

4. Clean up

Kill the instance by its EXACT proxy+child PIDs (lsof -nP -iTCP:<port> -sTCP:LISTEN -t, never broad pkill), and rm -rf the throwaway repo + its ~/.frizz/projects/<id>. A frizz server retains its artifact snapshot for its process lifetime, so the in-app restart button does NOT pick up a new build — only a full process kill + relaunch re-promotes from current source.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.