verify-with-playwright
Skill9 starsChanged 6 months agoMITView on GitHub
Drive a real browser against AI Atelie's running dev server to verify a change works, and capture screenshots + video as PR evidence. Use whenever the user asks to "verify in browser", "prove it works", "add PR evidence", or before opening a PR that touches `web/` or any user-facing surface. Assumes `bun run dev` is already running on http://localhost:5173.
- Deletes or force-pushes
In this repository20 other files it gives its agents.
- AGENTS.mdAGENTS.md
- canvas-syncSkill
- cuj-guardianSkill
- frontend-designSkill
- grill-meSkill
- pr-evidenceSkill
- semantic-commitSkill
- ship-taskSkill
- animated-videoSkill
- create-design-systemSkill
- design-aesthetic-presetsSkill
- design-critiqueSkill
- design-md-authorSkill
- exportSkill
- frontend-designSkill
- handoff-to-claude-codeSkill
- interactive-prototypeSkill
- make-tweakableSkill
- save-as-standalone-htmlSkill
- send-to-canvaSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
13 rules. About 0 are common, 8 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈8
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 14
After implementing any change that touches web/, the iframe canvas, the chat surface, the editor toolbar, or any other user-visible …
- Very specific: line 15
When the ship-task orchestrator delegates verification.
- Distinctive: line 16
When the user asks "does it actually work in the browser?" or "show me proof".
- Distinctive: line 25
Confirm the dev server is up: Run:
- Very specific: line 29
If down, STOP and tell the user: "Run bun run dev in another terminal, then retry." Do NOT spawn bun run dev from this skill — backgrounded …
- Distinctive: line 93
If your screenshot only shows the chat sidebar without the canvas, you've captured *the harness reacting* but NOT *the user-visible result*.
- Distinctive: line 107
Don't sleep a fixed timeout — poll for the *signal* that completion happened:
- Distinctive: line 144
Chat thread: wait for an "Undo" / "Restore" button to appear (which only renders post-completion in this codebase).
- Very specific: line 147
NEVER use await page.waitForTimeout(60_000) and assume the agent finished.
- Distinctive: line 151
After a successful run, the relevant artifacts live at:
- Distinctive: line 172
After bundling, prune older runs to keep the last 5:
- Very specific: line 202
When you finish, output a structured block the orchestrator (ship-task) can parse:
- Distinctive: line 216
If the run failed, do NOT delete the failure artifacts.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
verify-with-playwright
A contributor workflow for AI Atelie. Drives the local dev server with @playwright/test (runner mode, not MCP), produces a test-results/<spec>/ folder with video + screenshots + trace, then bundles the relevant files into .evidence/<run>/ for upload to a PR.
This skill is dev-time only. It does not load into adapter sessions spawned by the editor.
When to invoke
- After implementing any change that touches
web/, the iframe canvas, the chat surface, the editor toolbar, or any other user-visible behaviour. - When the
ship-taskorchestrator delegates verification. - When the user asks "does it actually work in the browser?" or "show me proof".
Skip when:
- The change is API-only with no observable surface (use
web/tests/test-*.mjsintegration tests instead). - The change is a doc-only or config-only edit.
Preflight (do this before writing any spec)
- Confirm the dev server is up. Run:
If down, STOP and tell the user: "Runcurl -sSf http://localhost:5173 > /dev/null && echo "up" || echo "down"bun run devin another terminal, then retry." Do NOT spawnbun run devfrom this skill — backgrounded dev servers leak across sessions and will clobber the user's own terminal. - Confirm Playwright is installed.
bunx playwright --versionshould printVersion 1.59.xor higher. If not, runbun add -D @playwright/test && bunx playwright install chromium. - Read
playwright.config.tsat repo root to confirm the test dir, baseURL, and artifact policy haven't drifted.
Workflow (paste this checklist into your reply and tick as you go)
- Preflight passed (dev server up, Playwright installed).
- Acceptance criteria identified. From the issue or the user's task description, list the user-observable properties that must hold. Example for #1 (live preview): "agent text deltas appear in iframe srcdoc within 500ms of arrival; full reload not required".
- Spec authored. Write
web/tests/e2e/<slug>.spec.ts. One scenario per file. Use role-based selectors. Each acceptance criterion → one or moreexpect(...)calls. - Spec runs.
bunx playwright test web/tests/e2e/<slug>.spec.ts --reporter=list. - Wait for the agent to FULLY COMPLETE before stopping the recording. This is the rule reviewers care about most — see "Evidence must show the post-completion state" below. A capture that ends mid-stream proves nothing.
- Capture the post-completion state explicitly. A final screenshot AFTER the busy/loading state clears AND the user-visible result is rendered. The reviewer must be able to see what the agent actually produced, not just that it was working.
- Evidence bundled. Copy
test-results/<slug>/video.webmand any*.pnginto.evidence/<ISO-timestamp>-<slug>/. - (Optional) headed run for video review.
--headedfor clearer screen recordings on visual changes. - Cleanup. Prune
.evidence/to keep only the last 5 run folders. - Report. Output the absolute paths of evidence artifacts so
ship-task(or the user) can attach them to the PR.
Authoring a spec
Minimal scaffold:
import { test, expect } from "@playwright/test";
test("live-preview streams text deltas into iframe", async ({ page }) => {
await page.goto("/");
// ... drive the surface that exercises the change
// ... assert acceptance criteria
await expect(page.frameLocator("iframe").locator("h1")).toHaveText(/Hello/);
});Conventions:
- Always
await page.goto('/')first. ThebaseURLresolves tohttp://localhost:5173. - Prefer role-based selectors (
getByRole('button', { name: 'New project' })) over CSS selectors. They survive markup churn. - One scenario per file, named
<feature>.spec.ts. - Test the surface, not the agent's output. The agent is non-deterministic; the surface's behaviour around it should be deterministic.
- No external network in tests. Mock the agent stream with
page.route('/api/...', route => route.fulfill({...}))if needed.
Running the spec
# Full run, list reporter (Claude reads this output well)
bunx playwright test web/tests/e2e/<slug>.spec.ts --reporter=list
# Headed mode for better video evidence
bunx playwright test web/tests/e2e/<slug>.spec.ts --headed
# View the HTML report after a failure
bunx playwright show-reportEvidence MUST show the post-completion state — including the canvas
Rule: a capture that ends while the agent is still working, OR that doesn't include the visual outcome on the canvas / iframe / final UI surface, is incomplete evidence and should never be submitted to a PR.
The point of evidence is to demonstrate, in order:
- The change produces the expected user-visible outcome. The agent finished, the result rendered, the iframe canvas shows the new content (or the file was written, or the UI updated to reflect the new state) — whatever the acceptance criteria says is "the thing the user will see."
- (Optionally) The chat / chip / loading UI was correct in flight — busy pill rotated, chip turned amber while editing, etc.
- (Optionally) Failures are valid evidence too. If the agent's API errored, hit a rate limit, or refused — capture that screenshot and explain. It's better than no evidence; reviewers can tell whether the feature works or whether the agent had a bad day.
If your screenshot only shows the chat sidebar without the canvas, you've captured the harness reacting but NOT the user-visible result. Reviewers can't tell whether the feature works.
Pick prompts that produce a visible canvas change
For features whose acceptance criteria is "the chat sidebar / chips / status indicator does X," a pure-chat capture is fine. But the default evidence prompt should be one that ALSO updates the iframe canvas, so the same screenshot proves both surfaces work:
- ✅
"Make this a Hello World page on a bright green background. Replace whatever is there."— agent writes index.html, canvas re-renders green, chip strip + canvas both visible. - ✅
"Add a red rectangle in the top-right corner of the canvas."— visible diff on the iframe. - ❌
"Just read the current index.html."— chat updates, canvas unchanged. You proved the chat works, not the product.
If the acceptance criteria genuinely is chat-only (e.g., "tool chips are colored"), a chat-only capture is acceptable but you should still trigger a chat that would have updated the canvas — that way the still-blank canvas is itself evidence that the agent ran in a normal mode, not a special test mode.
How to wait for completion correctly
Don't sleep a fixed timeout — poll for the signal that completion happened:
// Wait for the agent's busy state to CLEAR after we know it started.
// (Make sure we saw it busy at least once first — otherwise we'd race
// past a fast turn that finished before we started polling.)
let sawBusy = false;
const start = Date.now();
while (Date.now() - start < 6 * 60_000) {
const busy = await page.locator("[role=status]").isVisible().catch(() => false);
if (busy) sawBusy = true;
else if (sawBusy) break;
await page.waitForTimeout(2_000);
}
expect(sawBusy, "agent run was observed to be in flight at some point").toBe(true);Then add ONE extra await page.waitForTimeout(2_000) after the busy state clears so the iframe re-renders before you screenshot.
Use Sonnet, not Opus, for evidence runs
Evidence captures don't need the most expensive model — Sonnet produces equivalent visible results for the kinds of "Hello World" / "make a red rectangle" prompts these runs use, at a fraction of the token cost. Force it via localStorage before navigating:
await ctx.addInitScript(() => {
localStorage.setItem("editor-model-id", "claude-sonnet-4-6");
});(See web/src/data/modelPresets.ts for the canonical id list. Update if Anthropic ships a newer Sonnet that's cheaper still.)
Reserve Opus for the actual feature work the maintainer is shipping — not the dogfood test that re-runs on every PR.
Other valid completion signals
- Filesystem: poll
web/projects/<id>/until a target file is written or modified (usestat().mtimeMsagainst a pre-prompt snapshot — seeweb/tests/e2e/journeys/agent-edits-canvas.spec.tsfor the pattern). - DOM mutation:
page.waitForFunction(() => document.querySelector("[data-step-count]")?.textContent !== "0") - Chat thread: wait for an "Undo" / "Restore" button to appear (which only renders post-completion in this codebase).
- Iframe content:
await page.frameLocator("iframe").locator("body").innerText()returns text matching the expected result.
NEVER use await page.waitForTimeout(60_000) and assume the agent finished. Some runs are 5 seconds, some are 5 minutes.
Bundling evidence
After a successful run, the relevant artifacts live at:
test-results/<slug>-<browser>/video.webmtest-results/<slug>-<browser>/test-failed-*.png(if any)playwright-report/data/<hash>.zip(HTML report bundle)
Copy them into .evidence/<ISO-timestamp>-<slug>/:
TS=$(date -u +%Y-%m-%dT%H-%M-%SZ)
RUN_DIR=".evidence/${TS}-<slug>"
mkdir -p "$RUN_DIR"
cp test-results/<slug>*/video.webm "$RUN_DIR/" 2>/dev/null || true
cp test-results/<slug>*/*.png "$RUN_DIR/" 2>/dev/null || true
echo "Evidence bundled at $RUN_DIR".evidence/ is gitignored.
Disk hygiene
After bundling, prune older runs to keep the last 5:
ls -1t .evidence | tail -n +6 | xargs -I{} rm -rf ".evidence/{}"Once evidence is uploaded to a PR via bun run upload:evidence (see scripts/upload-evidence.mjs), the canonical copy lives on GitHub's user-attachments/assets/... CDN — videos play inline, images embed natively. Local .evidence/ is just scratch. Note: this requires a one-time bun run setup:attach to log into github.com via Chromium; the cookies live in ~/.local/state/aiatelie/ (per-user, never committed).
Known failure modes
:5173taken by stale Vite. Diagnose withlsof -ti:5173; surface the PID to the user. Don't auto-kill.- Headless vs headed divergence. Some CSS (
:focus-visible, scrollbar widths) renders differently. Use--headedfor visual evidence runs. localhostvs127.0.0.1. Vite 8.x binds tolocalhost(which resolves to IPv6::1on macOS), not127.0.0.1. Always uselocalhost:5173;curl 127.0.0.1:5173will fail.- MJS test files masquerading as Playwright specs. The existing
web/tests/test-*.mjsare integration tests, NOT Playwright specs. Don't put.spec.tsfiles inweb/tests/root; only underweb/tests/e2e/. - Bun loader edge cases. Install Playwright with
bun add, but run withbunx playwright test— Playwright's runner doesn't fully support Bun's loader at the time of writing. - Evidence too large. A 5-min Playwright recording at native fps is ~20MB. Compress with
ffmpeg -filter:v "setpts=0.25*PTS,scale=1024:-2" -an -c:v libx264 -preset fast -crf 28 -movflags +faststart -y out.mp4(4× speed, 1024w, no audio, web-tuned). Yields ~300KB / ~13s — small enough that the user-attachments inline player loads fast, fast enough to watch, still shows the entire idle → busy → completion arc. - Stopping the recording mid-stream. The single most common evidence failure mode. See "Evidence MUST show the post-completion state" above.
Anti-patterns the skill must refuse
- Submitting evidence that ends while the agent is still working. No matter how good the in-flight footage looks, if the recording stops before the post-completion state, the evidence is incomplete and the PR review will (correctly) bounce.
- Capturing only the chat sidebar without the canvas. The chat is the harness; the canvas is the product. A screenshot of just the chips proves the harness reacted, not that the user-visible result rendered. Default to a prompt that updates the iframe so a single full-page screenshot covers both.
- Hiding agent failures. If the API errors or rate-limits or refuses, capture that screenshot AND explain it in the PR body. "The flow ran, the agent declined for X reason, here's what the UI did when that happened" is much better evidence than no evidence.
- Burning Opus on dogfood runs. Set
editor-model-idtoclaude-sonnet-4-6via the init script (see "Use Sonnet, not Opus" above). Opus is for the maintainer's real feature work, not the harness check. - Aborting / cancelling the run to save time on the capture. The point of evidence is the full flow; if you cancel, you're not proving anything happened.
- Sleep-then-screenshot as a fake completion signal. Use a real signal (busy state cleared, file modified, DOM mutation) — never a fixed timeout.
- "It worked when I ran it manually" — without captured evidence, this doesn't appear in the PR. Re-run with capture before submitting.
Reporting back
When you finish, output a structured block the orchestrator (ship-task) can parse:
VERIFY-RESULT: pass | fail
EVIDENCE-DIR: .evidence/<ISO-timestamp>-<slug>/
ARTIFACTS:
- .evidence/<.../video.webm>
- .evidence/<.../before.png>
- .evidence/<.../after.png>
ACCEPTANCE-CRITERIA:
- [✓] Criterion 1 (assertion: ...)
- [✓] Criterion 2 (assertion: ...)If the run failed, do NOT delete the failure artifacts. Surface them so the user can debug.
See also
playwright.config.ts— the canonical config (artifact policy, baseURL, webServer).web/tests/e2e/README.md— contributor-facing e2e overview..claude/skills/ship-task/SKILL.md— the orchestrator that calls this skill.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

