agentleFS
codeaholicguycodeaholicguy/ai-devkit/skills/dev-smoke/SKILL.md

dev-smoke

Skill1.7k starsChanged 28 days agoApache-2.0View on GitHub

AI DevKit · Smoke-test the user flows a change affects (CLI, API, UI) end to end with a disposable sandboxed script against the real app. Use before calling a feature or fix done, when unit tests pass but the flow is untested, or when users ask for a smoke, e2e, or impact check.

Adopted by

Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data

Week of Sep 13: 0 new projectsWeek of Sep 20: 0 new projectsWeek of Sep 27: 0 new projectsWeek of Oct 4: 0 new projectsWeek of Oct 11: 0 new projectsWeek of Oct 18: 0 new projectsWeek of Oct 25: 0 new projectsWeek of Nov 1: 0 new projects
Sep 2026Oct 2026Nov 2026
New projects taking a version of this file each week, over the 8 weeks our latest run read: only 0 in all, so this is a hint, not a trend.

Who's talking about codeaholicguy/ai-devkit

The biggest 5 of 42 mentions linking to the repository that our crawl found. Each line is their own headline.

See all 42

Talked about on GitHub and Hacker News (42 times).

On GitHub · 40

35 more on GitHub

On Hacker News · 2

The file

We split this skill into the rules it gives an agent

10 rules. About 0 are common, 10 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈10

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Never use production, shared environments, real accounts or credentials, or paid APIs.

    line 14
  2. Distinctive:

    Start servers on a dedicated port, wait until they respond, and stop the whole process tree on exit.

    line 15
  3. Distinctive:

    Keep secrets out of the script and its output.

    line 16
  4. Distinctive:

    Write the script outside the repo.

    line 17
  5. Distinctive:

    Read the change: git diff against the base branch, plus untracked files.

    line 22
  6. Distinctive:

    Run it and read the full output.

    line 26
  7. Distinctive:

    Delete the script and sandbox, or offer to promote valuable checks into permanent tests or the project's release smoke suite.

    line 29
  8. Distinctive:

    Use the harness's browser tool when available (for example Claude in Chrome or a Playwright MCP server).

    line 33
  9. Distinctive:

    Use a fresh browser context per check, and act like a user: navigate, click, fill forms, submit.

    line 34
  10. Distinctive:

    Not covered: native mobile apps, desktop apps, pixel-level visual diffs.

    line 38

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Smoke

Exercise the flows a change touches the way a user would, then throw the script away. Report with verify; use dev-testing for permanent tests.

Rules

  • Drive the real entry point: built binary, running server, CLI, UI, or job. Imported functions and mocks are not smoke checks.
  • Sandbox everything: temp dir, isolated HOME and config, empty git hooks, seeded test data, test accounts and tokens.
  • Never use production, shared environments, real accounts or credentials, or paid APIs. Ask before outward-facing side effects: messages, emails, webhooks, deploys, pushes.
  • Start servers on a dedicated port, wait until they respond, and stop the whole process tree on exit. Never reuse or kill the user's own servers.
  • Keep secrets out of the script and its output.
  • Write the script outside the repo; commit it only if the user asks to promote it.
  • Timebox setup. If the app cannot run locally, say what was not checked instead of faking a pass.

Workflow

  1. Read the change: git diff against the base branch, plus untracked files.
  2. Map impact outward: changed symbols -> callers -> entry points (commands, routes, screens, jobs, hooks, config, migrations) -> user flows. Add adjacent flows sharing the changed code, config, or data; note skipped flows and why.
  3. List scenarios as setup / action / expected result, ordered by risk. Wait for the user to add or cut flows; when running unattended, proceed and include the list in the report.
  4. Search memory for known setup: npx ai-devkit@latest memory search --query "<project> smoke setup" --tags "smoke". Build once, then write the script from references/smoke-template.sh (CLI, API) or references/web-smoke-template.mjs (browser). Each check asserts output, files, HTTP status, or exit code, not just "it ran"; API checks also assert the body and one error case (bad input, missing auth). Print PASS/FAIL per check and exit non-zero on any failure.
  5. Run it and read the full output.
  6. On a failure, first rule out a wrong check (selector, timing, setup), then rerun it on the base commit in a separate worktree (dev-worktree) to separate regressions from pre-existing failures.
  7. Report with verify: scenarios, command, exit code, per-check result, pre-existing failures, flows not covered.
  8. Delete the script and sandbox, or offer to promote valuable checks into permanent tests or the project's release smoke suite. Store non-obvious setup: npx ai-devkit@latest memory store --title "<project> smoke setup" --content "<start command, port, health URL, seeding, traps>" --tags "smoke".

UI and Web

  • Use the harness's browser tool when available (for example Claude in Chrome or a Playwright MCP server); otherwise the web template, with Playwright installed in the sandbox.
  • Use a fresh browser context per check, and act like a user: navigate, click, fill forms, submit.
  • Assert visible content, URL, network status, and zero console errors, not just that the page loaded. Match visible elements by role or text; hidden duplicates (mobile menus, templates) cause false failures.
  • UI impact includes shared components, layouts, global styles, and every route rendering the changed component. Check narrow viewports or dark mode when layout or theming changed.
  • Save a screenshot per check as evidence, and read it before judging a failure.
  • Not covered: native mobile apps, desktop apps, pixel-level visual diffs. Report them as unchecked.

Stop

Pause when impacted flows are unclear, a scenario needs anything the Rules forbid, setup needs destructive actions, or failures come from the environment rather than the change.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.