agentleFS
marksmayomarksmayo/qa-skills/resilience-prober-skill/SKILL.md

resilience-prober

Skill1 starsChanged yesterdayMITView on GitHub

Check how a running web app behaves when what it depends on goes wrong - slow or failing APIs, rate limits, expired sessions, hung requests, dropped connections, going offline, malformed payloads, one dependency down, flapping endpoints and failed script chunks. Each fault is injected in the browser only (Playwright routing), on a chosen step of a real user flow, after a clean baseline, with the expected outcome written before the run. Records what the user sees and loses - crashes, blank screens, endless spinners, silent failures, misleading empty states, lost input, no way to recover, retry storms - with screenshots, request logs and a standalone repro script. Never calls an app resilient.

  • Installs packages

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Run plan_scenarios.py --plan-only first: it prints the scenarios and an estimate (runs, page loads, requests at requests_per_load) against …

    line 82
  2. Distinctive:

    Fallback to a full page load: When a route payload fails (a 500, a truncated or HTML body), Next aborts it and loads the page as a …

    line 173

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Resilience prober

Mission

Find out whether a user still gets somewhere sensible when a dependency fails. Drive real flows, break one thing at a time at the network layer, and record the user-visible effect against an outcome written down beforehand. Merge scenarios with one root cause, rate them, and hand over a repro and a fix for each.

The UI crash hunter starts from the code and asks what makes the UI crash; this skill starts from the running app. It reuses the crash hunter's tiers (B1 app-fatal, B2 screen-fatal). The code security auditor sends attacker payloads to the server; this skill never sends anything unusual to the server. Native apps belong to the mobile app auditor.

Non-negotiable rules

  1. Faults are injected in the browser only. Playwright fulfils, delays or aborts the request, or takes the context offline. A write a scenario targets is answered in the browser, never forwarded.
  2. The baseline must be clean. Each flow runs twice without faults first; a failed or unstable baseline makes that flow's scenarios untested.
  3. Expected behaviour is written before the run (plan_scenarios.py), with the budgets: an error message within 10 s, no spinner past 30 s, input kept, recovery without a reload.
  4. Every finding comes from a run: the flow, step, fault, effect, a screenshot and the request log. Reading code gives a suspected note, never a finding.
  5. A degradation needs a user-visible effect. A console error alone is a note.
  6. Retries are measured: request counts and gaps from the run log.
  7. "Nothing broke" is not "resilient". The report lists the flows, endpoints and faults that ran, separately the planned faults that didn't, and what the browser can't reach (unreachable). Never "production ready" or "recovers gracefully from every failure".
  8. Synthetic data only in typed input and injected bodies; input is recorded as length and SHA-256.
  9. Safe targets. Loopback needs nothing; other hosts need authorised_by. The site's host is registered with the browser harness and every other host is blocked. Writes are blocked unless allow_form_submission is true (default false; the report records which). Set a request_budget for anything shared: one page load of a modern app is 50-80 requests, and the default plan is 2 baselines per flow plus --runs per scenario.
  10. A fault that never fired tested nothing. A scenario whose fault matched no request is not_triggered, has no effects and is not counted as run.
  11. A broken journey without faults is the first result. A failed baseline step is screenshotted and observed, and the headline names the flow.

Inputs

InputDefaultNotes
flowsrequiredflows.json: site, app root, flows with named steps (actions, a done selector, input, write), critical_path. A step may add done_not (selectors or text= that must be absent for the step to count as done) and optional_endpoints (globs for background widgets the step does not need)
reload_after_recoverytrueAfter recovery, reload and run the flow to the step again (reads under a failing or static-asset fault; writes are never sent again), recording requests to the endpoint and the Cache Storage entries; false turns it off
scenariosrequiredExplicit {flow, step, endpoint, fault, method?, headers?, resource_type?, from?} or {"auto": {"endpoints": {...}, "faults": [...]}}
faultsdata/faults.json standard setLatency, server errors, rate limits, auth expiry, timeouts, connection, offline, payload, partial, flapping, static assets
budgets10 s message, 30 s spinnerdata/indicators.json
indicatorsdata/indicators.jsonThe product's own wording and selectors, in flows.json (global or per flow): spinner_selectors (e.g. .animate-pulse skeletons), loading_words ("loading availability"), empty_state_words ("no results"), boundary_fallback_words and boundary_selectors (the app's error boundary, e.g. "this route hit turbulence"), error_selectors, error_words. Lists are added to the defaults; run.json records the merged set
authorised_by, auth, allow_form_submissionwrites blockedallow_form_submission defaults to false here; targets, allowed_hosts, host_map and a harness block pass through to browser_harness.py; auth: {"storage_state": file} runs signed in, with a session a person captured by signing in by hand (python scripts/browser_harness.py capture-session --url <app> --out <file outside the repo>; cookies, localStorage, IndexedDB and sessionStorage); the run stays read-only and records only that a session was used, never the file or its contents
request_budget / --max-requestsnoneRequests the browser may send to the site in the whole run
setup, init_scriptnoneActions after the first load (dismiss a cookie banner), JavaScript before every page (seed localStorage)
prefetchallowblock aborts prefetch requests (Next-Router-Prefetch, Sec-Purpose: prefetch), so a navigation fetches and a fault can reach it. It changes a Next.js app: see below
service_workersblockallow (in flows.json or on one flow) lets the app's service worker run: the first load waits for it and reloads once so it controls the page, as for a returning visitor; offline then reaches the worker, and a request it answers from its cache counts as reached by the fault. Route faults may miss requests the worker answers
nav_timeout_ms, step_timeout_ms, run_timeout_s30000, 10000, from the budgetsPage loads and history steps; a step before the target; the watchdog's limit for one baseline or scenario run (--run-timeout)
baseline_retries, baseline_retry_pause_s1, 5A failed baseline run is tried again after a pause (a loaded machine can miss a step's budget once); a flow that passed only on a retry is recorded retried
requests_per_load60For the planner's request estimate
unreachablenone[{flow?, step?, what, reason}]: data the browser cannot reach (fetched by a server component, cached at the edge); copied to the plan and the report
devicesdesktop ChromiumAdd a mobile profile for mobile web
harnoneReal traffic, to weight reach by each endpoint's share of requests

Workflow

run_resilience.py needs Playwright (pip install -r requirements.txt, python -m playwright install chromium; ask first). Everything else is stdlib.

python scripts/plan_scenarios.py flows.json --out <run>/scenarios.json [--plan-only] [--runs 2] [--repo <app source>]
python scripts/run_resilience.py flows.json <run>/scenarios.json --out <run> --runs 2 [--resume]
python scripts/classify.py <run>/run.json <run>/scenarios.json flows.json --out <run>/candidates.json
python scripts/build_report.py --run <run>/run.json --scenarios <run>/scenarios.json --candidates <run>/candidates.json \
    --flows flows.json --promote K1,K2 --repros <run>/repros --out <run>/resilience_report.json

Run plan_scenarios.py --plan-only first: it prints the scenarios and an estimate (runs, page loads, requests at requests_per_load) against request_budget, with over_budget when the plan won't fit. Look at the baselines before spending the scenarios: a flow whose baseline fails makes its scenarios untested. With --repo the planner reads a Next.js app's source (next_app.py) and lists the service workers it ships with their fetch strategies (service_workers in the plan); it warns when the run would block them.

The runner never blocks: every read of the page has a deadline (Playwright's page.evaluate has none, and a page whose navigation timed out never answers it), and a watchdog runs the work in a child process with a time limit per baseline or scenario run. A run past its limit is stopped with its browser, listed in <run>/hung.json, recorded errored (a baseline run failed) with the watchdog's reason, and the run goes on after it. run.json is written after every flow's baselines and every scenario (run_status: partial until the end); --resume continues a stopped run, and build_report.py and the validator refuse a partial one.

  1. Recon (optional, with source): the data layer, retry and timeout settings, error boundaries (the UI crash hunter's recon.py).
  2. Plan. Scenarios with their expected outcome, written before anything runs. Choose faults on every critical step's first API call, and partial and static-asset faults once per flow.
  3. Run. Baselines twice, then each scenario twice (reproduced or only observed). Faults on a flow's first step are in place before the page loads. Recovery uses the app's own retry control (an accessible name matching retry, try again, reload) or the step's actions again, never a reload; a step that only types (type-ahead) is recovered by re-sending the field's value (refill_input). A step whose only action is a navigation cannot be repeated without reloading, so its recovery is reload (not tried) and a recovery_not_tried note, never recovered (so is a back, forward or reload step: steps may use browser history, {"back": true}). Then the page is reloaded with the fault gone and the flow run to the step again: a step that makes no request to the endpoint after the reload, when the server never answered it well in the run, replays a failure the app stored (persisted_failure: Cache Storage, IndexedDB, localStorage). A reload that never loaded is recorded with its error and a reload_not_checked note, never read as a stored failure. A failed baseline run is tried once more after a pause; a flow that passed only then is retried (confidence medium at most).
  4. Classify. Effects per run, merged by root cause (one effect at one step and endpoint across faults; no recovery across a step's endpoints; retry storms per endpoint), rated by RUBRIC.md.
  5. Triage. build_report.py fills everything the run decides (scope, metadata, baseline, scenarios, coverage, unreachable, ratings, evidence, headline counts and confidence, provable limitations) and writes a guarded repro script per promoted candidate. Replace every TODO: promote, keep for review, or dismiss with a checkable reason (anything left in needs_review keeps the headline confidence at medium at best; emptying it is what lets it reach high) (an optional background widget left empty, an optimistic update that rolls back with a message). For each finding write the title, the actual behaviour against the expected, the user impact, a fix (a timeout, an error state, keeping input, honouring Retry-After, a chunk retry), a verification step, and check the repro script.
  6. Validate and render.
python scripts/validate_resilience.py <run>/resilience_report.json --run <run>/run.json --scenarios <run>/scenarios.json --candidates <run>/candidates.json
python scripts/render_report.py <run>/resilience_report.json [--brief]

A step's done must prove the data arrived, not only that the UI moved on. An app that shows its "ready" text whether or not the download succeeded makes every server-error scenario look passed. A failing fault on a read whose step still completed, with no error shown, is a misleading_state (shown_as_done) unless the step lists the endpoint in optional_endpoints; give such steps a done_not (the app's error text) or a done only real data can produce. When the run cannot judge something the agent can document (stand-in data for a binary download, a canvas screen with no text), add confidence_caps: [{"cap": "medium" | "low", "reason": ...}]: it lowers the headline confidence, never raises it.

A step whose baseline already shows under 20 visible characters (a canvas, a video) cannot be judged blank or crashed by its text: such runs give a baseline_blank note to check against the screenshots, and a screen with a visible video, canvas or large image (media_visible) is never blank_screen. A page error counts toward a crash only when it is new against the baseline and the app root emptied where the baseline's root had text. A static-asset fault that leaves the step incomplete is a chunk_failure even when the static HTML stays.

Offline (and any fault) on a step that becomes a full page load can end on the browser's own error page (chrome-error://): the run records browser_error_page with the URL and screenshots, and classify makes it a browser_error_page note (no app code ran there), never errored. Offline counts only failed requests the scenario targets (its pattern, method, headers and resource type), never every request that failed in the context, and a not_triggered reason says whether matching requests came before the fault, only after it, or never (a client cache). A failure that only hit images, fonts or media is optional for the step by default (the step's data didn't depend on it), and visible broken images are counted (broken_images note).

Edge cases: service workers are blocked unless service_workers is allow (they would answer before routing); with allow, plan offline scenarios on navigations to see what a returning visitor gets (the worker's cached page is a served_by_service_worker note, the browser's offline page a browser_error_page note); GraphQL needs faults matched on the operation name; WebSockets can be closed but not faked; third-party scripts are a natural partial failure; optimistic UI is only misleading if it doesn't roll back or warn. A fault on the page's own HTML document (a navigation, a server-rendered page) shows the catalogue's injected body, not the app: it becomes a document_fault note, never a blank screen or crash; aim faults at the requests the page makes instead. Type-ahead queries for different terms are not retries (a storm counts one URL and query, cache-busters dropped).

Next.js and other server-rendering frameworks:

  • Prefetch. Links in view are prefetched at page load (?_rsc=, Next-Router-Prefetch: 1); a fault installed for a later step misses them and the scenario is not_triggered. The faithful choice is "from": "load" with "headers": {"Next-Router-Prefetch": "1"} (the fault hits the prefetch, as a real visitor's would fail). "prefetch": "block" makes the navigation fetch ("headers": {"RSC": "1"}) but changes the app: Next logs "Failed to fetch RSC payload ... Falling back to browser navigation" for every link and turns client navigations into full page loads, so a loading.tsx skeleton never shows and a hang looks like "nothing happens". classify notes it (prefetch_changed_app) and build_report.py puts it in limitations.
  • Fallback to a full page load. When a route payload fails (a 500, a truncated or HTML body), Next aborts it and loads the page as a document, which the fault didn't touch: the right page shows. The run records the page's document loads (documents), and a failed request followed by a 2xx document load of the same path is a full_page_fallback note, not misleading_state. A step answered from the router's client cache makes no request, so the fault doesn't trigger (not_triggered, with the reason).
  • Matching. The document /features and its payload /features?_rsc=x share a path; tell them apart with headers or resource_type (document, fetch).
  • Server components fetch on the server; browser faults reach only the payload carrying the result. List that data in unreachable so the report says it wasn't testable.
  • Error boundaries (error.tsx) catch the error, so there is no page error: put the app's boundary wording or selector in indicators (boundary_fallback_words, boundary_selectors), or a boundary render is classed as a silent failure instead of a crash (Fatal).
  • Skeletons (.animate-pulse) and text loaders go in spinner_selectors / loading_words, empty states ("No results") in empty_state_words, or endless loading and misleading empty states can't fire.
  • Watch window. Server-rendered pages show their main content at once; the runner keeps watching for the scenario's whole window (10 s, 30 s while loading shows) for failing faults, so a widget that never stops loading is still caught.

Output fields

scope (faults that ran, faults_untested), metadata (browser, requests sent, request budget, whether writes were allowed, blocked requests, runs per scenario, service workers), baseline (per flow: status, retried when a run passed only when tried again, and for a broken flow the failed step, what it showed and screenshots), scenarios[] (with expected outcome, status ran | untested | errored | not_triggered, reason, effects, screenshots), findings[] (effect, impact, likelihood, reach, score, priority, confidence, floors, endpoints, faults, scenarios, expected, actual, user impact, evidence, repro, fix, verification), notes[] (console_only, document_fault, recovery_not_tried, baseline_blank, browser_error_page, full_page_fallback, served_by_service_worker, broken_images, observation_unavailable, reload_not_checked, prefetch_changed_app), confidence_caps[] (optional: {cap, reason}), coverage (per fault group: ran, untested, errored, not_triggered), unreachable[], headline, needs_review[], triage[], limitations[]. run.json keeps each matching request with its query (sensitive values redacted), whether the injector answered or held it, and its response time.

What the validator enforces

run.json is complete (not partial). No TODO is left. Scenarios match the plan and the run, their expected outcome predates the run, and nothing ran for a failed baseline; a scenario whose fault never fired is not_triggered. The baseline section equals run.json, its screenshots exist, and the headline names every broken flow. scope.faults holds only faults that ran, faults_untested the rest; metadata and unreachable equal the run and the plan. Findings match their candidates, cite runs that triggered the fault, come from scenarios that ran with screenshots and request logs, and have repro scripts. A P0 is reproduced. Retry storms carry counts and timings; lost input carries lengths and hashes only. Console-only items are never findings. Coverage, the headline and confidence (low when under half the scenarios ran, a critical baseline failed, or scenarios ran once; medium at most when a baseline passed only on a retry; lowered by any confidence_caps) are recomputed, and no summary, limitation or finding claims resilience, production readiness or graceful recovery from every failure.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.