adversarial-tests
Skill24 starsChanged yesterdayMITView on GitHub
How to run (or commission) an adversarial security-test pass against this codebase without producing 3000 lines of repetitive matrix tests. Auto-loads when the user mentions "red tests", "adversarial tests", "security test pass", "find security gaps", or is about to spawn a sub-agent to author *_security_test.go files. Encodes the "property × surface, not case × file" rule, the naming + triage policy, and a ready-to-paste prompt template for the next pass.
In this repository16 other files it gives its agents.
- AGENTS.mdAGENTS.md
- framework/agents.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- app-introspectSkill
- chaos-testSkill
- component-buildSkill
- eval-reportSkill
- gofastr-docsSkill
- gofastr-hostSkill
- gofastr-mcp-debugSkill
- gofastr-uiSkill
- log-debugSkill
- plain-wordsSkill
- primitives-sweepSkill
- review-gateSkill
- twin-sweepSkill
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 6 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈6
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 44
Ask "what surfaces does this property apply to?" before "what attack strings can I generate?".
- Distinctive: line 49
When you commission a pass, ask the agent to scan for *each* of these properties across *every* call site, not for "more cases":
- Distinctive: line 69
When you commission the next pass, the prompt MUST mention this table and tell the agent to *find new surfaces*, not *new attack strings*.
- Distinctive: line 78
Where else does this property apply?: If only one place, it's either a real one-off or wrong-layer.
- Distinctive: line 116
Every test the pass *deletes* or *weakens* (rather than fixing production for) records its *why* in exactly two places: the commit message …
- Distinctive: line 176
When you want to run another adversarial pass, paste this as the agent's brief (edit the scope line):
- Very specific: line 254
After the pass: run ./scripts/test-all.sh and confirm exit 0.
- Very specific: line 304
CLEARED: only when the deep pass, the breadth pass, go vet, and govulncheck are ALL dry/clean on it.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Adversarial test pass: how to commission one without drowning
The previous adversarial pass authored 32 _red_test.go files / ~3000
lines, of which ~10 were genuinely unique bugs and ~17 were
distinct production fixes; the rest was 60-case matrices testing
the same property at different surfaces, plus a handful of wrong-
layer tests that contradicted documented contracts.
This skill exists so the next pass produces signal, not volume.
The one rule
Adversarial tests are organized along properties × surfaces, not cases × files.
A "property" is the underlying invariant: no control bytes in outbound headers, every URL field allow-lists schemes, every in-process CRUD method fails closed without tenant context. A "surface" is each place the property must hold (each header writer, each URL field, each method).
The wrong shape (what we got last time):
60 cases × 1 surface per test file, repeated across N files,
where 50 of the cases are noise that test the same byte range.The right shape:
For each property:
- 3-5 case shapes (the distinct attack classes, not 60 variants)
- Asserted at every surface where the property must hold (loop over surfaces, not over cases)
- One test file per property family, NOT one per surfaceA test file with 60 cases is almost always asking the wrong question. Ask "what surfaces does this property apply to?" before "what attack strings can I generate?".
Properties × surfaces this codebase keeps re-encountering
When you commission a pass, ask the agent to scan for each of these properties across every call site, not for "more cases":
| Property | Common surfaces | Reference fix |
| URL scheme allow-list (surface-specific: anchors may allow http(s) / relative / fragment / mailto / tel; head/media/header use narrower policies) | any field that flows to <a href>, <img src>, <form action>, <link href>, Link: response header, OpenAPI servers[].url, SEO meta tags, Image/File entity fields | core-ui/urlsafe/urlsafe.go::Clean / CleanAnchor (with framework/ui/safety.go::safeImageURL), framework/uihost/uihost.go::isSafeHeadURL, framework/crud/crud_upload.go::isSafeMediaURL, framework/experimental/apiversions/version.go::safeReplacementURL |
C0 / DEL sanitation on outbound strings (protocol-specific, NOT uniform: SSE scrubSSEDataLines strips CR+NUL only to preserve LF framing; safeLogMethod/safeLogPath fast-path returns clean input unchanged) | response header values (Content-Type, Access-Control-*, Link:), log attribute values (method, path), SSE field bodies, DSL opaque literals, route-group prefix | core/textsafe/controlbytes.go::SanitizeControlBytes (called from core/handler/respond.go, core/middleware/cors.go::sanitizeCORSTokens, framework/dsl/dsl.go and framework/routegroup/group.go), core/textsafe/controlbytes.go::ScrubControlBytes (log sinks), core/middleware/logging.go::safeLogMethod (test the fast-path too), core/stream/sse.go::scrubSSEDataLines |
| Strict JSON top-level parsing (reject duplicate / case-folded / unknown keys); no-op for non-struct-pointer destinations and non-object bodies | every Bind consumer decoding a top-level JSON object into a struct pointer | core/handler/bind.go::validateBodyKeys |
| Fail-closed scoping (multi-tenant + owner-required) on in-process paths | every CRUD method that touches DB state (not just the HTTP path; middleware doesn't protect in-process callers) | framework/crud/owner.go::requireOwnerContext + framework/crud/owner.go::requireTenantContext + every method in framework/crud/crud_api.go |
| Forensic completeness on rollback / soft-delete | batch envelope, upsert ON CONFLICT, audit hook | framework/crud/crud_batch.go::scrubRolledBackData, framework/crud/crud_upsert.go::errSoftDeletedResurrection |
| Panic isolation at extension points | every place a third-party callback runs (hooks, plugins, custom handlers) | framework/hook/hook.go::runHookSafely |
| Panic-payload scrub at log sinks (a recover() value is attacker-influenced text: control bytes, bidi, secrets) | every recover() whose value, or an fmt.Sprint/%v of it, reaches slog/log/fmt | core/textsafe.Recovered |
| Per-principal seat caps on long-lived streams | every SSE / WebSocket / Hijack park loop (a2a, rtc, harness rest/ws/mcpserver, kiln events, crud event stream, webmcp) | core/stream/seats.go, framework/experimental/harness/control/seats.go |
Cache-Control: no-store on per-user 2xx responses | every handler that resolves a principal and writes a body | battery/auth writeCredentialHeaders, framework/crud responses, the nostore analyzer |
| Owner-only file mode on overwrite (a 0600 literal in os.WriteFile applies only at create) | every state/secret file written more than once | internal/fileperm.WriteOwnerOnly / SeedOwnerOnly |
| Strict decode at every inbound seam (duplicate/case-folded keys at any depth, trailing data) | every json.Unmarshal/Decoder.Decode of a client-borne body, frame, cursor, or envelope | core/handler.UnmarshalStrict / DecodeStrict / CheckObjectKeys |
| Fenced terminal settles (no status regression, no unfenced claim-state write) | queue reclaim, webhook inbound status transitions, outbox leases | battery/queue releaseClaim, battery/webhook terminal fence, GOFASTR1409 |
| Deadlines on probes and children | every http.Client construction, every exec.CommandContext with captured output | hygiene clienttimeout, nowaitdelay |
| Credentials never in argv | every CLI flag that accepts a secret | GOFASTR_AUDIT_PASSWORD / GOFASTR_HARNESS_SECRET + the - stdin marker |
When you commission the next pass, the prompt MUST mention this table and tell the agent to find new surfaces, not new attack strings.
Triage rubric (apply per test, before writing or accepting)
For every adversarial test the agent proposes:
- What property does this assert? State it in one sentence. If you can't, the test is wrong-shaped.
- Where else does this property apply? If only one place, it's either a real one-off or wrong-layer. If many, the agent should be testing every surface, not 60 cases on one.
- Is this contradicting a documented contract? Grep for the
nearest sibling
_security_test.goand any doc comment on the exercised function. If yes → flag for human (this is where the "flip the contract" vs "delete the test" decision lives). - Is the test asserting on developer input or attacker input? Developer-supplied configuration (OpenAPI path keys, server URLs passed by the host app, etc.) has a different threat model than request-borne input. Tests that treat developer input as attacker input are wrong-layer → delete.
- Cap case count at ~5 per property. 1 happy path + 3-4 distinct attack shapes (e.g. CR, LF, NUL, DEL, not 60 random control bytes). If you need 60 to feel covered, you're confusing coverage with count.
Naming + location policy
The policy:
- An open finding is a
*_red_test.goprobe tagged//go:build redthat asserts the secure behaviour and fails until the fix lands (make red-testsruns the probes in packages your change affects,GOFASTR_TEST_ALL=1 make red-testsruns every probe;go test ./...never sees them). The fix turns the probe into a permanent_security_test.gotest. - Permanent file suffix:
_security_test.go, with no_red_infix - Drop the redundant
<pkg>_prefix when the file lives in a package of that name - Function names ≤ 40 chars stating the behaviour
(NOT
TestFoo_RedRejectsBarBlahMatrixessays) - One file per property family, not one per case shape
- Merge into the closest existing
_security_test.gosibling unless the topic is genuinely new
Record decisions where they're enforced, not in a ledger
Every test the pass deletes or weakens (rather than fixing
production for) records its why in exactly two places: the commit
message that deletes/weakens it, and, when a surviving sibling test
carries the contract, a short comment on that sibling naming what it
supersedes. No separate ledger file: the pinning *_security_test.go
IS the record, and git log -p is the audit trail. A permanently
t.Skipped test is never acceptable. Delete it and leave a comment
where the contract is actually tested.
Where to start a pass
The list below is the 2026-07-25 pass's own never-looked list: no pass
had run all four signals over these surfaces at that date. Later passes
have covered some of them (core/upload and battery/auth OIDC now
carry _security_test.go files), so check each entry for an existing
_security_test.go before treating it as never-looked, and delete the
entries a pass clears. A package that carries _security_test.go
siblings has been looked at for the properties those tests name, not
for every property in the table above.
- Auth:
battery/auth/oidc.go+oidc_jwks.go(1000+ lines: JWKS fetch,iss/aud, nonce,email_verified; oidc allowshttpfor a localhost issuer, so check a host that merely resolves to loopback),entity_oauth_links.go,oauth_token_store.go,battery/adminentity-screen handler bodies (gating verified, internals not),framework/access/store.gofanout/refresh race, thedecidechain. - Data:
core/upload/,framework/{file,image,openapi,sdk,sdkdocs, tx.go,hook,typed_hooks},framework/{db,pagination,datexport},core/schemabeyond the Pattern note,crud_batch/stream/events. Thesqlite/**in-house engine was REMOVED from the tree (2026-09 round-5 recon confirmed: only the stdlib driver alias + DSN compat shim remain, sibling-tested); the untrusted-.dbrobustness surface now lives in modernc.org/sqlite — out of repo scope. - Agent surface:
framework/experimental,kiln/{expr,effect,render}CSP,cmd/gofastr/internal/{packcmd,agentscmd,docscmd},core/moduleproto/{peer,handshake,methods}.go,framework/agentsinv. - Browser: most
core-ui/runtime/src/*.js(lightbox, sortablelist, panehost, carousel, combobox, tree, toasts, …), the formercore-ui/patterns/**(deleted; its replacements live in framework/ui + framework/headless),core-ui/{seo,app,registry,di,compute,interactive},framework/ui(100+ files, only the URL-guard sites were read),core/fuzzy. - Infra:
framework/migratelock/checksum/dirty + tenant_id injection internals,framework/{cron,event,outbox,fanout,lifecycle}leasing SQL / relay state machine,core/{router,config,featureflag, i18n}+framework/i18n.go(locale-tag → file lookup traversal),core/handler/{bind,respond,context}deep pass,battery/queuescheduler.
Surfaces that pass CLEARED the same day, so do not re-derive them: kiln journal
replay, core/markdown XSS, the service worker's fetch(originalRequest)
(the shape the 2026 Angular-SW CVE class needs you to get wrong),
sandbox _other.go failing closed, QuoteIdent/SafeIdent, webhook
delivery + harness/webfetch.go dial-time Control re-check, core/yaml,
core/moduleproto framing, CORS, the XFF rate-limit key, static path
traversal, the Timeout middleware race, log injection, cache cross-user
replay, kiln request→exec, and the codegen file-write sink.
Ready-to-paste prompt for the next pass
When you want to run another adversarial pass, paste this as the agent's brief (edit the scope line):
You are running an adversarial security test pass against the
GoFastr codebase. Read
`.claude/skills/adversarial-tests/SKILL.md` first; it
encodes the policy you must follow. Prior keep/flip/delete decisions
live in git history and in comments beside the surviving tests.
Scope: <e.g. framework/crud + framework/uihost; or "any package
touched on this branch">
Rules:
- Organize by **property × surface**, NOT case × file. For each
security property you find, scan EVERY surface where it applies
and assert at each one, not 60 cases at one surface.
- Cap case count at ~5 per property: one happy path + three or four
distinct attack shapes covering the threat class. No 60-row
matrices.
- Naming: an open finding is a `_red_test.go` probe behind the `red`
build tag; its fix converts it to a `_security_test.go` test.
Function names ≤40 chars stating the behaviour.
- Before authoring a test, grep for the closest existing
`_security_test.go` sibling. If one exists, merge into it
unless the topic is genuinely new.
- Before authoring a test that contradicts a documented escape
hatch or behaviour comment, FLAG it for the user instead of
silently asserting the opposite. The doc comment is the
contract until the user explicitly flips it.
- Tests that treat developer-supplied input (OpenAPI path keys,
server URLs from host app config, etc.) as attacker input are
wrong-layer; do not author them.
- For every test that the pass deletes or weakens, put the *why* in
the commit message and a short comment beside the surviving sibling
test. Never leave a permanently-skipped test.
For each finding, produce:
- The underlying property (one sentence)
- Every surface where the property applies
- The production fix (or `weaken` / `delete` decision)
- The new short test name
Pre-existing properties to extend coverage on (don't re-derive; pin
to known surfaces only). See the table in
`.claude/skills/adversarial-tests/SKILL.md` for symbol references:
- URL scheme allow-list
- C0 / DEL strip on outbound strings
- Strict JSON top-level parsing
- Fail-closed multi-tenant / owner scoping on in-process paths
- Forensic completeness on rollback / soft-delete
- Panic isolation at extension points
- Panic-payload scrub at log sinks (textsafe.Recovered)
- Per-principal seat caps on long-lived streams
- no-store on per-user 2xx responses
- Owner-only file mode on overwrite (fileperm.WriteOwnerOnly)
- Strict decode at every inbound seam (any-depth key walk)
- Fenced terminal settles (queue/webhook/outbox)
- Deadlines on every HTTP client and exec child
- Credentials never in argv (env/stdin channels)
Hard NO:
- No `_red_test.go` file left behind once its finding is fixed
- No 60-case matrices
- No Module_Feature_Precondition_Assertion essay test names
- No silent contradictions of documented contractsHow to invoke
Two ways:
- Paste the prompt block above into a sub-agent task (this is the common case; adversarial passes are heavy enough to warrant their own agent context).
- Drive the pass yourself by following the rules in this file. The prompt block IS the policy, just internalized.
After the pass: run ./scripts/test-all.sh and confirm exit 0;
audit find . -maxdepth 3 -type f -size +500k … for stray binaries
per AGENTS.md; review the pass's commit messages for the
keep/flip/delete rationale.
Two-pass protocol (MANDATORY)
A security pass is run by two passes with different jobs, never one. This is not optional: a single-pass audit does not get to mark anything clean.
Both profiles run on the same model: each pins model: opus in its
frontmatter. A breadth pass on a cheaper tier is not acceptable, because
a breadth pass whose silence means nothing cannot be half of a
clean-gate.
| Role | Profile | Model | Job |
| breadth | sec-recon | opus | walk every file against the property×surface checklist, emit all candidates, report coverage honestly |
| depth | sec-auditor | opus | threat-intel anchor → deep discovery (authz/TOCTOU/state-machine) → refute + fix + rule on every candidate |
Why two passes: exhaustive breadth and deep reasoning are different jobs; one agent doing both drops coverage the moment a thread gets interesting. The split is job, not capability.
Honest limit. With both halves on the same model, the two passes share a lineage and a tier: their blind spots overlap almost completely. Agreement between them is weak evidence, and one confirming the other is not independent verification. Say so out loud rather than banking it. The diversity that does not come from a model is therefore load-bearing; it is the only diversity in the pass:
- Web search (
sec-auditorjob 1) injects vuln classes from the live CVE/advisory corpus that no Claude tier would spontaneously recall. Anchor every pass to current CWE Top 25 / OWASP ASVS / recent Go + primitive-specific advisories, and turn the result into a checklist delta both profiles sweep. - Deterministic tools:
go vetandgovulncheck(both first-party Go-team; no third-party analyzers) are the one second opinion that shares no blind spot with any LLM.govulncheckis a separate install (golang.org/x/vuln/cmd/govulncheck); run it viamake vulncheck, which fails closed with the install command if the binary is missing. - Proof: a failing test, a
curl, a real browser. Evidence is not a second opinion, it is the end of the argument. Prefer it to every model judgment, including your own.
The clean-gate (the non-optional invariant): a surface is
CLEARED only when the deep pass, the breadth pass, go vet, and
govulncheck are ALL dry/clean on it. One signal's silence never
clears a surface. A category nobody ran all four against is
"never-looked", not "clean". Track the difference, and publish the
not-looked list alongside the findings.
Cross-verify both directions: recon finds → auditor refutes (kills false positives); auditor finds → prove it, then run the deterministic tools where analyzable (catches over-reads). A same-model re-derivation is a sanity check, never a promotion.
Spawn the two profiles with the Agent tool (subagent_type: sec-recon
/ sec-auditor; the model: is pinned in each profile's frontmatter)
or drive them as stages of a Workflow. When running several scopes at
once, give each its own scope boundary so the passes don't collide.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

