agentleFS
Mohammad-AlbabaMohammad-Albaba/api-contract-test-skill/SKILL.md

api-contract-test

Skill1 starsChanged 5 months agoMITView on GitHub

Test an API layer directly as a contract — validate live responses against an OpenAPI/JSON Schema, check status codes, authentication and per-endpoint authorization (object-level authZ / IDOR / BOLA), idempotency, pagination, input boundaries, and versioning drift — independent of any UI. Use when asked to test an API, validate API contracts, check endpoint authorization, find schema drift, test idempotency or pagination, verify status codes, or quality-check a backend service before or alongside UI testing.

  • Reads credentials

1 other project uses a version of this.

Adopted by

Most recently taken up by

  • Other versions1 project uses a version of this.

    Word for word · 0

    None found.

    Edited · 0

    None found.

    Wrote their own version · 1

    1. ayouaiyouwei-arch/claude_pm_workflows★ 1same name

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

The file

We split this skill into the rules it gives an agent

12 rules. About 0 are common, 10 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈10

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Do not use it for load/performance testing or as a full penetration test.

    line 32
  2. Distinctive:

    ask for required missing information before active testing

    line 42
  3. Distinctive:

    keep coverage explicit: which endpoints are tested, which are not, and why

    line 44
  4. Distinctive:

    Use the smallest mode that answers the request.

    line 51
  5. Distinctive:

    Before any active API call, establish the boundary.

    line 78
  6. Very specific:

    Secrets handling: read credentials from environment variables or a local untracked config (e.g.

    line 87
  7. Distinctive:

    If any required item is missing, pause before active checks and ask one focused question with a safe default.

    line 89
  8. Very specific:

    Every drift or break finding requires concrete evidence, not just a rationale:

    line 117
  9. Distinctive:

    Keep raw request/response captures under a run folder and reference them from the report.

    line 124
  10. Distinctive:

    Treat a removed/renamed field or a narrowed type as breaking.

    line 155
  11. Distinctive:

    Never write to production.

    line 161
  12. Distinctive:

    Never store secrets in the repo or in saved evidence.

    line 162

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

API Contract Test

On invocation, first tell the user in one line what this skill does, then ask the intake question below before any active call. Opening line to show:

🔌 API Contract Test — I test your API directly as a contract (schema, status codes, authZ/IDOR, idempotency, pagination, input limits, drift), independent of any UI, on a safe staging target. First I need to know what to test and where.

Then ask the Intake question (see "Intake — ask before starting" below) to collect the spec/collection, the staging base URL, and auth — and do not run any live call until scope is confirmed.


A workflow for testing an HTTP/JSON API layer as a contract, independent of any user interface. It verifies what the backend actually returns and enforces — not what a UI happens to render. Use it for web, mobile, and standalone API products.

A silent API break — a renamed field, a leaked record, a double charge — usually hurts trust, money, or integrations long before a human notices it in the UI. This workflow catches those at the source.

Explain findings in plain product language. The reader may be a PM or Quality Lead, not a backend engineer: say what broke for the user or the business, not only which field changed.

When To Use

  • "test the checkout API against its OpenAPI spec on staging"
  • "validate the orders endpoints for schema drift"
  • "check that role=user gets 403 on DELETE /admin/orders/{id}"
  • "verify POST /payments is idempotent with a repeated idempotency-key"
  • "test pagination and input limits on GET /products"
  • "quality-check the backend contract before we ship"

Do not use it for load/performance testing or as a full penetration test. It is contract & behavior testing of the API surface.

User Knowledge Assumption

Assume the reader may not be an API specialist.

Do:

  • explain each check in one plain sentence before or while running it
  • translate technical drift into business impact (who is harmed, what value is at risk)
  • ask for required missing information before active testing
  • suggest safe defaults when the user is unsure
  • keep coverage explicit: which endpoints are tested, which are not, and why
  • separate confirmed contract breaks from untested scope

Define any acronym the first time it appears: schema drift (the response no longer matches the agreed shape), IDOR / BOLA (one user can read or change another user's object), idempotency (calling the same write twice has the same effect as once).

Operating Modes

Use the smallest mode that answers the request.

  • Spec check: a spec (OpenAPI / Swagger / JSON Schema / Postman collection) exists — validate live responses against it and report drift. Default when a spec is available.
  • Endpoint path: the user names specific endpoints or a feature (checkout, profile, admin approval) — test those across the contract dimensions below.
  • Discovery: no spec and no named endpoints — derive the surface from a base URL, captured traffic, or docs; list what was found and state what still needs a spec or scope.
  • Regression retest: a known contract or prior finding is given — verify only that contract unless new risk is obvious.

Intake — ask before starting

Before doing anything, present a single structured question so the user can hand over the contract source and target in one step. Use the runtime's question UI if available; otherwise ask in plain text. Offer these as the choices for "How will you give me the API to test?":

  1. Postman / Insomnia collection — attach or give the path to the exported collection (.json). I'll derive endpoints + example requests from it.
  2. OpenAPI / Swagger spec — attach or give the path/URL to the spec. I'll generate a baseline with tools/gen_baseline.py, then test against it.
  3. Staging base URL only — give me the base URL (e.g. https://staging.example.com/api) and I'll discover/derive the surface, stating what still needs a spec.
  4. Existing baseline — point me at a saved contracts/<service>/baseline.json to retest against.

Alongside the source, collect in the same intake (with safe defaults offered):

  • Base URL — staging or an authorized test environment only (never production for mutating calls).
  • Auth — how to authenticate (bearer token, header, or login flow). Secrets via env vars only — e.g. export TOKEN=…; never pasted into the repo or the spec.
  • Mutation allowed? — may I send POST/PUT/PATCH/DELETE, and against which test records? Default: read-only unless told otherwise.
  • Scope — which endpoints/feature to focus on first. Default: smallest safe set, then widen.

If the user is unsure, suggest: "attach your Postman collection or OpenAPI spec + give me a staging base URL and a token in an env var; I'll start read-only on the smallest set." Do not run any active call until the base URL, auth, and mutation boundary are confirmed — otherwise pause and ask one focused question.

Authorization & Scope Gate

Before any active API call, establish the boundary. Collect or infer:

  • target environment / base URL — must be staging or an authorized test environment; never run mutating calls against production
  • the spec source if any (OpenAPI/Swagger URL, JSON Schema, Postman collection), or "none"
  • endpoints or feature under test
  • allowed accounts, roles, tenants, and how to authenticate (token, header, login flow)
  • whether mutating calls (POST/PUT/PATCH/DELETE) are permitted, and on which records
  • forbidden actions: real payments, production writes, email/SMS floods, third-party live calls, destructive deletes

Secrets handling: read credentials from environment variables or a local untracked config (e.g. .env, a secrets manager) — never hardcode them into the repo, the spec, or saved evidence. Redact tokens and keys from every captured request/response.

If any required item is missing, pause before active checks and ask one focused question with a safe default.

Control Plane

API requests are made over HTTP — no browser or device is required, which makes this workflow light and fast. Every verdict must come from a fresh live call made this run — captured pairs from prior runs are planning input only (they tell you what to expect and how to shape the call), never a substitute for re-driving the request. If a needed call cannot be driven live this run, report BLOCKED with the reason rather than falling back to a cached result. Prefer, in order:

  1. The bundled executable core (tools/run_contract.py, stdlib-only) — generate a baseline with tools/gen_baseline.py from an OpenAPI spec, then run the auto-checkable dimensions repeatably. See tools/README.md.
  2. A configured API/HTTP MCP server or HTTP client tool, if available in the runtime.
  3. Direct authenticated HTTP requests via the runtime's request capability, using credentials resolved from env/secrets only.

Prior captured request/response pairs may seed expectations or replay setup, but the result you classify must be a fresh response from this run. Record the auth method used (role/tenant, not the secret) in the run evidence.

Contract Dimensions

For each in-scope endpoint, evaluate the dimensions that apply and label each result pass / drift / break:

  1. Schema conformance — response body matches the agreed schema: required fields present, types correct, no unexpected null, no removed/renamed fields, enums in range.
  2. Status & error contract — correct status codes for success, validation error, auth failure, not-found; error bodies follow the agreed error shape.
  3. AuthN / AuthZ — protected endpoints reject missing/expired tokens; object-level authZ (IDOR/BOLA): a user cannot read or mutate another user's/tenant's object; role boundaries hold (e.g. user gets 403, not 200, on admin routes). API-tier authZ is often faster and more reliable to prove here than through a UI.
  4. Idempotency & side effects — repeated writes with the same idempotency key do not double-apply (no double charge, no duplicate order).
  5. Pagination & filtering — limits/offsets/cursors behave; no over-fetch; counts and next links are consistent.
  6. Input boundaries — required-field omission, wrong types, oversize payloads, and injection-shaped input are rejected gracefully (no 500, no leak in error text).
  7. Versioning / drift vs baseline — capture the current response with a fresh live call this run, then diff it against the last saved contract baseline; flag breaking changes (removed field, narrowed type, changed status) vs additive ones. The saved baseline is the comparison reference, never the verdict — never diff two saved files and call it a result.

Scope to the smallest relevant set first; widen only if asked or if risk is obvious. If coverage is bounded (top-N endpoints, no mutation), state plainly what was not tested — silent truncation reads as full coverage when it is not.

Evidence Rules

Every drift or break finding requires concrete evidence, not just a rationale:

  • the request (method + path + relevant headers, secrets redacted) and the response (status + body excerpt)
  • the expected-vs-actual diff (schema mismatch, wrong status, leaked object id)
  • a one-line business impact in plain language
  • the auth context used (role/tenant, not the token)

Keep raw request/response captures under a run folder and reference them from the report.

Workflow

  1. Confirm scope & auth via the gate above.
  2. Resolve the contract — load the spec, or derive expected shape from a baseline / captured traffic; state which source was used.
  3. Run dimension checks per endpoint, smallest scope first; capture evidence as you go.
  4. Classify each result pass / drift / break; separate confirmed from untested.
  5. Report in plain language: summary verdict, a table of endpoint × dimension × result, evidence references, and the single highest-risk break.
  6. Persist — save the run and update the contract baseline so the next run can detect drift.
  7. File issues — for each confirmed break/drift the user approves, file into the configured issue tracker with evidence and priority. If the user asked for report-only / dry-run, return findings and recommended priority without filing. If filing preference was unspecified, ask before creating issues.

Report Shape

Return a PM-readable brief:

  • Verdict — clean / drift found / breaking change found
  • Coverage — endpoints and dimensions tested; what was not
  • Findings table — endpoint | dimension | result | business impact | evidence ref
  • Top risk — the single most damaging break and the fix or regression check that closes it
  • Next step — file now / baseline updated / scope to widen

Baseline & Memory

Save a contract baseline (the agreed schema + status map per endpoint) alongside the project so future runs compare against it and flag drift. A simple convention:

contracts/<service>/baseline.json     # expected shapes + status map
contracts/<service>/runs/<timestamp>/ # captured req/resp + findings

Treat a removed/renamed field or a narrowed type as breaking; an added optional field as additive.

The baseline is comparison input, never proof — a verdict always requires fresh live responses from this run. If a run cannot complete (target unreachable, auth fails, scope blocked), report BLOCKED with the reason; do not fall back to the baseline or a prior run as the answer.

Boundaries

  • Never write to production. Never run real payments, floods, or destructive deletes.
  • Never store secrets in the repo or in saved evidence; read from env/secrets and redact.
  • This is contract & behavior testing, not load/performance testing and not a full pentest.
  • Prior run history plans the path; it is never the verdict. Always run fresh live calls, and report BLOCKED with the reason if a run cannot complete.

References

  • references/contract-dimensions.md — expanded checklist per dimension with example checks.
  • references/report-template.md — the report skeleton to fill in.
  • references/example-run.md — a full worked example against the sample baseline.
  • tools/ — the stdlib-only executable core: gen_baseline.py (OpenAPI → baseline), run_contract.py (run dimensions on fresh live calls), selftest.py (offline checks). See tools/README.md.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.