mcp-release-qa
Skill21 starsChanged 27 days agoMITView on GitHub
Verify an MCP server before release by exercising a real protocol session, comparing runtime capabilities with source and documentation, testing failure paths, and recording reproducible evidence. Use when shipping or reviewing an MCP server, tool, resource, prompt, catalog, or install path.
2 other projects use a version of this, 1 of them edited; 1 new project in the last 4 weeks, against 0 the 4 before.
Adopted by
Surging: 1 new project took it in the last 4 weeks, none the 4 weeks before.thin data
Most recently taken up by
- github/awesome-copilotword for word, first seen Oct 25, 2026
- XGENT-ai/skillsedited, first seen Oct 13, 2026
Other versions2 projects use a version of this.
Word for word · 1
- github/awesome-copilot★ 41k100% same
Edited · 1
- XGENT-ai/skills★ 195% same
Wrote their own version · 0
None found.
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository16 other files it gives its agents.
- AGENTS.mdAGENTS.md
- .claude/skills/AGENTS.mdAGENTS.md
- .github/workflows/AGENTS.mdAGENTS.md
- helm/AGENTS.mdAGENTS.md
- scripts/AGENTS.mdAGENTS.md
- tests/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- .github/copilot-instructions.mdCopilot instructions
- dependabotSkill
- docs-sync-auditSkill
- github-actions-hardeningSkill
- mcp-implementation-security-reviewSkill
- mcp-security-auditSkill
- pytest-coverageSkill
- security-reviewSkill
- test-gap-auditSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
18 rules. About 0 are common, 15 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈15
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 8
Test the server that users will run.
- Distinctive: line 17
Run checks against a fresh server process built from the candidate revision.
- Very specific: line 18
Keep initialize, notifications/initialized, discovery, and invocation in the same session.
- Distinctive: line 20
Treat source registrations as implementation truth and public documentation as a contract that must match it.
- Distinctive: line 24
Do not invoke mutation-capable tools against production data.
- Very specific: line 40
Prefer repository-native commands.
- Distinctive: line 59
If the server cannot start from its documented instructions, report that as a release failure and preserve the startup error verbatim.
- Distinctive: line 64
Run this sequence through a real MCP client or the repository's integration harness:
- Distinctive: line 71
Call each supported discovery method:
- Distinctive: line 78
Follow pagination until no cursor remains when a list method is paginated.
- Distinctive: line 80
Do not send post-initialization requests through separate one-shot processes.
- Distinctive: line 146
Verify that each response has the correct request ID, a useful error message, and no successful side effect.
- Distinctive: line 153
When the project publishes an install command:
- Distinctive: line 155
Create a temporary destination outside the source checkout.
- Distinctive: line 156
Run the public install command exactly as documented.
- Distinctive: line 157
Start the installed artifact without relying on files from the source tree.
- Distinctive: line 159
Remove the temporary destination after preserving the command output.
- Very specific: line 195
Use FAIL for a server that cannot start, complete a valid session, keep the transport parseable, or safely reject invalid input.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
MCP Release QA
Test the server that users will run. A schema review or a passing unit test is not runtime evidence.
This skill complements security review. It focuses on protocol behavior, published-contract drift, transport correctness, and reproducible release evidence.
Rules
- Run checks against a fresh server process built from the candidate revision.
- Keep
initialize,notifications/initialized, discovery, and invocation in the same session. A new process is a new STDIO session. - Treat source registrations as implementation truth and public documentation as a contract that must match it.
- Record exact commands and raw responses. Do not replace missing evidence with "looks correct."
- Do not invoke mutation-capable tools against production data. Use fixtures, a sandbox, or stop and name the missing safe test environment.
- Derive the expected capability inventory from the candidate source on every run.
1. Establish the release surface
Identify:
- candidate commit and build command;
- server entry point and transport: STDIO, Streamable HTTP, or SSE;
- supported MCP protocol versions;
- source files that register tools, resources, resource templates, and prompts;
- generated catalogs, manifests, README tables, and install instructions;
- existing protocol, integration, and smoke-test commands.
Prefer repository-native commands. Inspect package.json, pyproject.toml,
Makefile, CI workflows, and contributor instructions before inventing a test
harness.
2. Start a clean server
Build the candidate and start the documented entry point with test-safe configuration. Capture:
- the exact command;
- commit SHA;
- environment variable names, with values redacted;
- stdout, stderr, and exit status;
- the endpoint or child-process transport used by the client.
For STDIO, stdout is protocol-only. Logs, banners, and stack traces belong on stderr. For HTTP transports, record the status, relevant MCP headers, and session identifier handling without printing credentials.
If the server cannot start from its documented instructions, report that as a release failure and preserve the startup error verbatim.
3. Exercise one complete session
Run this sequence through a real MCP client or the repository's integration harness:
initializewith a protocol version the server claims to support.- Confirm the negotiated version and advertised capabilities.
- Send
notifications/initialized. - Call
ping. - Call each supported discovery method:
tools/listresources/listresources/templates/listprompts/list
- Exercise at least one representative read-only item from every advertised capability class.
- Follow pagination until no cursor remains when a list method is paginated.
Do not send post-initialization requests through separate one-shot processes. That accidentally tests several incomplete sessions instead of one valid session.
4. Prove inventory parity
Build four inventories from current evidence:
| Surface | Evidence |
| Source | Registered tool, resource, template, and prompt definitions |
| Runtime | Results from the live discovery methods |
| Generated metadata | Catalogs, manifests, or generated indexes |
| Documentation | README, reference pages, and install output |
Compare by stable identifier. Report:
- source entries missing at runtime;
- runtime entries absent from metadata or documentation;
- stale names, descriptions, arguments, URIs, or prompt parameters;
- documented install commands that do not start the candidate server.
Regenerate derived files with the repository's own build command, then fail if the working tree still contains unexplained generated changes.
5. Check published contracts
For every discovered item, verify the runtime definition against its source:
Tools
- Name and description are stable and specific.
inputSchemadefines types, required fields, enums, and bounds where needed.- Unknown properties are rejected when the tool contract is closed.
- Mutation, idempotence, read-only, and open-world annotations match behavior.
- Successful calls conform to
outputSchemawhen one is published. - Errors are protocol errors or structured tool failures, not leaked stack traces.
Resources and templates
- URIs and MIME types match the registered definitions.
- Static resources are readable.
- Template parameters are validated before resolution.
- Missing or forbidden resources fail explicitly.
Prompts
- Required and optional arguments match discovery output.
prompts/getreturns usable messages for valid arguments.- Missing required arguments and unknown prompt names fail explicitly.
6. Test failure paths
At minimum, probe:
- a request before initialization completes;
- malformed JSON or an invalid JSON-RPC envelope;
- an unknown method;
- an unsupported protocol version;
- repeated initialization;
- unknown tool, resource, and prompt names;
- missing, extra, wrong-type, and out-of-bounds arguments;
- a request at the documented transport-size limit and one beyond it;
- a controlled internal failure with credentials and stack traces redacted.
Verify that each response has the correct request ID, a useful error message, and no successful side effect. For STDIO, also confirm every stdout line is a complete protocol message and a healthy session leaves stderr clean unless the server explicitly documents diagnostic output.
7. Smoke-test installation
When the project publishes an install command:
- Create a temporary destination outside the source checkout.
- Run the public install command exactly as documented.
- Start the installed artifact without relying on files from the source tree.
- Repeat initialization, discovery, and one read-only invocation.
- Remove the temporary destination after preserving the command output.
An install string that was only inspected is unverified.
8. Report the evidence
Use this format:
# MCP Release QA
Candidate: [commit]
Transport: [STDIO | Streamable HTTP | SSE]
Verdict: PASS | PASS WITH CAVEATS | FAIL
## Commands and results
- `[exact command]` — [exit status and result]
## Session transcript
- initialize: [result]
- discovery: [result]
- representative calls: [result]
- negative paths: [result]
## Parity
| Identifier | Source | Runtime | Metadata | Docs | Result |
|---|---|---|---|---|---|
## Findings
| Severity | Evidence | Impact | Narrowest fix |
|---|---|---|---|
## Missing evidence
- [check that could not run and why]Use FAIL for a server that cannot start, complete a valid session, keep the
transport parseable, or safely reject invalid input. Use PASS WITH CAVEATS
only for bounded documentation or metadata drift that does not misrepresent a
dangerous capability. Otherwise use PASS.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

