agentleFS
vllm-projectvllm-project/speculators/.claude/skills/pr-review/SKILL.md

pr-review

Skill945 starsChanged 11 days agoApache-2.0View on GitHub

Review a GitHub PR with design-first analysis, posted as a GitHub review.

195 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use oss-pr, pr-automation and 10 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • CES-LtdSkill

    oss-pr

    Use when creating a pull request, after committing changes, or when user invokes /oss-pr. Covers branch management, quality checks, commit, push, and PR creation.

    Also in 3 projects that use this

  • CES-LtdSkill

    pr-automation

    PR Automation Orchestrator: poll open PRs, check CI, run review, fix, and merge eligible PRs. Use when: (1) Invoked by daemon via scripts/pr-automation.sh, (2) User says "/pr-automation".

    Also in 3 projects that use this

  • jtenniswoodSkill

    expert-reset

    Reimagine this repository from first principles using the current product and user experience as the target. Use when the user says "/expert-reset", asks how the project should be rebuilt if starting again, wants bold architecture or component-structure recommendations, or wants a research-only reset plan unconstrained by current implementation details.

    Also in 3 projects that use this

  • jtenniswoodSkill

    updates

    Summarize user-facing project updates from the last 7 days, ordered by importance and written as one-sentence "Feature: description" entries. Use when the user says "/updates", asks what changed recently, or wants a concise list of recent features, device support, fixes that users will notice, or other product value.

    Also in 3 projects that use this

  • openchamberSkill

    clack-cli-patterns

    Use when creating or modifying OpenChamber CLI commands, prompts, terminal output, non-TTY behavior, `--quiet`, or `--json` behavior.

    Also in 3 projects that use this

  • openchamberSkill

    drag-to-reorder

    Use when implementing or modifying OpenChamber sortable or drag-to-reorder behavior, especially `@dnd-kit`, touch/mobile interactions, variable-width items, or wrapping layouts.

    Also in 3 projects that use this

  • +6 more from openchamber
  • Other versions195 projects use a version of this.

    Word for word · 0

    None found.

    Edited · 0

    None found.

    Wrote their own version · 195

    1. Significant-Gravitas/AutoGPT★ 188ksame name
    2. pytorch/pytorch★ 106ksame name
    3. rtk-ai/rtk★ 87ksame name
    4. saadeghi/daisyui★ 43ksame name
    5. rustfs/rustfs★ 36ksame name
    6. dotnet/maui★ 23ksame name
    7. NVIDIA/Megatron-LM★ 18ksame name
    8. MiniMax-AI/skills★ 14ksame name
    9. openchamber/openchamber★ 13ksame name
    10. btriapitsyn/openchamber★ 13ksame name

    185 more not listed

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

  • In this repository2 other files it gives its agents.

The file

Show its rules and text

We split this skill into the rules it gives an agent

23 rules. 0 are common, 18 are distinctive, and 5 are very specific to this project.

  • 0

    Common

    Most of the 179 edited versions we read keep these.

  • 18

    Distinctive

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Check gh pr view -R vllm-project/speculators <number> --json state,isDraft.

    line 21
  2. Distinctive:

    Run these in parallel:

    line 27
  3. Distinctive:

    If the PR description contains links (issues, discussions, external docs, benchmarks, etc.), fetch them and incorporate the context.

    line 37
  4. Distinctive:

    Before forming any opinions, catalog:

    line 41
  5. Distinctive:

    If reviewers or the author linked to sources (docs, issues, code snippets, benchmarks, papers) in their comments, fetch and read them — …

    line 49
  6. Very specific:

    If the PR description or linked issues reference an RFC (issues labeled rfc in this repo), fetch the RFC issue via gh issue view and verify:

    line 73
  7. Distinctive:

    If the implementation deviates from the RFC, is the deviation documented and justified in the PR description?

    line 77
  8. Distinctive:

    If the PR description, commit messages, or code comments reference a paper (arXiv, conference proceedings, etc.):

    line 83
  9. Distinctive:

    Do the equations match? Check operator ordering, normalization, index conventions.

    line 88
  10. Distinctive:

    If no paper is referenced but the PR implements a known algorithm (Eagle, EAGLE-2, Eagle3, DFlash, MTP, Medusa, etc.), check whether the …

    line 92
  11. Distinctive:

    Only flag mismatches you can concretely demonstrate by quoting both the paper and the code.

    line 94
  12. Distinctive:

    If the issue involves an external API, library behavior, or spec, link to the source or quote the relevant documentation.

    line 136
  13. Distinctive:

    If you propose an alternative, show a concrete code suggestion and confirm the alternative works in context.

    line 137
  14. Distinctive:

    If the author's intent is unclear, ask a question instead of assuming.

    line 138
  15. Distinctive:

    Do NOT speculate: Only flag issues you can validate from the diff, surrounding code, or linked documentation.

    line 140
  16. Distinctive:

    If everything looks good at the design level, say so briefly.

    line 150
  17. Very specific:

    When linking to code, use full-SHA permalink format for proper rendering: …

    line 156
  18. Distinctive:

    Is this already addressed in the PR?: Check if a later hunk in the diff fixes the issue.

    line 163
  19. Distinctive:

    Is this preference disguised as correctness?: If yes, either drop it or explicitly mark as suggestion.

    line 165
  20. Very specific:

    Use gh api to post the review with all inline comments in a single atomic review submission.

    line 173
  21. Distinctive:

    If there are zero inline comments beyond the summary, post just the review body.

    line 190
  22. Distinctive:

    If no issues survive filtering (all < 80 confidence), do not post a review.

    line 192
  23. Very specific:

    Never approve a PR automatically: Always use event="COMMENT", never event="APPROVE".

    line 194

“Common” is measured: kept by at least 6 in 10 of the 179 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

PR Review

Review a GitHub pull request with design-first analysis, posted as a GitHub review.

Input

$ARGUMENTS — PR number or URL. If empty, detect from the current branch.

Instructions

You are reviewing a PR on the speculators repo — a library for training and serving speculative decoding models (Eagle3, DFlash, MTP) with vLLM integration.

Phase 0: Eligibility check

  1. Resolve the PR number from $ARGUMENTS (or current branch via gh pr view -R vllm-project/speculators --json number).
  2. Check gh pr view -R vllm-project/speculators <number> --json state,isDraft. Do not proceed if the PR is closed, merged, or a draft.
  3. Check if you (the current gh user) already posted a review on this PR. If so, compare the latest commit SHA on the PR against the commit SHA at the time of your last review. Do not post again if there are no new commits since your last review — report to the user that the PR hasn't changed since the last review. Even with new commits, if your review would say substantially the same thing as your previous one (same verdict, same outstanding items), do not post — report to the user that nothing has materially changed.
  4. After gathering context (Phase 1–2), reassess: if the discussion thread shows the PR is effectively still in progress — active experiments, trial-and-error debugging, the author saying "still working on this", or unresolved back-and-forth about the approach — treat it as a draft and do not review. Report to the user that the PR appears to still be in flux.

Phase 1: Gather context

Run these in parallel:

  • gh pr view -R vllm-project/speculators <number> --json title,body,baseRefName,headRefName,author,labels,files
  • gh pr diff -R vllm-project/speculators <number>
  • gh api repos/vllm-project/speculators/pulls/<number>/reviews — existing reviews
  • gh api repos/vllm-project/speculators/pulls/<number>/comments — inline comments
  • gh api repos/vllm-project/speculators/issues/<number>/comments — conversation thread

Then read the full diff. For changed files larger than 300 lines, also read the full file for surrounding context.

If the PR description contains links (issues, discussions, external docs, benchmarks, etc.), fetch them and incorporate the context. Verify that the PR actually addresses what the linked resources describe — flag mismatches between linked context and the implementation.

Phase 2: Understand existing discussion

Before forming any opinions, catalog:

  • What has each reviewer already said?
  • What has the author responded to?
  • What is unresolved?

Do NOT echo points already raised by other reviewers. If you agree with an existing comment, do not post your own version of it — even with additional detail or a different explanation. The only exception is if the author asked a clarifying question on the existing comment and the original reviewer hasn't answered; in that case, reply to that comment thread rather than posting a separate inline comment. Drop any finding that overlaps with an existing review comment.

If reviewers or the author linked to sources (docs, issues, code snippets, benchmarks, papers) in their comments, fetch and read them — they often contain context that informs whether a concern is valid or already resolved.

Phase 3: Design & big-picture review

Evaluate these first — they matter more than line-level nits:

  • Bloat check: Does the value this PR adds justify the maintenance burden it introduces? Every new file, abstraction, and code path has an ongoing cost — someone has to understand it, update it, and debug it. Flag PRs that add significant complexity (new modules, abstractions, config options, CLI flags) without proportional value. Also flag: vendored files that should be dependencies, IDE/editor files (.idea/, .vscode/), or code that reimplements something already available in a dependency.
  • AI slop check: Does the code read like unedited LLM output? Signs: excessive docstrings on every function (especially multi-paragraph ones restating the function signature), over-commented code explaining the obvious, cargo-culted error handling that catches exceptions only to re-raise them, unnecessary abstractions wrapping single calls, verbose variable names that read like natural language sentences, or boilerplate patterns copy-pasted without adaptation to the context. If you spot these patterns, flag it directly — ask the author to trim the noise.
  • Splittability: Does the PR bundle independent changes that could be reviewed separately? Look for: unrelated bug fixes alongside a feature, refactors mixed with new functionality, changes to multiple subsystems with no dependency between them, or "while I was here" cleanup. If the PR contains independently reviewable units, ask the author to split it — smaller PRs get better reviews and merge faster. Only flag this when the pieces are truly independent; don't ask to split tightly coupled changes.
  • Motivation: Is the PR description clear about why this change is needed? If the motivation is missing or vague, ask the author to clarify. A PR that doesn't explain the problem it solves is hard to review properly.
  • Purpose & scope: Does the PR do what the description claims? Is the scope appropriate or should it be split?
  • Architectural fit: Does this fit the existing patterns in speculators? Does it introduce unnecessary abstractions or bypass existing ones? Read surrounding code in the same module to understand local conventions (naming, error handling, structure) and flag deviations.
  • Correctness of approach: For the problem being solved, is this the right approach? Have alternatives been considered? If you see a clearly better alternative, name it concretely and explain the tradeoff.
  • Design principles:
    • DRY: Does this duplicate logic that already exists elsewhere in the codebase? If so, point to the existing code.
    • YAGNI: Does this add abstractions, parameters, or code paths not justified by the PR's stated goal? Premature generalization is a flag.
    • KISS: Is there a simpler way to achieve the same result? Complexity should be proportional to the problem.
  • Backward compatibility: Could this break existing checkpoints, configs, or CLI invocations?
  • Test coverage: Does the PR include tests proportional to the change? New logic should have unit tests. Changes to training, data generation, or model forward passes that affect end-to-end behavior should have integration tests (or the author should explain why they're impractical). For bug fixes specifically, there should be a regression test that fails without the fix and passes with it — if missing, ask for one. If tests are missing, ask for them — specify what scenarios should be covered.
  • Distributed training correctness (if applicable): barrier placement, device consistency, FSDP wrapping, gradient handling.
  • Tensor/numerical correctness (if applicable): shape mismatches, dtype promotions, masking, off-by-one in shift-based alignment.

Phase 3.5: RFC compliance

If the PR description or linked issues reference an RFC (issues labeled rfc in this repo), fetch the RFC issue via gh issue view and verify:

  1. Does the implementation match what was agreed in the RFC discussion? Check the final consensus, not just the original proposal — decisions often evolve in comments.
  2. Are there open objections or unresolved questions in the RFC that this PR should not proceed without addressing?
  3. If the implementation deviates from the RFC, is the deviation documented and justified in the PR description?

Flag any discrepancy between the RFC and the implementation, quoting the relevant part of the RFC discussion.

Phase 3.6: Paper validation

If the PR description, commit messages, or code comments reference a paper (arXiv, conference proceedings, etc.):

  1. Fetch the paper (use WebFetch on the arXiv abstract page or PDF URL).
  2. Identify the specific equations, algorithms, or architectural details the PR claims to implement.
  3. Compare the implementation against the paper:
    • Do the equations match? Check operator ordering, normalization, index conventions.
    • Are any simplifications or deviations from the paper intentional and documented, or silent divergences?
    • Are hyperparameter defaults consistent with what the paper recommends?
  4. In your review, quote the relevant section/equation from the paper and compare it to the code. For example: "Paper Eq. 5 defines the loss as L = ..., but the implementation at file:line computes ... instead — is this intentional?"
  5. If no paper is referenced but the PR implements a known algorithm (Eagle, EAGLE-2, Eagle3, DFlash, MTP, Medusa, etc.), check whether the implementation aligns with the canonical paper. If you cannot fetch or verify the paper, state that explicitly rather than guessing.

Only flag mismatches you can concretely demonstrate by quoting both the paper and the code. Do not flag stylistic differences in how math is expressed if the computation is equivalent.

Phase 4: Line-level review

Apply path-specific focus based on which files changed:

  • src/speculators/train/**: Distributed training correctness (barriers, device placement, FSDP), LR scheduler logic, checkpoint save/resume, loss computation with padding masks, multi-step loss aggregation and decay, bfloat16 safety.
  • src/speculators/models/**: Architecture correctness, attention mask construction for speculative positions, vocabulary mapping (draft→target) in forward/loss, KL divergence alignment, Pydantic config serialization.
  • src/speculators/data_generation/**: Hidden state extraction correctness, shift-based alignment (off-by-one), loss mask application before storage, vLLM client error handling.
  • src/speculators/convert/**: Weight mapping completeness, shape compatibility, key renaming correctness, legacy format assumptions.
  • src/speculators/config.py: Pydantic validators, serialization roundtrip, backward compat with saved checkpoints, registry auto-discovery.
  • scripts/launch_vllm.py: This script runs inside the vLLM virtualenv (VLLM_PYTHON), which does not have speculators installed. Make sure there's no speculators dependency and flag any such import with confidence 95+.
  • tests/**: Coverage of new code paths, edge cases for speculative decoding (vocab boundaries, multi-step loss), proper mocking of vLLM/GPU.

Ignore: **/*.pyc, **/build/**, **/.pytest_cache/**, **/.ruff_cache/**, **/__pycache__/**, **/*.egg-info/**.

Phase 5: Confidence scoring & filtering

Rate each finding 0–100:

  • 0–25: False positive, pre-existing issue, or something a linter/typechecker would catch.
  • 26–50: Minor nitpick not tied to a project convention or concrete bug.
  • 51–75: Real but low-impact or unlikely in practice.
  • 76–90: Verified issue that will impact functionality, or violates a documented project convention.
  • 91–100: Confirmed critical bug with concrete failure scenario.

Only keep findings with confidence ≥ 80.

False positives to actively filter out:

  • Pre-existing issues not introduced by this PR.
  • Pedantic style nitpicks a senior engineer would skip.
  • Issues a linter, typechecker, or CI would catch — do not run these yourself.
  • General quality concerns (documentation, naming style) that are purely cosmetic and not tied to correctness.
  • Issues silenced in code (e.g., lint ignore comments).
  • Intentional functionality changes that align with the PR's stated purpose.

Phase 6: Draft the review

For each surviving finding:

  • State the problem in one sentence.
  • If the issue involves an external API, library behavior, or spec, link to the source or quote the relevant documentation. Do not make unverified claims about external behavior.
  • If you propose an alternative, show a concrete code suggestion and confirm the alternative works in context.
  • If the author's intent is unclear, ask a question instead of assuming.

Do NOT speculate. Only flag issues you can validate from the diff, surrounding code, or linked documentation. If you're unsure whether something is a bug, phrase it as a question to the author.

Source attribution: If the PR implements logic derived from a paper, spec, or external reference and the code doesn't link to it (in comments, docstrings, or the PR description), ask the author to add a reference. Future readers shouldn't have to reverse-engineer which paper or doc a piece of logic came from.

Structure:

Review body (high-level summary):

  • 1–3 sentences on the overall design assessment.
  • Note any high-level concerns or open questions.
  • If everything looks good at the design level, say so briefly.

Inline comments (line-level):

  • Each non-high-level comment must reference a specific file and line range from the diff.
  • Order: correctness issues → design concerns → suggestions → questions.
  • When linking to code, use full-SHA permalink format for proper rendering: https://github.com/vllm-project/speculators/blob/{full_sha}/{path}#L{start}-L{end} Get the SHA via git rev-parse HEAD on the PR branch, and include ≥1 line of context above and below.

Phase 7: Verify before posting

Re-check every comment against this checklist:

  1. Is the claim actually true? Re-read the relevant code. If you reference external behavior (PyTorch, HuggingFace, vLLM), verify it via docs or source — or remove the claim.
  2. Is this already addressed in the PR? Check if a later hunk in the diff fixes the issue.
  3. Was this already raised by another reviewer? Drop if so.
  4. Is this preference disguised as correctness? If yes, either drop it or explicitly mark as suggestion.
  5. For alternatives you propose: have you verified the alternative actually works in this context?
  6. Does this pass the confidence ≥ 80 bar after re-examination?

Drop any comment that fails any check.

Phase 8: Post

Use gh api to post the review with all inline comments in a single atomic review submission. Do not use -f 'comments[0][...]' flags — gh serializes those as a JSON object with string keys, not an array, causing a 422 error. Instead, pipe raw JSON via --input -:

cat <<'REVIEW_EOF' | gh api repos/vllm-project/speculators/pulls/{number}/reviews -X POST --input - --jq '.html_url'
{
  "event": "COMMENT",
  "body": "<review summary>",
  "comments": [
    {"path": "<file>", "line": <line>, "body": "<comment>"},
    {"path": "<file>", "start_line": <start>, "line": <end>, "side": "RIGHT", "body": "<comment>"}
  ]
}
REVIEW_EOF

For single-line comments, use line only. For multi-line, use start_line + line. Use side: "RIGHT" for additions.

If there are zero inline comments beyond the summary, post just the review body.

If no issues survive filtering (all < 80 confidence), do not post a review. Report to the invoker that no actionable findings were found, but do not leave a comment on the PR — clean PRs don't need noise.

Never approve a PR automatically. Always use event="COMMENT", never event="APPROVE". Only the human reviewer should submit an approval.

End every review body with:

🤖 Generated with [Claude Code](https://claude.ai/code) using the `/pr-review` skill

Report the review URL when done, along with a brief summary: number of findings posted, number filtered out, and wall-clock time elapsed since Phase 0 started.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.