smoke-test-triage
Skill247 starsChanged 24 days agoMITView on GitHub
Triage and fix panache smoke-test regressions (idempotency,
In this repository11 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- add-lint-ruleSkill
- add-syntax-constructSkill
- commonmark-conformanceSkill
- external-toolsSkill
- html-conformanceSkill
- linter-investigationSkill
- math-parser-formatterSkill
- pandoc-ir-migrateSkill
- perf-investigationSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
17 rules. About 0 are common, 10 are distinctive, and 7 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈10
Distinctive · estimate
This author's own choices. Worth a look.
7
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 8
Use this skill when asked to investigate failures reported by smoke-test scans or debug format CI issues (especially …
- Distinctive: line 15
Add regression coverage in the right test surface.
- Distinctive: line 23
Read the issue/report details first:
- Very specific: line 43
When practical, reproduce with the scan's Panache revision as a baseline to establish that a later commit fixed the failure.
- Distinctive: line 53
if needed, collect pass artifacts with:
- Very specific: line 58
keep source syntax realistic (especially chunk/hashpipe/YAML edge cases)
- Very specific: line 69
If panache's CST differs *structurally* from pandoc — different block types (e.g.
- Very specific: line 81
Use debug artifacts (--dump-passes) to compare:
- Distinctive: line 85
If you find yourself reaching for a formatter helper to make pass1 == pass2 (e.g.
- Distinctive: line 94
If uncertain, state the best hypothesis and why before implementing — and include the pandoc-native output in the hypothesis.
- Distinctive: line 97
Add regression fixture(s):
- Very specific: line 101
add case name to tests/golden_cases.rs
- Very specific: line 103
add parser golden case under crates/panache-parser/tests/fixtures/cases/<case-name>/
- Distinctive: line 110
avoid papering over by changing expected outputs only
- Distinctive: line 127
Prefer adding one focused regression fixture per bug.
- Distinctive: line 128
Do not update unrelated golden fixtures.
- Distinctive: line 131
Use Pandoc behavior as reference for ambiguous syntax.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Use this skill when asked to investigate failures reported by smoke-test scans
or debug format CI issues (especially idempotency/losslessness regressions).
Goals
- Reproduce the exact failure from the report.
- Minimize to a stable local fixture.
- Add regression coverage in the right test surface.
- Fix root cause (not symptom).
- Validate targeted cases, then full repository checks.
- Close a report that no longer reproduces when the verification and GitHub permissions support that conclusion.
Triage workflow
Read the issue/report details first:
- failing check type (
idempotency,losslessness, parse, etc.) - sample file path
- upstream repo + commit SHA
- panache commit/version used by scan
- diff excerpt and approximate line
Close verified stale reports. The issue may have been fixed between the scan and triage. If the exact report command, pinned upstream commit, and matching configuration no longer reproduce the reported failure, scan the whole target repository with the current checkout. If that also passes, close the issue automatically when GitHub authentication permits it. Add a concise closure comment naming the original command, the upstream commit, the whole-repository before/after counts, and the fixing Panache commit when it can be identified, then run:
gh issue close <issue-number> --reason completed --comment "<evidence>"When practical, reproduce with the scan's Panache revision as a baseline to establish that a later commit fixed the failure. Do not close an issue when the reproduction setup is uncertain—for example, the target revision or its configuration could not be matched. If GitHub credentials or permissions prevent closing it, leave it open and report the failed closure command.
- failing check type (
- Reproduce in a local clone of the target repository:
- checkout the exact target commit from the report
- run:
panache debug format --checks all --report <sample-file>
- if needed, collect pass artifacts with:
panache debug format --checks all --dump-dir <dir> --dump-passes <file>
- Minimize:
- reduce to smallest snippet that still reproduces
- keep source syntax realistic (especially chunk/hashpipe/YAML edge cases)
- confirm reproduction is deterministic across repeated runs
- Classify the failure before fixing — and verify against pandoc-native
before any formatter-side fix:
- Mandatory pandoc check. On the minimized reproducer, run:
and compare to panache's CST (pandoc <repro>.md -f markdown -t nativecargo run -- parse < <repro>.md). Pandoc-native is the behavioral reference (per the rootAGENTS.md). If panache's CST differs structurally from pandoc — different block types (e.g. BulletList where pandoc has CodeBlock), missing/extra nesting, wrong attribute attachment — the bug is parser-side, no matter which pass shows the symptom. Idempotency is a downstream symptom of upstream shape divergence. - Quick heuristic for the initial hypothesis (still subject to the
pandoc check above):
- parser issue: losslessness mismatch, CST/marker/trivia dropped, parse shape changes, syntax not captured correctly.
- formatter issue: idempotency drift, wrapping/whitespace churn, hashpipe/YAML reflow instability, output normalization changing across passes.
- Use debug artifacts (
--dump-passes) to compare:- input vs parsed output (parser/losslessness boundary)
- first format vs second format (formatter/idempotency boundary)
- Anti-pattern: fixing in the formatter because the symptom lives there.
If you find yourself reaching for a formatter helper to make pass1 ==
pass2 (e.g. propagating looseness, normalizing markers, injecting
separators), stop and re-run the pandoc check. A formatter fix is only
correct when panache's CST already matches pandoc's structure and the
divergence is purely in rendering. If the CST is wrong, the formatter
fix is papering over a parser bug — fix the parser and add a
corpus case under
crates/panache-parser/tests/fixtures/pandoc-conformance/corpus/so the regression is guarded by the pandoc conformance harness. - If uncertain, state the best hypothesis and why before implementing — and include the pandoc-native output in the hypothesis.
- Mandatory pandoc check. On the minimized reproducer, run:
- Add regression fixture(s):
- Formatter user-visible cases:
tests/fixtures/cases/<case-name>/input.{md,qmd,Rmd}tests/fixtures/cases/<case-name>/expected.{md,qmd,Rmd}- add case name to
tests/golden_cases.rs
- Parser-only behavior:
- add parser golden case under
crates/panache-parser/tests/fixtures/cases/<case-name>/ - wire in parser golden runner
- add parser golden case under
- Formatter user-visible cases:
- Fix implementation at root cause:
- parser lossless/CST bugs -> parser crate
- formatting/idempotency bugs -> formatter crate
- avoid papering over by changing expected outputs only
- preserve existing behavior for unrelated fixtures
- Validate:
- targeted first:
cargo test --test golden_cases <case-name>panache debug format --checks all --report <fixture-input>
- then full validation:
cargo check --workspacecargo test --workspacecargo clippy --workspace --all-targets --all-features -- -D warningscargo fmt -- --check
- targeted first:
Panache-specific guidance
- For
.qmdsmoke regressions, ensure tests run with correct flavor context (Quarto detection matters for hashpipe/chunk behavior). - Prefer adding one focused regression fixture per bug.
- Do not update unrelated golden fixtures.
- Be careful with trailing whitespace and wrapping in hashpipe/YAML paths: these are common idempotency triggers.
- Use Pandoc behavior as reference for ambiguous syntax.
Report-back format
When done, report:
- Whether the issue reproduced (and exact command).
- Minimal reproducer summary.
- Fixture(s) added/updated.
- Root cause and code path changed.
- Validation commands run and outcomes.
- What you did not fix: remaining failure buckets by root cause, which are open modeling gaps worth work. Say plainly whether the issue can be closed or should stay open—and whether you closed it automatically.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

