update-executive-report
Skill1 starsChanged 2 months agoApache-2.0View on GitHub
Update and regenerate the evaluation tools executive report, comparative metrics, and visual presentation charts. Use this whenever the user asks to update the executive report, refresh eval tool metrics, or re-run evaluation visualization after expert-judgment restamps.
In this repository39 other files it gives its agents.
- agent-core/AGENTS.mdAGENTS.md
- .agents/AGENTS.mdAGENTS.md
- AGENTS.mdAGENTS.md
- behavioral-regression/AGENTS.mdAGENTS.md
- claude-foundation/AGENTS.mdAGENTS.md
- config/AGENTS.mdAGENTS.md
- corpora/AGENTS.mdAGENTS.md
- demo/AGENTS.mdAGENTS.md
- docs/AGENTS.mdAGENTS.md
- examples/AGENTS.mdAGENTS.md
- experiments/AGENTS.mdAGENTS.md
- flow-corpus/AGENTS.mdAGENTS.md
- flow-protocol/AGENTS.mdAGENTS.md
- openspec/AGENTS.mdAGENTS.md
- scripts/AGENTS.mdAGENTS.md
- skills/AGENTS.mdAGENTS.md
- tests/AGENTS.mdAGENTS.md
- claude-foundation/CLAUDE.mdCLAUDE.md
- merge-gate-auditorSkill
- update-executive-reportSkill
- architecture-drift-guardSkill
- commonSkill
- corpus-guardianSkill
- dataset-lintSkill
- deploySkill
- e2e-matrix-sentinelSkill
- eval-corpus-forgeSkill
- hierarchical-recursive-brainstormSkill
- model-benchSkill
- openai-judgeSkill
- openspec-implementation-reviewSkill
- openspec-peer-reviewSkill
- openspec-quality-planSkill
- pre-pr-gateSkill
- project-setupSkill
- quality-gateSkill
- refactoring-decomposerSkill
- repo-invariant-reviewSkill
- test-completeness-guardSkill
The file
We split this skill into the rules it gives an agent
9 rules. About 0 are common, 2 are distinctive, and 7 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
7
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 35
Ensure the JSON strictly adheres to docs/eval_metrics_schema.json (optional keys stay optional.
- Very specific: line 36
Update metadata.version and metadata.generated_date only when the dataset itself changed.
- Distinctive: line 39
Run dry-run validation to verify schema conformity and SVG freshness:
- Very specific: line 51
If scores or census/means changed, update docs/executive-report-eval-tools.md and refresh VP_DECK.md forbidden-numbers / honesty lines.
- Very specific: line 52
Ensure markdown links to docs/matrix-coverage.md, docs/phoenix-spike.md, docs/braintrust-spike.md, and VP_DECK.md remain valid.
- Very specific: line 66
If schema validation fails, inspect the missing or mismatched keys reported by generate_eval_metrics.py.
- Very specific: line 67
Do not hardcode metrics in python scripts.
- Very specific: line 68
If matplotlib rendering fails due to display/GUI issues, verify matplotlib.use('Agg') is active.
- Distinctive: line 72
Run the mechanical proof command:
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Update Executive Report — E2E Action Skill
Perform the update and maintenance of the LLM Evaluation Tools Executive Report (docs/executive-report-eval-tools.md), its companion visual metrics chart (docs/eval_metrics_comparison.png / .svg), and the speaker deck (docs/plans/scenario-eval-matrices/VP_DECK.md).
Category error (do not regress): Langfuse / Phoenix / BrainTrust are operations UIs / sinks, not the products that compute testgen / RCA / requirements scores. Those scorers are harness-side (F-065 / F-067 / F-068 / F-069). A Phoenix 9.5 on RCA is OpenInference UX, not rca_ac_at_k.
Scoring basis: docs/eval_metrics.json metadata.scoring_basis is expert_judgment. The 0–10 cells are not verified bake-off outcomes. Do not present them as measured capability scores.
Live e2e is not a restamp path: a --tiers all campaign (including PR #244) is recorded in docs/e2e-live-journey.md. Never python tests/test_e2e_matrix.py --update from that report — SKIP is not NOT-RUN. Never retune 0–10 cells from Langfuse/Phoenix smokes or contains sink PASS.
1. Preconditions (Input Contract)
Confirm these hold before proceeding:
- The schema definition exists at
docs/eval_metrics_schema.json. - The current metrics dataset exists at
docs/eval_metrics.json. - The generator script exists at
scripts/generate_eval_metrics.py. - The speaker deck exists at
docs/plans/scenario-eval-matrices/VP_DECK.md. - Python environment has
matplotlibandnumpyavailable.
2. Procedure (The E2E Steps)
- Review latest evidence, not bake-off numbers:
- Inspect spike reports (
docs/phoenix-spike.md,docs/braintrust-spike.md), registry rows indocs/matrix-coverage.md, and corpus honesty notes inVP_DECK.md/DECK_A_PLUS.md. - Identify seam/UX changes that might justify an expert-judgment restamp. Do not treat
artifacts/e2e-report/results.jsonas a three-vendor bake-off.
- Inspect spike reports (
- Update metrics data (never silently retune 0–10s):
- Edit
docs/eval_metrics.jsononly when the expert rationale actually changed. Do not nudge scores to make a chart look better. - Preserve
metadata.scoring_basis: expert_judgmentand the expert-judgment chart caption (chart_subtitle; optionalchart_taglinefalls back to the historic comparative-analysis line). - Ensure the JSON strictly adheres to
docs/eval_metrics_schema.json(optional keys stay optional; fixtures omit them). - Update
metadata.versionandmetadata.generated_dateonly when the dataset itself changed.
- Edit
- Validate dataset integrity (Check Mode):
- Run dry-run validation to verify schema conformity and SVG freshness:
python scripts/generate_eval_metrics.py --check
- Run dry-run validation to verify schema conformity and SVG freshness:
- Regenerate visual metric assets:
- Execute the generator to output both high-DPI PNG and SVG assets:
python scripts/generate_eval_metrics.py --input docs/eval_metrics.json --output docs/eval_metrics_comparison.png --format both
- Execute the generator to output both high-DPI PNG and SVG assets:
- Update narrative and deck:
- If scores or census/means changed, update
docs/executive-report-eval-tools.mdand refreshVP_DECK.mdforbidden-numbers / honesty lines. Keep vendors = sinks. - Ensure markdown links to
docs/matrix-coverage.md,docs/phoenix-spike.md,docs/braintrust-spike.md, andVP_DECK.mdremain valid.
- If scores or census/means changed, update
3. Output Contract (Definition of Done)
docs/eval_metrics.jsonis valid JSON, conforms todocs/eval_metrics_schema.json, and still carriesscoring_basis: expert_judgment.- The expert-judgment caption is preserved on the chart (subtitle and/or tagline); 0–10 cells were not silently retuned.
docs/eval_metrics_comparison.pngexists and is non-empty (> 100KB).docs/eval_metrics_comparison.svgexists and is valid XML/SVG.VP_DECK.mdforbidden-numbers match the current census/means if those changed.python scripts/generate_eval_metrics.py --checkexits with return code 0.pytest tests/test_generate_eval_metrics.pypasses 100%.
4. Failure Handling
- If schema validation fails, inspect the missing or mismatched keys reported by
generate_eval_metrics.py. - Do not hardcode metrics in python scripts. All data changes must reside in
docs/eval_metrics.json. - If matplotlib rendering fails due to display/GUI issues, verify
matplotlib.use('Agg')is active.
5. Verification Gate
Run the mechanical proof command:
python scripts/generate_eval_metrics.py --check && pytest tests/test_generate_eval_metrics.py -vDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

