execution-observability
Skill1 starsChanged 23 days agoMITView on GitHub
'Log what ran, on which model, with which skill, at what cost, and whether
In this repository46 other files it gives its agents.
- AGENTS.mdAGENTS.md
- .codex/AGENTS.mdAGENTS.md
- .gemini/AGENTS.mdAGENTS.md
- .cursor/rules/agent-foundry-orchestrator.mdcCursor rule
- .cursor/rules/agent-foundry-skills.mdcCursor rule
- web/llms.txtllms.txt
- anti-slopSkill
- api-designSkill
- automation-pickSkill
- bottleneck-gatingSkill
- budget-guardSkill
- cache-identitySkill
- constraint-then-solveSkill
- context-optimizationSkill
- cron-troubleshootSkill
- e2e-test-strategySkill
- engram-routingSkill
- fallback-model-routingSkill
- feedback-loopSkill
- judge-integritySkill
- knowledge-extractSkill
- landscape-firstSkill
- measure-firstSkill
- plan-before-codeSkill
- plan-then-actSkill
- prompt-disciplineSkill
- pushback-when-wrongSkill
- quality-protocolSkill
- read-before-buildSkill
- re-verify-findingsSkill
- session-closeoutSkill
- session-distillSkill
- show-your-workSkill
- structured-output-contractSkill
- verify-firstSkill
- workflow-decomposeSkill
- agent-memory-vaultSkill
- agent-team-standardsSkill
- autonomous-briefingSkill
- book-to-skillSkill
- chrome-devtools-mcp-bridgeSkill
- design-languageSkill
- funnel-pr-guardSkill
- persistent-memorySkill
- sql-migration-trioSkill
- token-compressionSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 5 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈5
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 24
After wiring an executor, loop, or batch runner.
- Distinctive: line 25
When cost or quality regressions appear.
- Distinctive: line 26
When multiple models or skills can serve the same request, or before claiming a pipeline is production-ready.
- Distinctive: line 71
If the record cannot answer these, add fields before adding dashboards.
- Distinctive: line 87
No model field, so cost changes are unattributable.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Execution Observability
An execution you cannot inspect is an execution you cannot debug, bill, or trust.
When to Use
- New code path calls a model, or a routing decision changes behavior.
- The user asks why a run was slow, expensive, or wrong, or an audit needs evidence.
- After wiring an executor, loop, or batch runner.
- When cost or quality regressions appear.
- When multiple models or skills can serve the same request, or before claiming a pipeline is production-ready.
Procedure
1 — Define the record
One row per execution, with at minimum:
| Field | Why |
| timestamp | Ordering and incident windows |
| skill id | Attribution and routing quality |
| model | Which engine answered |
| tokens and cost | Spend accounting |
| duration | Latency regressions |
| success and error | Failure taxonomy |
| fallback flag | Separates routed runs from preferred-model runs |
| schema validity | Distinguishes valid structure from best-effort text |
2 — Store hashes, not prompts
Prompts can contain user data, credentials, or private context. Persist a hash and the skill id. Keep full output only when the user explicitly opts in and the storage location is agreed. Never log API keys or authorization headers.
3 — Record both outcomes
Log success and failure. A table that only records successes hides the failures that matter most. Include the error text, truncated, and the model that failed.
4 — Query, do not dump
Expose a bounded history read: newest first, optional skill filter, limit and offset. Do not return full output bodies by default. A caller debugging a run needs metadata, not a wall of generated text.
5 — Correlate
The useful questions are joins:
- Which skill has the highest failure rate?
- Which model serves fallback runs most often?
- Did cost per success rise after a prompt change?
- Are schema failures concentrated in one skill?
If the record cannot answer these, add fields before adding dashboards.
Worked example
Two stored records: one successful execution and one failed attempt.
{"ts": "2026-10-25T09:00:00Z", "skill_id": "verify-first", "model": "m1", "tokens": 1234, "cost_usd": 0.004, "duration_s": 2.1, "success": true, "fallback": false, "schema_valid": true}
{"ts": "2026-10-25T09:01:00Z", "skill_id": "verify-first", "model": "m2", "success": false, "error": "timeout", "prompt_hash": "sha256:..."}Anti-patterns
- Logging prompts or secrets "for debugging".
- Recording only successful runs.
- Storing unbounded output in the index table.
- No model field, so cost changes are unattributable.
- A history endpoint that returns everything with no limit.
References
- fields
Verification Checklist
- Every execution path writes one record. (step 1)
- Failures are recorded with their error. (step 3)
- Prompts are hashed, not stored. (step 2)
- The model and fallback flag are present. (step 1)
- History reads are bounded and exclude full output bodies. (step 4)
Record Contract
- Record
trace_digestover canonical messages, model chain, seed, temperature, schema, skill hash, runtime version, and output. - Record
request_meta_jsonwith model revision, git SHA, message hash, roles, and char counts. A model-visible request must be reconstructable from the log. - Record
phasesfor plan, completion, schema, judge, and persist. A missing phase is a blind spot, not a zero. - Record
judge_status(ok,unparseable,call_failed,timeout) separately from the score. Never collapse judge failure into a low score. - Record
failure_kind(provider,schema,policy,attest,budget,internal) so callers do not parse error strings. - Store success and failure latency separately, and keep the full output behind
output_ref, not in the row.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

