agentleFS
youcislayoucisla/Agent-Foundry/skills/core/execution-observability/SKILL.md

execution-observability

Skill1 starsChanged 23 days agoMITView on GitHub

'Log what ran, on which model, with which skill, at what cost, and whether

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 5 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈5

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    After wiring an executor, loop, or batch runner.

    line 24
  2. Distinctive:

    When cost or quality regressions appear.

    line 25
  3. Distinctive:

    When multiple models or skills can serve the same request, or before claiming a pipeline is production-ready.

    line 26
  4. Distinctive:

    If the record cannot answer these, add fields before adding dashboards.

    line 71
  5. Distinctive:

    No model field, so cost changes are unattributable.

    line 87

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Execution Observability

An execution you cannot inspect is an execution you cannot debug, bill, or trust.

When to Use

  • New code path calls a model, or a routing decision changes behavior.
  • The user asks why a run was slow, expensive, or wrong, or an audit needs evidence.
  • After wiring an executor, loop, or batch runner.
  • When cost or quality regressions appear.
  • When multiple models or skills can serve the same request, or before claiming a pipeline is production-ready.

Procedure

1 — Define the record

One row per execution, with at minimum:

FieldWhy
timestampOrdering and incident windows
skill idAttribution and routing quality
modelWhich engine answered
tokens and costSpend accounting
durationLatency regressions
success and errorFailure taxonomy
fallback flagSeparates routed runs from preferred-model runs
schema validityDistinguishes valid structure from best-effort text

2 — Store hashes, not prompts

Prompts can contain user data, credentials, or private context. Persist a hash and the skill id. Keep full output only when the user explicitly opts in and the storage location is agreed. Never log API keys or authorization headers.

3 — Record both outcomes

Log success and failure. A table that only records successes hides the failures that matter most. Include the error text, truncated, and the model that failed.

4 — Query, do not dump

Expose a bounded history read: newest first, optional skill filter, limit and offset. Do not return full output bodies by default. A caller debugging a run needs metadata, not a wall of generated text.

5 — Correlate

The useful questions are joins:

  • Which skill has the highest failure rate?
  • Which model serves fallback runs most often?
  • Did cost per success rise after a prompt change?
  • Are schema failures concentrated in one skill?

If the record cannot answer these, add fields before adding dashboards.

Worked example

Two stored records: one successful execution and one failed attempt.

{"ts": "2026-10-25T09:00:00Z", "skill_id": "verify-first", "model": "m1", "tokens": 1234, "cost_usd": 0.004, "duration_s": 2.1, "success": true, "fallback": false, "schema_valid": true}
{"ts": "2026-10-25T09:01:00Z", "skill_id": "verify-first", "model": "m2", "success": false, "error": "timeout", "prompt_hash": "sha256:..."}

Anti-patterns

  • Logging prompts or secrets "for debugging".
  • Recording only successful runs.
  • Storing unbounded output in the index table.
  • No model field, so cost changes are unattributable.
  • A history endpoint that returns everything with no limit.

References

  • fields

Verification Checklist

  • Every execution path writes one record. (step 1)
  • Failures are recorded with their error. (step 3)
  • Prompts are hashed, not stored. (step 2)
  • The model and fallback flag are present. (step 1)
  • History reads are bounded and exclude full output bodies. (step 4)

Record Contract

  • Record trace_digest over canonical messages, model chain, seed, temperature, schema, skill hash, runtime version, and output.
  • Record request_meta_json with model revision, git SHA, message hash, roles, and char counts. A model-visible request must be reconstructable from the log.
  • Record phases for plan, completion, schema, judge, and persist. A missing phase is a blind spot, not a zero.
  • Record judge_status (ok, unparseable, call_failed, timeout) separately from the score. Never collapse judge failure into a low score.
  • Record failure_kind (provider, schema, policy, attest, budget, internal) so callers do not parse error strings.
  • Store success and failure latency separately, and keep the full output behind output_ref, not in the row.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.