agentleFS
ats4321ats4321/claude-engineering-skills/skills/agent-engineering/SKILL.md

Agent Engineering

Skill0 starsChanged 4 months agoMITView on GitHub

Build tool-using LLM agent systems — loop architecture, tool design, memory, and safety. Auto-load when building, extending, or reviewing an agent, ReAct loop, tool-calling system, or autonomous workflow; when designing agent tools or tool registries; when adding agent memory (short-term window, long-term retrieval); when deciding sandboxing/isolation for an agent that can execute code or shell commands; or when asked "should this be an agent?". NOT for parsing/retry mechanics of individual LLM calls (llm-integration-reliability) and NOT for the whether-an-LLM-at-all decision (llm-system-design).

  • Deletes or force-pushes

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Prefer a self-registering registry (tools register on import/definition) so adding a tool means adding one file, not editing central …

    line 55
  2. Distinctive:

    Every tool's arguments are validated in code.

    line 178

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Agent Engineering

Purpose

An agent is a loop in which a model chooses actions — which means every defect in the model's judgment becomes a runtime behavior. This skill is the engineering discipline that makes that survivable: hard bounds designed in from day one, small typed tools, memory only when needed, safety proportional to a written threat model, and runs that can be reconstructed after the fact.

Metadata

  • Prerequisites: llm-system-design (confirms an agent loop is actually warranted); security-review-playbook (the threat model that sizes agent safety).
  • Related Skills: llm-integration-reliability (per-call reliability inside the loop), prompt-and-context-engineering (the agent's system prompt and context), evaluation-frameworks (measuring agent task success), observability-and-diagnostics (run reconstructability), engineering-minimalism (most "agents" should be pipelines).
  • Owns: agent loop architecture (bounds, termination); agent tool design and registries; agent memory design; agent sandboxing/isolation decisions.

When to Use / When NOT to Use

Use when:

  • Building or modifying an agent loop, its tools, or its memory.
  • Reviewing agent code for safety, boundedness, or debuggability.
  • Deciding how much isolation an agent needs (guardrails vs container/VM).
  • An agent misbehaves at the loop level: runs forever, thrashes between tools, forgets context, or does something destructive.

Do NOT use (load the sibling instead):

  • Deciding whether an LLM (or an agent at all) belongs in the system → llm-system-design.
  • A single call's output is malformed / needs retries / needs type-guards → llm-integration-reliability.
  • Writing the agent's prompts or managing its context window → prompt-and-context-engineering.
  • Measuring whether the agent completes tasks correctly → evaluation-frameworks.
  • Auditing the security of the host system generally → security-review-playbook.

Definitions & Mental Model

  • Agent loop: repeated cycle of (model proposes action) → (system executes tool) → (observation returned to model), until the model answers or a bound trips.
  • Tool: a single-purpose function the model may invoke, described by name, description, and a typed argument schema.
  • Observation: the string/structure returned to the model after a tool runs — including tool errors, which are observations, not crashes.
  • Bound: any hard limit the model cannot exceed regardless of its choices — iteration caps, per-tool timeouts, token/cost budgets.
  • Guardrail vs boundary (definitions owned by engineering-minimalism; classification procedure by security-review-playbook): a guardrail catches accidents and is bypassable; a boundary cannot be bypassed. Blocklists are guardrails. Containers are boundaries.

Mental model: treat the agent as an enthusiastic intern with root access and no fear. You do not make the intern safe by asking nicely (prompts are guardrails); you make the environment safe by deciding in advance what the intern can touch (tools), how long they may try (bounds), what they can remember (memory), and what room they are locked in (isolation). Every design question below reduces to one of those four. And before all of it: most tasks assigned to "agents" have a knowable step sequence — those are pipelines, and pipelines are easier to test, bound, and price.

Core Methodology

  1. Confirm the loop is earned. From llm-system-design: an agent loop is justified only when the step sequence is genuinely unknowable in advance. If you can write the steps down, write a pipeline instead. Record this decision; it is the most common design error in the field.
  2. Design the bounds before the loop. Non-negotiable, in code, on day one:
    • Iteration cap (single-digit-to-low-tens; pick a number and enforce it). On trip: return the best partial result plus an explicit "iteration limit reached" marker — never spin.
    • Per-tool timeouts, sized per tool (a shell command deserves more than a math eval; a web fetch sits between).
    • Budget cap where calls cost money: max tokens or max spend per run.
    • Termination condition the model controls (a final-answer action) plus the caps the model cannot control.
  3. Design tools as small, single-purpose, typed functions.
    • One tool, one capability. "run_shell" and "read_file" — never "do_stuff".
    • Each tool declares: name, one-sentence description written for the model, and a typed args schema. The description is an interface contract — write it like API documentation (api-and-interface-design owns interface craft).
    • Validate tool arguments in code before executing; the model's respect for the schema is a guardrail, your validation is the boundary.
    • Prefer a self-registering registry (tools register on import/definition) so adding a tool means adding one file, not editing central dispatch.
    • Return tool failures as observations — "Error running {action}: {e}" — so the loop survives and the model can adapt. (Reliability mechanics: llm-integration-reliability step 9.)
  4. Add memory only when the task demands it (decision tree):
Does the task span more than one loop run?
├─ NO → context window IS the memory. Add nothing. (Most agents end here.)
└─ YES → What must persist?
    ├─ Recent conversational state → SHORT-TERM: rolling window of the
    │   last N exchanges. A list. No database.
    └─ Facts retrievable by meaning across sessions → LONG-TERM:
        embed + store + top-k similarity retrieval.
        ├─ Treat stored blobs as UNTRUSTED at load time
        │   (deserialization safety: security-review-playbook)
        ├─ Fix the embedding model — the vector space is a stable
        │   contract; changing it invalidates the store
        └─ Design the clear/expiry story before the write path
  1. Size safety to a written threat model. Answer in writing (procedure owned by security-review-playbook):
    • What can this agent execute or modify? (shell? filesystem? network? payments?)
    • Who chooses the model and the prompts — a trusted operator, or anyone?
    • Then choose isolation:
Trusted operator + trusted model + local machine
  → guardrails suffice (blocklists, confirmations, path allowlists)
    BUT label them guardrails, with the bypass documented.
Untrusted input reaches the agent (webhooks, user prompts, retrieved web content)
  → prompt injection is now in scope: the model may be turned against you.
    Real boundaries required: container/VM, least-privilege credentials,
    no destructive tools, egress restrictions.
Agent output triggers irreversible external effects (publish, pay, delete)
  → human-in-the-loop gate or dry-run-first on that tool, regardless of trust.
  1. Make every run reconstructable. Log each cycle: iteration number, proposed action, arguments, observation (truncated but present), and termination reason. The acceptance test (owned by observability-and-diagnostics): a failed run must be diagnosable from its log alone, without re-running.
  2. Test the loop's failure surface, not its happy path. Fake the model (monkeypatch/stub) and assert: malformed action → strike behavior; unknown tool → survivable observation; tool exception → loop continues; iteration cap → partial answer with marker; blocklist/validation → rejection. (Test mechanics: validation-and-testing; per-call parsing behavior: llm-integration-reliability.)
  3. Measure task success before shipping changes. Loop tweaks (new tool, new prompt, new bound) are evaluated against a task suite, not vibes — evaluation-frameworks.

Agent design checklist

  • "Should this be a pipeline?" answered in writing — the loop is earned
  • Iteration cap, per-tool timeouts, and budget cap in code from day one
  • Cap-trip behavior returns partial result + explicit marker
  • Every tool single-purpose with typed, code-validated arguments
  • Tool errors returned as observations; loop survives any single tool failure
  • Memory tier chosen by the decision tree; none is the default
  • Long-term store treats its own blobs as untrusted at load
  • Written threat model; isolation level matches it; guardrails labeled with bypasses
  • Irreversible-effect tools gated (human approval or dry-run-first)
  • Every run reconstructable from logs alone
  • Failure-surface tests exist (malformed action, unknown tool, tool crash, cap trip)

Discovery & Audit Commands

Audit an existing agent codebase:

# Find the loop and its bounds (absent bounds are the finding)
grep -rn -E "while|for .* in range|max_iter|MAX_ITER|iteration" --include="*.py" --include="*.ts" . | grep -v node_modules | head -20

# Tool inventory: registrations, schemas, dispatch
grep -rn -E "register|TOOL|tools\s*=|args_schema|@tool" --include="*.py" --include="*.ts" . | grep -v node_modules | head -30

# Timeouts per tool (each execution path needs one)
grep -rn -E "timeout" --include="*.py" --include="*.ts" . | grep -v node_modules | head -20

# Destructive capability surface (what can the agent actually do?)
grep -rn -E "subprocess|os\.system|shell=True|exec\(|child_process|unlink|rmtree|requests\.|httpx|fetch\(" --include="*.py" --include="*.ts" . | grep -v node_modules | head -30

# Memory: persistence paths and deserialization safety
grep -rn -E "sqlite|\.db|pickle|np\.load|allow_pickle|embed" --include="*.py" . | head -20

# Is the run reconstructable? (logging of actions/observations)
grep -rn -E "log|console|print" --include="*.py" --include="*.ts" . | grep -iE "action|observation|tool" | head -20

Audit rule: every tool execution path must show (a) argument validation, (b) a timeout, (c) error-to-observation conversion. Any path missing one is a defect.

Failure Modes & Anti-patterns

SymptomMistakeCorrection
Agent runs forever on a confusing taskNo iteration capHard cap + partial-answer return with explicit marker (step 2)
One tool exception kills a 30-step runTool errors propagate as crashesCatch → return as observation; loop survives (step 3)
Agent deleted/overwrote something irreplaceableDestructive tool with guardrail-only protectionBoundary (isolation/gating) for irreversible effects; guardrails are for accidents (step 5)
"It's safe, the prompt says to be careful"Prompt instructions treated as a security boundaryPrompts are guardrails; isolation and code validation are boundaries
Retrieved web content hijacks the agentPrompt injection outside the threat modelUntrusted input in scope → container/least-privilege/no destructive tools (step 5)
Fixed 3-step task built as an agentLoop chosen for a knowable sequencePipeline; agents only for unknowable sequences (step 1)
"Why did it do that?" unanswerableActions/observations not loggedLog every cycle; runs reconstructable from logs alone (step 6)
God-tool execute(anything)Tool surface not decomposedSmall single-purpose tools with typed, validated args (step 3)
Embedding store breaks after model swapEmbedding model treated as swappableFix the embedding model; the vector space is a contract (step 4)
Loop hangs on one slow toolGlobal timeout only, or nonePer-tool timeouts sized to each tool (step 2)
RCE via tampered memory fileOwn DB blobs trusted at loadSafe deserialization; treat stored blobs as untrusted (step 4)
New tool required editing 4 dispatch sitesCentral hardcoded dispatchSelf-registering registry; a tool is one file (step 3)

Worked Example

Task: an agent that triages incoming bug reports — reads the report, searches the codebase, and files a labeled ticket.

  1. Earned? The search path depends on what the report reveals — the sequence is genuinely unknowable. Loop earned. (Filing the ticket alone would be a pipeline.)
  2. Bounds: 8 iterations; timeouts — code search 10s, file read 5s, ticket API 15s; budget 50k tokens/run.
  3. Tools: search_code(query: str), read_file(path: str, max_bytes: int), file_ticket(title: str, body: str, label: enum). Args validated in code — read_file rejects paths outside the repo root (path traversal is a boundary, not a guardrail); file_ticket.label validated against the enum.
  4. Memory: none — each triage is one run; the context window suffices.
  5. Threat model: bug reports are untrusted input that the model reads → prompt injection in scope. Mitigation: no shell tool, no network beyond the ticket API, least-privilege API credential (create-only), agent runs in a container.
  6. Irreversible effects: file_ticket is append-only and labeled by the agent — acceptable without human gate; a close_ticket tool would require one.
  7. Reconstructability: every cycle logged as iter=N action=search_code args={...} → obs[240 chars]; termination reason logged.
  8. Failure tests: stubbed model emits malformed action twice → run ends surfacing raw output; search_code raises → next observation is the error string and the loop continues; 9th iteration → partial triage + "iteration limit" marker.

Repository Examples

Repo facts below are point-in-time illustrations (as of 2026-07-04) — examples, never assumptions about your system.

  • agentix (~/agentix) — a ~700-line local ReAct agent framework embodying most of this skill: self-registering tool registry (Tool dataclass: name, description, args_schema, fn; register-on-import — adding a tool is one file plus an import); hard cap of 10 loop iterations; per-tool timeouts (shell 30s, python 15s, web 15s); tool exceptions returned as observation strings so the loop survives; dual-tier memory (short-term rolling window + long-term SQLite embeddings at ~/.agentix/memory.db with cosine top-k) with the embedding model fixed (nomic-embed-text) and the chat model selectable; written threat model in the README — "the security boundary is you and the model you point it at, not the code" — with the shell blocklist ["rm -rf /", "sudo", "mkfs"] honestly documented as a guardrail (bypass named) and container/VM advised for real isolation; failure-surface tests (malformed-JSON strikes, unknown tool, blocklist) using pytest monkeypatch only.
  • prism (~/prism) — the contrast case: an LLM system that is correctly NOT an agent. Its review sequence is knowable (fetch diff → chunk → judge chunks → post), so it is a bounded fan-out pipeline — the step-1 decision made right.

Validation Criteria

You applied this skill correctly when:

  1. The pipeline-vs-agent decision is recorded, and the loop's step sequence is demonstrably unknowable.
  2. Reading the code reveals iteration cap, per-tool timeouts, and budget caps without searching hard — and a test trips each.
  3. Every tool's arguments are validated in code; feeding an out-of-schema call in a test produces a rejection observation, not an execution.
  4. Killing any single tool mid-run leaves the loop alive with the error visible in the transcript.
  5. The threat model document exists and the isolation level matches it; every guardrail is labeled with its bypass.
  6. A teammate can reconstruct any failed run — actions, observations, termination reason — from logs alone.
  7. Memory, if present, matches the decision tree; if absent, that was the deliberate default.

Provenance & Maintenance

  • Sources: ~/agentix (primary case study), ~/prism (contrast case) — investigated 2026-07-04. Owner context: primary consumers are Sonnet-class agents; minimalism doctrine owner-confirmed 2026-07-04. Skill authored 2026-07-06; methodology is repo-independent.
  • Assumptions: agentix constants (10 iterations, 30s/15s/15s timeouts, blocklist contents, memory path) are point-in-time. Prompt-injection guidance reflects the 2026 threat landscape — attacks evolve; re-check.
  • Re-verification commands:
    grep -rn "register\|args_schema" ~/agentix/agentix | head
    grep -rn "timeout\|BLOCKED" ~/agentix/agentix/tools | head
    grep -in "boundary" ~/agentix/README.md
  • Likely to drift: agentix's caps/timeouts/blocklist; agent-framework idioms industry-wide; the prompt-injection mitigation state of the art.
  • Maintenance checklist:
    • Re-run re-verification commands; re-stamp Repository Examples.
    • Re-check the isolation decision tree against current prompt-injection research annually.
    • Confirm cross-referenced skills still exist under their directory names.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.