Agentic System Audit

Audit the whole agentic contract, not isolated files.

Find where code, prompts, tools, state, results, docs, tests, configuration, packaging, or runtime capabilities disagree or leave the next safe action ambiguous.

Prefer deletion, consolidation, one canonical owner, and the smallest durable correction.

This is a read-only audit. Do not modify the repository.

Scope and execution boundary

Establish the actual audited snapshot before investigating: exact commit SHA for committed source; base SHA and included staged, unstaged, or untracked changes (with an exact diff or snapshot reference) for a working tree; base and head SHAs for a PR. Attribute CI, tests, and prior reviews to the revisions they actually cover. Record relevant supported upstream versions.

During scoping, identify relevant active ADR decisions and follow their supersession chains. Runtime and supported external contracts establish actual behavior; active decisions constrain architectural recommendations, while superseded decisions explain history rather than current semantics. Revalidate applicable decisions when closing material findings.

Permission to inspect source does not authorize executing it. Run a targeted smoke only when execution is authorized and a disposable environment is verified isolated from the audited checkout and live user resources, including configuration, credentials, and processes. Otherwise use read-only evidence and report any material unresolved uncertainty.

Apply Herdsman-specific checks only where the target has equivalent capabilities or lifecycle concepts; otherwise mark those checks not applicable.

Operating model

Own the audit and final verdict. Delegate bounded, independent evidence work when a suitable read-only Agent is available and delegation is useful. Select Agent definitions by their effective capabilities, never by a bundled name. Respect available tools, definition allowlists, ownership, and execution depth. If no suitable Agent exists, investigate and synthesize locally using permitted read-only capabilities; report any material capability gap.

Start with the current list_agents roster when available. Refresh it if configuration changes. A configured child allowance does not guarantee that the child is available or that the current controller can delegate.

Coverage lenses

Cover three concerns, delegating independent lenses when useful:

• A — runtime: producers, validation, state, identity, lifecycle, and reachability;
• B — instructions: instruction ownership and effective capabilities;
• C — consumers: model/UI projections, docs, tests, and package glue.

External/API research is conditional on the research trigger and actual available tools. One capable Agent may cover multiple lenses; three Agents are not required. Avoid redundant analysis between the audit owner and delegated work.

Delegated audit contract

Pass each investigator the relevant lens, canonical evidence, and the following rules; do not assume it receives this skill automatically:

Audit the assigned area as part of one agentic system. Remain strictly read-only.

Before reporting any P0-P2 finding:

• identify the semantic source and relevant producers;
• prove the disputed state/value is reachable on a production path when behavior is at issue;
• trace it through every consumer boundary relevant to the claim;
• resolve material upstream/runtime assumptions for the repository's supported versions using primary evidence when available;
• verify that a claimed regression check would fail for the old behavior.

Do not report representable-but-unreachable states as defects. Do not treat differently named fields as equivalent without tracing semantics. Do not leave an answerable material uncertainty as follow-up.

If required evidence genuinely cannot be established, report a bounded Unverified risk, not a definitive finding.

Compare root/parent/child, definition/session/fork, model/UI, success/failure/recovery, configured/effective capability, and static/dynamic guidance.

Return candidate findings with exact evidence and the smallest durable correction.

The audit owner independently verifies and reconciles material candidates, de-duplicates findings, and returns one report using the required output contract. Pass supplied canonical diffs, snapshots, specs, or source artifacts to investigators instead of making them rediscover the same material.

Material finding closure

Each lens returns candidate evidence. The audit owner closes candidates before severity is assigned, rechecking applicable active ADR constraints for architectural findings and corrections.

Trace every boundary used by the claim. For result/state findings, prefer:

semantic source
→ producer
→ validation/normalization
→ runtime/durable state
→ structured/public projection
→ model consumer
→ human/API consumer
→ docs/tests

Not every step applies to every finding.

Provenance

Find where the disputed fact originates and how it changes.

Do not infer semantic equivalence from similar field names. Trace producers and scope first.

Reachability

Do not confuse representability with reachability.

A type, schema, fixture, or doc proves a value can be described. A behavioral finding requires a production path that can actually produce the material state/value.

If the harmful state cannot occur on the claimed path, dismiss or narrow the candidate.

Consumer contract

Establish what the actual consumer receives.

For model-visible claims, determine whether the model gets content, details, both, or another transformation.

For human claims, distinguish compact TUI, expanded TUI, plain/RPC/JSON, and logs/files.

Never assume renderer-visible metadata is model-visible.

External contract

If a material conclusion depends on upstream behavior, resolve it for the repository's supported version range.

Prefer:

supported-version primary source/API/type contract
→ supported-version official docs
→ installed dependency source
→ authorized isolated runtime smoke only if still unresolved

Do not substitute latest behavior for the supported range. Do not require a smoke test when source plus local integration already establishes the behavior.

Test discrimination

When claiming a test gap or proposing a regression:

Would this test fail for the old bug and pass for the corrected behavior?

For lifecycle/recovery bugs, pair the harmful case with its nearest legitimate opposite when that distinction matters, such as a stale working parent versus a parent correctly waiting on unresolved child work.

Do not count assertion volume as coverage.

Closure outcomes

Every material candidate becomes exactly one of:

• Verified finding: evidence establishes defect and consequence.
• Dismissed/narrowed: intentional, unreachable, semantically different, or immaterial.
• Unverified risk: evidence genuinely cannot be established with available read-only capabilities.

An Unverified risk must state known evidence, missing evidence, version/runtime scope, why it matters, and one exact resolver.

Do not use it for an answerable question that was not investigated.

Lens A: runtime contracts and reachability

Own:

Can this state/value actually happen, where is it produced, and what runtime contract governs it?

Prove this liveness invariant for unresolved directly-owned assignments:

Every unresolved directly-owned assignment must either remain capable of making progress without its owner, or have a reliable reconciliation path that brings the exact owner back when action is required.

Trace:

schema/input
→ validation
→ effective configuration
→ execution
→ runtime/durable state
→ public result
→ next legal action

Check:

• schema vs runtime acceptance/rejection;
• defaults, normalization, mutually exclusive fields;
• validation before mutation;
• sibling-path validation consistency;
• logical label, definition, Pi session, owner, request, ask, run, workspace, tab, pane, Herdr identity;
• stale/replacement identity binding;
• public state vs actual action eligibility;
• working, blocked, settling, unknown, steerability, staleness, pending ask/result, child gating, cleanup convergence;
• unresolved-work liveness across owner idle/busy state, failed attention delivery, reminder recurrence, episode termination, and restart;
• parent/child distinctions where a working parent may itself be stale while a legitimately waiting parent projects blocked;
• fresh delegation, continuation, fork, replacement, completion, parent-child completion, ask/reply, close, rollback, restart/recovery;
• source identity, request correlation, result paths, truncation, cleanup warnings, primary/cleanup errors, rollback/retry state.

Return exact production paths, reachability evidence, and candidate mismatches. Do not decide architecture from local evidence alone.

Lens B: instruction and capability stack

Own:

Who owns this rule, and is it truthful for the effective execution context?

Map:

system/base prompt
tool description
controller scope
shared agent prompt
role body
body @file
skills
dynamic success/error guidance
human docs

For each normative rule, find its intended single owner.

Flag:

• zero owner: instruction gap;
• multiple authoritative owners: DUPLICATE_AUTHORITY;
• conflicting owners: CONTRADICTION;
• wrong owner: AUTHORITY_LEAK.

Compare:

definition
→ override composition
→ effective tools/skills/extensions/context
→ launch args
→ actual role/depth
→ model instructions
→ presentation

Find:

• GHOST_CAPABILITY;
• HIDDEN_CAPABILITY;
• root/parent/leaf capability mismatches;
• metadata advertising removed capabilities;
• infrastructure accidentally removable by normal policy;
• user-configurable policy treated as mandatory infrastructure;
• impossible or underspecified instructions;
• global policy embedded in role prompts;
• API mechanics redundantly taught by prompts/skills.

Return exact instruction text, owner, effective runtime evidence, and candidate conflicts.

Lens C: presentation, docs, tests, and repository glue

Own:

Where does runtime truth go, what does each consumer actually see, and do docs/tests describe that contract?

Model-visible presentation

Inspect list, assignment/steer/reply/close acknowledgements, completion delivery, errors, and truncation.

Ask:

Using only what the model actually receives, can it identify the source, understand the state, and choose the next legal action?

Flag missing correlation, hidden material details, ambiguous terminology, duplicated IDs, static policy repeated on every result, or missing dynamic next-action evidence.

For opacity candidates, trace projection semantics, not just field names.

Human presentation

Compare status widget, definitions view, completion renderer, warnings, compact/expanded views, and any plain/API surface.

Find hidden failures, duplicated data, display identity reused as machine identity, or UI claims stronger than runtime evidence.

Documentation

Find shipped-but-undocumented behavior, documented-but-unshipped behavior, duplicate canonical owners, stale terminology, rejected examples, and implementation details presented as public contracts.

Tests

Find public contracts without discriminating regressions, tests that pass for correct and broken behavior, mocks that cannot detect claimed integration failures, missing sibling invariants, and stale fixtures.

Repository/package glue

Check package files, discovery, README links, AGENTS/maintainer guidance, validation scripts, dead compatibility/configuration, and removed concepts that still ship or remain referenced.

Return projection chains, consumer visibility, docs/tests evidence, and candidate mismatches.

Research trigger

Research is conditional. Once triggered, verification is required when capability exists.

Use external research when:

a candidate likely to affect a P0-P2 finding, verdict, or remediation
depends on external API/runtime behavior

Research input should contain only:

exact disputed assumption
supported version range
preferred primary source/repository
required direct answer

Require primary/official evidence, exact version relevance, and only material findings.

If web capability is unavailable:

1. inspect installed/local upstream source;
2. inspect pinned/supported dependency source where reachable;
3. use other authoritative read-only evidence;
4. otherwise return Unverified risk.

Do not add dependencies or require optional web tooling to run the audit.

Cross-surface contract matrix

Cover each applicable chain.

Input

schema ↔ description ↔ validation ↔ errors ↔ docs ↔ tests

Capability

definition ↔ override ↔ effective config ↔ launch ↔ role/depth ↔ tools/prompts ↔ presentation

Instruction

controller ↔ shared prompt ↔ role body ↔ skills ↔ dynamic results

Identity

label ↔ definition ↔ session ↔ request ↔ owner ↔ Herdr identity ↔ result/error/UI

State

runtime evidence ↔ public state ↔ steerability ↔ eligibility ↔ displayed state ↔ next action

Completion

completion ↔ durable result ↔ result file ↔ owner message ↔ model content ↔ human rendering ↔ cleanup

Configuration

syntax ↔ parsing ↔ merge ↔ validation ↔ effective metadata ↔ runtime ↔ persistence/UI ↔ docs

Recovery

failure ↔ retained evidence ↔ autonomous progress/attention owner ↔ wake/delivery ↔ recurrence/termination ↔ structured error ↔ model/human guidance ↔ retry boundary

Distribution

repository ↔ package manifest ↔ installed files ↔ discovery ↔ maintainer guidance

Mandatory boundary comparisons

Check where applicable:

Controller depth

root | direct agent/parent | nested child/leaf | unmanaged session

Agent generation

fresh | exact-session continuation | fork

Visibility

model content | structured details | compact TUI | expanded TUI | plain/RPC/JSON | logs/files

Outcome

success | blocked | failure | rollback failure | close | cleanup pending | overflow | restart recovery

Configuration

bundled | partial override | standalone global | empty fields | false | [] | invalid

Defect taxonomy

Use consistently:

• CONTRADICTION: authoritative surfaces prescribe incompatible behavior.
• GAP: required knowledge/behavior has no appropriate owner or result.
• GHOST_CAPABILITY: instructions/metadata claim unavailable capability.
• HIDDEN_CAPABILITY: required capability exists without enough safe guidance.
• DUPLICATE_AUTHORITY: one normative rule has multiple authoritative owners.
• AUTHORITY_LEAK: policy is owned by a layer that should not control it.
• CONTEXT_MISMATCH: rule is correct in one context and wrong in another.
• IDENTITY_AMBIGUITY: identity loses exact or single semantic meaning.
• STATE_AMBIGUITY: displayed/instructed state does not map cleanly to legal actions.
• RESULT_OPACITY: receiver lacks material source, state, correlation, outcome, or next-action evidence.
• FAIL_OPEN: missing/ambiguous evidence causes unsafe continuation.
• DRIFT: code, tests, docs, metadata, examples, or prompts describe different generations.
• TEST_BLIND_SPOT: material public contract lacks a discriminating regression.
• TOKEN_WASTE: repeated model context adds no decision value.
• DEAD_COMPLEXITY: compatibility/abstraction/state/configuration has no justified current use.
• UX_AMBIGUITY: human presentation obscures correct system behavior.

Create another label only when none fits.

Severity

• P0: unsafe/destructive behavior, wrong-target control, trust-boundary violation, data loss, or durable corruption.
• P1: materially wrong action, stuck lifecycle, broken delegation/control, or misunderstanding of an authoritative result.
• P2: meaningful confusion, drift risk, redundant authority, weak diagnostics, avoidable context cost, or recurring maintainer mistakes.
• P3: small cleanup, naming, presentation, or maintainability issue with little behavioral risk.

Do not inflate severity because a finding is interesting.

Evidence standard

A material cross-surface finding needs:

A: semantic behavior/claim + exact source
B: conflicting/missing behavior/claim + exact source
Connection: why A and B are the same contract crossing a boundary
Reachability: production path when behavior is at issue
Consequence: what can actually go wrong
Minimal correction: narrowest root owner

An investigator's conclusion is evidence, not authority.

The audit owner independently verifies every P0/P1, every P2 affecting architecture/verdict/external assumptions, and any disputed conclusion.

Do not report architectural preference as a defect.

Audit heuristics

Use these to find candidates. They do not replace closure.

• Who owns this sentence? One normative rule should normally have one authoritative owner.
• Can the recipient act on this? Check only what that consumer actually receives.
• Can the agent do what it is told? Compare prompts to effective capability at every depth.
• Does runtime enforce the important part? Safety, ownership, identity, validation, and data-loss boundaries should not rely only on prompts.
• Does code know something the model does not? Inspect details, hidden metadata, cleanup evidence, capability inference, state subconditions, correlation IDs.
• Does the model know something code does not enforce? Separate strategy from invariants such as exact ownership, read-only, no overlap, or one-shot rules.
• What changes at continuation? Generation-time facts described as dynamically refreshed are suspect.
• What changes at root/child depth? Global policy projected differently by depth is suspect.
• What was deleted but still has a shadow? Search docs, tests, package files, comments, compatibility branches, fields, examples.
• Would this test fail for the old bug? Prefer one discriminating regression over many weak assertions.
• Can unresolved work disappear from attention forever? Prove either autonomous progress remains possible or reconciliation eventually wakes the exact responsible owner. Check hierarchy boundaries, owner idle/busy state, failed delivery, recurring versus one-shot attention, restart behavior, and episode termination.
• Is the same fact sent twice? Check model/human identity, paths, model, state, warnings, next-action text.
• Is this field mismatch semantic drift? Find all producers, scope, reachability, and consumers first.

Efficiency rules

1. Start from entry points and contract surfaces.
2. Search ownership before opening large files.
3. Trace callers and producers before declaring a bug.
4. Keep delegated investigators inside their lenses.
5. Pass canonical artifacts instead of rediscovering them.
6. Research only external assumptions that can change a material conclusion.
7. Verify candidate findings, not every inspected line.
8. Do not spawn agents merely to confirm No findings.
9. Use an independent second pass only for an uncertain P0/P1 or disputed architecture.
10. Do not assign implementation work during diagnosis.
11. Stop when the contract matrix is covered and new searches produce no materially new issue class.

Optimize for useful evidence per agent, not agent count.

Required output

1. Verdict

Issue a verdict only after completing the required investigation for the declared scope. Skipped available evidence means the audit is incomplete: continue investigating; do not issue a verdict or reclassify the omission as an Unverified risk.

For a completed audit, apply the first matching rule:

• FAIL: at least one verified P0/P1 defect.
• PASS WITH FINDINGS: no verified P0/P1, but verified P2/P3 findings or material evidence genuinely unavailable after investigation.
• PASS: no findings or material unresolved risks within the completed scope.

State unavailable evidence and its potential consequence explicitly: PASS WITH FINDINGS does not establish the absence of unverified higher-severity defects. A scoped PASS does not imply whole-system verification.

2. Executive summary

At most five bullets.

3. Findings

Order by severity, then confidence.

[P1][CONTEXT_MISMATCH] Short title

Evidence:
- path:line ...
- path:line ...

Why it matters:
...

Minimal durable fix:
...

Confidence: high

Do not bury findings in prose.

4. Instruction ownership map

Summarize meaningful ownership gaps/collisions across:

tool schema
tool description
controller scope
shared agent prompt
role body
skill
dynamic success guidance
dynamic error guidance
human docs

5. Contract coverage

For:

input
capability
instruction
identity
state
completion
configuration
recovery
distribution

report:

status: verified | finding | not applicable | not verified
scope: what was actually traced

Example:

recovery | verified | startup rollback, retained cleanup evidence, retry markers
completion | finding | compact cleanup-warning projection

verified applies only to the stated scope. Use not verified for required evidence genuinely unavailable after investigation, not skipped available evidence. State exclusions from a bounded scope explicitly; do not claim whole-system PASS from partial verification.

6. Remediation order

Give the shortest dependency-aware sequence.

Prefer:

fix root contract once
→ delete duplicate authority
→ add/update one discriminating regression
→ update canonical docs

7. Unverified risks

Include only genuinely unresolved material claims after the closure workflow. State known evidence, missing evidence, version/runtime scope, consequence, and exact resolver.

Omit when empty.

8. Non-blocking follow-up

Include only useful work not required to validate a material finding, such as a separate implementation task or non-material integration smoke.

Never put answerable material uncertainty here.

Omit when empty.

Finalization

Return the synthesized report without repeating completed review work.

Before returning, verify:

• the audited source snapshot and relevant active ADR constraints are established;
• every applicable contract chain has scoped coverage and honest limitations;
• effective capabilities, authority, identity, liveness, and legal next actions were assessed where applicable;
• P0/P1 findings have exact evidence;
• material P2 findings have closed evidence;
• result/state findings trace producer, reachability, and consumer semantics;
• material external assumptions were resolved for supported versions when possible;
• duplicate lens candidates were consolidated;
• fixes target root owners, not symptoms;
• only genuinely unresolved claims are Unverified risk;
• no answerable material question was deferred to follow-up;
• implementation remains separate from diagnosis;
• a verdict is issued only for completed investigation and follows the rules above.

If a check fails, reconcile the affected finding before finalizing.

Otherwise stop.

After the audit

Do not automatically fix findings.

If implementation is requested later:

1. use the audit report as canonical scope;
2. use the smallest capable execution Agent, or work locally when permitted;
3. keep unrelated cleanup out unless it simplifies the same proven root cause;
4. independently review the final diff against the audit findings;
5. run focused and repository validation appropriate to the change.