Agent instruction audit

Evaluate whether a specific agent instruction or prompt rule misdirects work, induces superfluous tool calls, or wastes tokens. Test by comparing matched, fresh-context runs against an identical task and repository snapshot while changing only the suspect instruction.

Triggers

• An agent prompt instruction, rule, or system guideline is suspected of causing unneeded tool calls, investigative detours, omissions, or degraded output quality.
• Maintainers seek empirical evidence before adding, altering, or removing a guideline from agent instructions.
• Team observations indicate that an instruction incurs context overhead or user friction disproportionate to its benefit.

Non-triggers

• General prompt rewriting, stylistic editing, or aesthetic polishing without an isolated suspect instruction.
• Simultaneously changing multiple instructions or evaluating different underlying foundation models.
• Removing safety, authorization, read-only constraints, or security safeguards to optimize token counts or speed.
• Claiming generalized benchmark conclusions from an isolated run or small sample.

Process

1. Isolate the suspect instruction. Quote the exact text of the single instruction under review. Formulate a specific hypothesis describing the failure mode (such as an unnecessary directory scan or redundant confirmation) and identify observable outcomes that confirm or refute it. Keep all unrelated instructions unchanged.
2. Freeze comparison conditions. Hold all operational variables constant across experimental conditions:
   • Identical repository snapshot and commit.
   • Identical task wording and objectives.
   • Identical model version, temperature, and generation parameters.
   • Identical available tool definitions and permission boundaries.
   • Fresh context for each run (no shared session memory, previous turn history, or scratchpad artifacts).
   • Change only the isolated instruction: Condition A (instruction present) versus Condition B (instruction absent). Do not substitute an alternative instruction unless the replacement itself is the explicit test variable.
3. Execute controlled matched runs. Run Condition A and Condition B under identical environment constraints. Randomize or alternate execution order where practical. Use read-only repository snapshots or isolated sandboxes to prevent cross-run pollution.
4. Collect traces and telemetry. Record full input prompts, assistant responses, tool-call logs, and resource metrics (prompt tokens, completion tokens, total tokens, wall-clock duration). If monetary cost tracking is available, record it; otherwise explicitly note cost as unmeasured.
5. Evaluate against a predefined rubric. Prior to inspecting results, establish an objective rubric evaluating: task correctness, completeness, adherence to constraints, safety safeguards, and tool efficiency. Assess substantive behavior and quality, not merely response brevity.
6. Propose a minimal change. If empirical findings show the instruction causes misdirection or wasted resources without compromising safety or output quality, formulate a small reviewable diff or draft PR modifying or removing only that instruction. Never modify live production prompts automatically.

Evidence and Reporting

Provide a structured comparison report containing:

• Variable & Hypothesis: Exact quoted instruction, hypothesized mechanism of inefficiency, and observable predictions.
• Controls & Environment: Documented repository commit, model version, generation settings, tool configuration, and fresh-context confirmation.
• Comparison Results: Side-by-side comparison of Condition A vs. Condition B detailing: task completion status, tool-call trace differences, token consumption, elapsed duration, and rubric evaluations.
• Analysis: Observed behavioral divergence, clear separation of verified facts from hypotheses, confidence rating, and identified confounds (e.g., model stochasticity).
• Recommendation: Clear proposal to retain, revise, or remove the instruction, backed by a reviewable diff.

Verification

• Replication Runs: Conduct repeated run pairs (e.g., 3–5 matched pairs) whenever feasible to establish consistency across model variance.
• Control Task Testing: Evaluate the proposed instruction modification against adjacent control tasks to verify that intended behaviors and safety constraints remain effective.
• Guardrail Integrity: Confirm that security restrictions, safety rules, and operational safeguards were not weakened.
• Sample Caveats: Explicitly state sample size limitations, noting that isolated run pairs constitute case studies rather than universal benchmark claims.

Example

Trigger: An agent prompt instruction—"Always list the full repository root directory before executing any targeted tool"—is suspected of causing redundant tool calls on narrowly scoped bugfix tasks.

Experimental Setup:

• Suspect instruction isolated against a controlled bugfix task (updating a regex pattern in src/utils/text.py).
• Fresh contexts used for both runs with identical commit snapshot, model version, temperature, and tool access.
• Condition A includes the suspect instruction; Condition B omits it.

Matched Run Comparison (Single Pair):

• Condition A (instruction present): Completed task successfully; executed 4 tool calls (including 1 root directory listing); consumed 3,250 total tokens across 14.2s elapsed time (hypothetical illustrative values).
• Condition B (instruction absent): Completed task successfully with identical patch output; executed 3 tool calls (direct file read without root listing); consumed 2,680 total tokens across 11.5s elapsed time (hypothetical illustrative values).

Analysis & Caution:

• The single matched pair suggests a reduction of 1 unnecessary tool call and ~570 tokens without quality loss.
• However, this single observation is too small to establish general benefit or discount model stochasticity.
• Recommendation: Do not alter the instruction based on one run. Replicate across 3–5 matched task pairs representing diverse maintenance workflows before proposing a prompt diff.