agentleFS
Mathews-TomMathews-Tom/armory/skills/prompt-lab/SKILL.md

prompt-lab

Skill340 starsChanged 8 months agoMITView on GitHub

LLM prompt engineering: analyzes failure modes, generates variants (direct, few-shot, CoT), designs rubrics, produces test suites. Triggers on: "prompt engineering", "generate prompt variants", "A/B test prompts", "optimize prompt", "improve this prompt". NOT for SKILL.md files, use skill-evaluator.

1 other project uses a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 3 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    If an existing prompt is provided:

    line 47
  2. Distinctive:

    Create 2-4 prompt variants, each testing a different hypothesis:

    line 59
  3. Distinctive:

    Simplest effective prompt wins: If a zero-shot prompt scores as well as a few-shot prompt, use the zero-shot.

    line 180

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Prompt Lab

Replaces trial-and-error prompt engineering with structured methodology: objective definition, current prompt analysis, variant generation (instruction clarity, example strategies, output format specification), evaluation rubric design, test case creation, and failure mode identification.

Reference Files

FileContentsLoad When
references/prompt-patterns.mdPrompt structure catalog: zero-shot, few-shot, CoT, persona, structured outputAlways
references/evaluation-metrics.mdQuality metrics (accuracy, format compliance, completeness), rubric designEvaluation needed
references/failure-modes.mdCommon prompt failure taxonomy, detection strategies, mitigationsFailure analysis requested
references/output-constraints.mdTechniques for constraining LLM output format, JSON mode, schema enforcementFormat control needed

Prerequisites

  • Clear objective: what should the prompt accomplish?
  • Target model (GPT-4, Claude, open-source) — prompting techniques vary by model
  • Current prompt (if improving) or task description (if creating)

Workflow

Phase 1: Define Objective

  1. Task specification — What should the LLM produce? Be specific: "Classify customer support tickets into 5 categories" not "Handle support tickets."
  2. Success criteria — How do you know the output is correct? Define measurable criteria before writing any prompt.
  3. Failure modes — What does a bad output look like? Missing information? Wrong format? Hallucinated content? Refusal to answer?

Phase 2: Analyze Current Prompt

If an existing prompt is provided:

  1. Structure assessment — Is the instruction clear? Are examples provided? Is the output format specified?
  2. Ambiguity detection — Where could the model misinterpret the instruction?
  3. Missing components — What's not specified that should be? (output format, tone, length constraints, edge case handling)
  4. Failure mode mapping — Which known failure patterns (see references/failure-modes.md) apply to this prompt?

Phase 3: Generate Variants

Create 2-4 prompt variants, each testing a different hypothesis:

Variant TypeHypothesisWhen to Use
Direct instructionClear instruction is sufficientSimple tasks, capable models
Few-shotExamples improve output consistencyPattern-following tasks
Chain-of-thoughtReasoning improves accuracyMulti-step logic, math, analysis
Persona/roleRole framing improves tone/expertiseDomain-specific tasks
Structured outputFormat specification prevents errorsJSON, CSV, specific templates

For each variant:

  • State the hypothesis (why this variant might work)
  • Identify the risk (what could go wrong)
  • Provide the complete prompt text

Phase 4: Design Evaluation

  1. Rubric — Define weighted criteria:

    CriterionWhat It MeasuresTypical Weight
    CorrectnessOutput matches expected answer30-50%
    Format complianceFollows specified structure15-25%
    CompletenessAll required elements present15-25%
    ConcisenessNo unnecessary content5-15%
    Tone/styleMatches requested voice5-10%
  2. Test cases — Minimum 5 cases covering:
    • Happy path (standard input)
    • Edge cases (unusual but valid input)
    • Adversarial cases (inputs designed to confuse)
    • Boundary cases (minimum/maximum input)

Phase 5: Output

Present variants, rubric, and test cases in a structured format ready for execution.

Output Format

## Prompt Lab: {Task Name}

### Objective
{What the prompt should achieve — specific and measurable}

### Success Criteria
- [ ] {Criterion 1 — measurable}
- [ ] {Criterion 2 — measurable}

### Current Prompt Analysis
{If existing prompt provided}
- **Strengths:** {what works}
- **Weaknesses:** {what fails or is ambiguous}
- **Missing:** {what's not specified}

### Variants

#### Variant A: {Strategy Name}

{Complete prompt text}

**Hypothesis:** {Why this approach might work}
**Risk:** {What could go wrong}

#### Variant B: {Strategy Name}

{Complete prompt text}

**Hypothesis:** {Why this approach might work}
**Risk:** {What could go wrong}

#### Variant C: {Strategy Name}

{Complete prompt text}

**Hypothesis:** {Why this approach might work}
**Risk:** {What could go wrong}

### Evaluation Rubric

| Criterion | Weight | Scoring |
|-----------|--------|---------|
| {criterion} | {%} | {how to score: 0-3 scale or pass/fail} |

### Test Cases

| # | Input | Expected Output | Tests Criteria |
|---|-------|-----------------|---------------|
| 1 | {standard input} | {expected} | Correctness, Format |
| 2 | {edge case} | {expected} | Completeness |
| 3 | {adversarial} | {expected} | Robustness |

### Failure Modes to Monitor
- {Failure mode 1}: {detection method}
- {Failure mode 2}: {detection method}

### Recommended Next Steps
1. Run all variants against the test suite
2. Score using the rubric
3. Select the highest-scoring variant
4. Iterate on the winner with targeted improvements

Calibration Rules

  1. One variable per variant. Each variant should change ONE thing from the baseline. Changing instruction style AND examples AND format simultaneously makes results uninterpretable.
  2. Test before declaring success. A prompt that works on 3 examples may fail on the 4th. Minimum 5 diverse test cases before concluding a variant works.
  3. Failure modes are more valuable than successes. Understanding WHY a prompt fails guides improvement more than confirming it works.
  4. Model-specific optimization. A prompt optimized for GPT-4 may not work for Claude or Llama. Always note the target model.
  5. Simplest effective prompt wins. If a zero-shot prompt scores as well as a few-shot prompt, use the zero-shot. Fewer tokens = lower cost + latency.

Error Handling

ProblemResolution
No clear objectiveAsk the user to define what "good output" looks like with 2-3 examples.
Prompt is for a task LLMs are bad at (math, counting)Flag the limitation. Suggest tool-augmented approaches or pre/post-processing.
Too many variables to testFocus on the highest-impact variable first. Iterative refinement beats combinatorial testing.
No existing prompt to analyzeStart with the simplest possible prompt. The first variant IS the baseline.
Output format requirements are strictUse structured output mode (JSON mode, function calling) instead of prompt-only constraints.

When NOT to Use

Push back if:

  • The task doesn't need an LLM (deterministic rules, regex, SQL) — use the right tool
  • The user wants prompt execution, not design — this skill designs and evaluates, it doesn't run prompts
  • The prompt is for safety-critical decisions without human review — LLM output should not be the sole input

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.