agentleFS
tamldtamld/defense-in-depth/.agents/skills/skill-ai-dspy-validator/SKILL.md

skill-ai-dspy-validator

Skill4 starsChanged 7 months agoMITView on GitHub

Evaluate the quality of a DSPy integration in DiD — endpoint correctness, graceful degradation, and WARN-NOT-BLOCK contract.

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    AbortController is mandatory: Every fetch() call to a DSPy endpoint must have an AbortController with a configurable timeout.

    line 33

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

SKILL: skill-ai-dspy-validator

Identity

Role: The DSPy Integration Auditor
Philosophy: An AI evaluator that can BLOCK governance decisions is an AI evaluator that needs to be Red-Teamed.


Mission

Audit a DSPy integration in DiD to verify:

  1. The WARN-NOT-BLOCK contract is respected (DSPy never emits BLOCK)
  2. Graceful degradation is real and tested (timeout → null → pipeline continues)
  3. The endpoint is correctly guarded by AbortController with timeout
  4. Quality gate failure (score < 0.5) does not crash — it rejects with a message
  5. The integration is covered by tests that simulate DSPy being unreachable

Hard Gates

  1. WARN-NOT-BLOCK is absolute: Any DSPy integration that emits a BLOCK finding is architecturally invalid. Period.
  2. AbortController is mandatory: Every fetch() call to a DSPy endpoint must have an AbortController with a configurable timeout.
  3. null return on failure: callDspy() must return null on any error — network, timeout, non-2xx — never throw.
  4. Test required: Any code path that calls callDspy() must have a corresponding test that simulates DSPy being unreachable.

Workflow

Step 1: Audit the call site

For every call to callDspy() or callDspyRank(), verify:

☐ AbortController present with configurable timeoutMs
☐ Response status checked (non-2xx → return null)
☐ catch block returns null (not throws)
☐ Result drives WARN severity only (never BLOCK)

Step 2: Audit the integration point (hollow-artifact, memory, etc.)

☐ useDspy=false is the default
☐ DSPy is only called AFTER Tier 0 checks pass
☐ DSPy result = null → pipeline continues without error
☐ DSPy result.score < 0.5 → WARN finding (with feedback), not BLOCK

Step 3: Audit the quality gate (memory.ts)

☐ DSPy fail → lesson is persisted (graceful degrade)
☐ A WARN is logged when quality gate is skipped due to DSPy failure
☐ qualityScore === null in result when DSPy unavailable (not 0)

Step 4: Write the audit report

## DSPy Integration Audit: <module>

### WARN-NOT-BLOCK Contract: ✅ / ❌
<detail>

### AbortController: ✅ / ❌
Timeout: <configured value> ms
<detail>

### Graceful Degradation: ✅ / ❌
- Network fail: null returned? ✅ / ❌
- Timeout: null returned? ✅ / ❌
- Non-2xx: null returned? ✅ / ❌

### Quality Gate (if applicable): ✅ / ❌
- WARN emitted on skip? ✅ / ❌
- qualityScore === null (not 0)? ✅ / ❌

### Test Coverage: ✅ / ❌
- DSPy unreachable test exists? ✅ / ❌
- Timeout simulation test exists? ✅ / ❌

### Verdict: COMPLIANT / NON-COMPLIANT

Context: DiD DSPy Architecture

src/core/dspy-client.ts  — callDspy(), callDspyRank() — shared client
src/guards/hollow-artifact.ts — L3 opt-in semantic eval
src/core/memory.ts — quality gate + semantic search
src/cli/eval.ts — forced-on DSPy eval for manual inspection

Reference implementation: src/core/dspy-client.ts — four architectural contracts are documented in the file header comment.


Output Contract

A structured audit report (markdown). If non-compliant issues found, create GitHub issues with labels bug and priority: P1.


Anti-Patterns

  1. Accepting "graceful degradation" without testing it — If there's no test that kills the DSPy server, degradation is a claim, not a fact.
  2. Using evalResult.score === 0 to mean "DSPy failed" — Score 0 means the content was evaluated and scored 0. null means DSPy was unavailable.
  3. Logging DSPy failures to stdout — Use console.warn() to stderr. stdout is for structured output.
  4. Treating timeout as an error — AbortController timeout is expected behavior. It should log WARN, not ERROR.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.