skill-ai-dspy-validator
Skill4 starsChanged 7 months agoMITView on GitHub
Evaluate the quality of a DSPy integration in DiD — endpoint correctness, graceful degradation, and WARN-NOT-BLOCK contract.
In this repository16 other files it gives its agents.
- .agents/AGENTS.mdAGENTS.md
- AGENTS.mdAGENTS.md
- .agents/skills/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- Bootstrap AgentSkill
- CreatorSkill
- Deep ResearchSkill
- skill-devops-github-actionsSkill
- skill-guard-governanceSkill
- skill-impact-predictorSkill
- skill-review-codeSkill
- Self ReflectionSkill
- skill-surgical-refactorerSkill
- skill-test-architectSkill
- skill-threat-modeling-expertSkill
- [Skill Name]Skill
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 33
AbortController is mandatory: Every fetch() call to a DSPy endpoint must have an AbortController with a configurable timeout.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
SKILL: skill-ai-dspy-validator
Identity
Role: The DSPy Integration Auditor
Philosophy: An AI evaluator that can BLOCK governance decisions is an AI evaluator that needs to be Red-Teamed.
Mission
Audit a DSPy integration in DiD to verify:
- The WARN-NOT-BLOCK contract is respected (DSPy never emits BLOCK)
- Graceful degradation is real and tested (timeout → null → pipeline continues)
- The endpoint is correctly guarded by AbortController with timeout
- Quality gate failure (score < 0.5) does not crash — it rejects with a message
- The integration is covered by tests that simulate DSPy being unreachable
Hard Gates
- WARN-NOT-BLOCK is absolute: Any DSPy integration that emits a BLOCK finding is architecturally invalid. Period.
- AbortController is mandatory: Every
fetch()call to a DSPy endpoint must have an AbortController with a configurable timeout. nullreturn on failure:callDspy()must returnnullon any error — network, timeout, non-2xx — never throw.- Test required: Any code path that calls
callDspy()must have a corresponding test that simulates DSPy being unreachable.
Workflow
Step 1: Audit the call site
For every call to callDspy() or callDspyRank(), verify:
☐ AbortController present with configurable timeoutMs
☐ Response status checked (non-2xx → return null)
☐ catch block returns null (not throws)
☐ Result drives WARN severity only (never BLOCK)Step 2: Audit the integration point (hollow-artifact, memory, etc.)
☐ useDspy=false is the default
☐ DSPy is only called AFTER Tier 0 checks pass
☐ DSPy result = null → pipeline continues without error
☐ DSPy result.score < 0.5 → WARN finding (with feedback), not BLOCKStep 3: Audit the quality gate (memory.ts)
☐ DSPy fail → lesson is persisted (graceful degrade)
☐ A WARN is logged when quality gate is skipped due to DSPy failure
☐ qualityScore === null in result when DSPy unavailable (not 0)Step 4: Write the audit report
## DSPy Integration Audit: <module>
### WARN-NOT-BLOCK Contract: ✅ / ❌
<detail>
### AbortController: ✅ / ❌
Timeout: <configured value> ms
<detail>
### Graceful Degradation: ✅ / ❌
- Network fail: null returned? ✅ / ❌
- Timeout: null returned? ✅ / ❌
- Non-2xx: null returned? ✅ / ❌
### Quality Gate (if applicable): ✅ / ❌
- WARN emitted on skip? ✅ / ❌
- qualityScore === null (not 0)? ✅ / ❌
### Test Coverage: ✅ / ❌
- DSPy unreachable test exists? ✅ / ❌
- Timeout simulation test exists? ✅ / ❌
### Verdict: COMPLIANT / NON-COMPLIANTContext: DiD DSPy Architecture
src/core/dspy-client.ts — callDspy(), callDspyRank() — shared client
src/guards/hollow-artifact.ts — L3 opt-in semantic eval
src/core/memory.ts — quality gate + semantic search
src/cli/eval.ts — forced-on DSPy eval for manual inspectionReference implementation: src/core/dspy-client.ts — four architectural contracts
are documented in the file header comment.
Output Contract
A structured audit report (markdown). If non-compliant issues found, create
GitHub issues with labels bug and priority: P1.
Anti-Patterns
- Accepting "graceful degradation" without testing it — If there's no test that kills the DSPy server, degradation is a claim, not a fact.
- Using
evalResult.score === 0to mean "DSPy failed" — Score 0 means the content was evaluated and scored 0.nullmeans DSPy was unavailable. - Logging DSPy failures to stdout — Use
console.warn()to stderr. stdout is for structured output. - Treating timeout as an error — AbortController timeout is expected behavior. It should log WARN, not ERROR.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

