shadow-eval
Skill1 starsChanged yesterdayMITView on GitHub
AI, LLM, RAG & Agent Evaluation and Red-Teaming. Use when evaluating RAG pipelines (Context Recall, Faithfulness, Citation Integrity), testing prompt/regex heuristics with precision/recall probes, or red-teaming OWASP Top 10 for LLM 2025 & NIST AI 600-1.
In this repository51 other files it gives its agents.
- antislop-codeSkill
- antislop-copywritingSkill
- antislop-humanSkill
- antislop-layoutmobileSkill
- antislopSkill
- antislop-uiSkill
- context7Skill
- handoffSkill
- shadow-accessibilitySkill
- shadow-apiSkill
- shadow-architectSkill
- shadow-authSkill
- shadow-bootstrapSkill
- shadow-browserSkill
- shadow-cacheSkill
- shadow-compatSkill
- shadow-concurrencySkill
- shadow-dataSkill
- shadow-dbSkill
- shadow-debugSkill
- shadow-designSkill
- shadow-discoverySkill
- shadow-docSkill
- shadow-domainSkill
- shadow-eaSkill
- shadow-finopsSkill
- shadow-gitSkill
- shadow-grcSkill
- shadow-handoverSkill
- shadow-integrateSkill
- shadow-maintainSkill
- shadow-mobileSkill
- shadow-modernizeSkill
- shadow-monarchSkill
- shadow-observeSkill
- shadow-opsSkill
- shadow-perfSkill
- shadow-planSkill
- shadow-pmoSkill
- shadow-presalesSkill
- shadow-privacySkill
- shadow-qualitySkill
- shadow-realtimeSkill
- shadow-refactorSkill
- shadow-releaseSkill
- shadow-resilienceSkill
- shadow-searchSkill
- shadow-secSkill
- shadow-supplychainSkill
- shadow-testSkill
- shadow-uatSkill
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 3 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 8
Use this skill whenever the Monarch invokes /shadow-eval, builds or modifies RAG pipelines, prompt classifiers, LLM-as-a-Judge workflows, …
- Very specific: line 27
Faithfulness / Groundedness: every factual claim and citation in the answer is directly supported by retrieved context (prevents LLM09:2025 …
- Very specific: line 32
Run Precision Probes (>= 5 realistic inputs that must NOT trigger, e.g., non-anaphoric root words like hanya/punya/bertanya in Indonesian …
- Distinctive: line 33
Run Recall Probes (>= 3 differently phrased inputs that MUST trigger).
- Distinctive: line 34
Never treat positive tests alone as proof of safety.
- Distinctive: line 37
Verify technical defenses across all 10 categories:
- Very specific: line 40
LLM05:2025Improper Output Handling: Never pass raw LLM output to eval(), exec(), shell=True, raw SQL interpolation, or unsanitized HTML …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
/shadow-eval — AI/LLM Evaluation, RAG Quality & GenAI Red-Teaming Gate
Use this skill whenever the Monarch invokes /shadow-eval, builds or modifies RAG pipelines, prompt classifiers, LLM-as-a-Judge workflows, or autonomous tool-calling agents, or prepares an AI feature for Staging/Pre-UAT release.
Grounded in authoritative AI risk and security standards documented in references/NIST-AI-600-1-AND-OWASP-LLM-2025.md:
- OWASP Top 10 for Large Language Model Applications (2025 Edition) (
LLM01:2025throughLLM10:2025) —https://genai.owasp.org/llm-top-10/. - NIST AI RMF 1.0 (
AI 100-1) & Generative AI Profile (NIST AI 600-1, July 2024) —https://doi.org/10.6028/NIST.AI.600-1(Govern,Map,Measure,Manage).
1. The 5-Pillar AI & RAG Evaluation Workflow
Pillar 1: Golden Dataset & Adversarial Contrast Set
- Curate representative real-world queries plus out-of-scope queries ("must refuse / I don't know"), multilingual/code-switched phrasing, and topic-switch negative probes (so follow-up heuristics or query rewriters do not pollute new topics).
- Tag every dataset item with
tenant_id,expected_chunk_ids,expected_claims, andmust_refuseboolean.
Pillar 2: Retrieval & RAG Triad Metrics (NIST AI 600-1 Measure)
Measure retrieval and generation separately so failures are attributable:
- Retrieval Layer:
Recall@K,MRR@K,NDCG@K, and Tenant ACL Isolation = 100% (zero cross-tenant chunks allowed in context, enforcingLLM08:2025). - Generation Layer (RAG Triad):
- Context Relevance / Precision: retrieved chunks actually contain evidence for the query.
- Faithfulness / Groundedness: every factual claim and citation in the answer is directly supported by retrieved context (prevents
LLM09:2025hallucinations). - Answer Relevance: response directly resolves the user's question without evading or drifting.
Pillar 3: Heuristic, Regex & Router Falsification
Whenever an AI pipeline includes deterministic pre-filters, follow-up detectors, or intent routers:
- Run Precision Probes (>= 5 realistic inputs that must NOT trigger, e.g., non-anaphoric root words like
hanya/punya/bertanyain Indonesian-nyafilters). - Run Recall Probes (>= 3 differently phrased inputs that MUST trigger).
- Never treat positive tests alone as proof of safety.
Pillar 4: OWASP Top 10 for LLM Applications (2025) Red-Teaming
Verify technical defenses across all 10 categories:
LLM01:2025Prompt Injection: Delimit untrusted retrieved/user text, validate tool schemas, and separate system instructions from retrieved content.LLM02:2025Sensitive Information Disclosure &LLM07:2025System Prompt Leakage: Redact PII/secrets before prompt assembly; never embed API keys or sensitive business thresholds in system prompts.LLM05:2025Improper Output Handling: Never pass raw LLM output toeval(),exec(),shell=True, raw SQL interpolation, or unsanitized HTML (dangerouslySetInnerHTML).LLM06:2025Excessive Agency: Enforce least-privilege tool scopes and mandatory Human-in-the-Loop (HITL) confirmation for financial, destructive, or external communication tools.LLM10:2025Unbounded Consumption: Capmax_output_tokens,thinking_budget, per-user rate limits, and max agent loop iterations.
Pillar 5: Automated Eval & Red-Team Gate Script
python "shadow-eval/scripts/ai_eval_gate.py" --self-test
python "shadow-eval/scripts/ai_eval_gate.py" --score-rag path/to/rag_eval_dataset.json
python "shadow-eval/scripts/ai_eval_gate.py" --redteam-check path/to/src_or_prompt.pyReport metrics with explicit Claim -> Measured Score -> Threshold -> Untested Surfaces rather than subjective "model sounds good" claims.
Pillar 6: Search Retrieval Handoff, Dataset Splits & Independent Judge Calibration
/shadow-searchRetrieval Handoff: Split RAG evaluation strictly into Retrieval (/shadow-search: Recall@k, MRR, NDCG@k, pre-generation ACL tenant leak rate) vs Generation (/shadow-eval: faithfulness, citation grounding, refusal accuracy). Never tune prompts to mask broken retrieval ranking or missing RBAC filters.- Dataset Versioning & Held-Out Split: Version every golden evaluation dataset with an immutable version tag/SHA hash (
dataset_vX.Y). Enforce a strict Dev/Tuning Split vs Blind Held-Out Split so prompt engineers cannot overfit system prompts against the acceptance gate set. - Independent Judge Calibration: Never rely on the same model family/temperature that generated outputs as an uncalibrated sole judge. Calibrate LLM-as-a-judge rubrics against human-labeled anchor items, mitigate positional/verbosity bias via swapped-order trials, and measure inter-rater agreement (e.g., Cohen's kappa or accuracy >= threshold) before promoting judge thresholds.
- Execution Approval Boundary (
L1vsL3): Offline deterministicai_eval_gate.pyscoring is boundedL1_SAFE_VERIFICATION, whereas high-cost live LLM judge sweeps or--agentbatches (>= $API budget) areL3: RESOURCE_INTENSIVErequiring explicit 7-part Monarch approval with estimated token cost (shadow-monarch/scripts/execution_approval_gate.py).
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

