agentic-workflow-testing
Skill1 starsChanged 17 days agoMITView on GitHub
Test complete agent workflows with unit, contract, integration, trajectory, fault-injection, and regression tests while controlling nondeterminism and external side effects.
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 4 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 9
Test agent systems as stateful programs with model variability, external tools, and side effects.
- Distinctive: line 22
Use fake tools and deterministic fixtures for fast tests.
- Distinctive: line 26
Keep flaky tests visible; do not hide failures with unbounded retries.
- Distinctive: line 28
Run tests at the right gates: quick checks locally, broader suites in CI, and live smoke tests after deployment.
≈ “Common” is an estimate: at least 1 in 5 of the 308,232 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Agentic Workflow Testing
Goal
Test agent systems as stateful programs with model variability, external tools, and side effects. A successful final response alone is insufficient evidence that the workflow behaved safely.
Test layers
- Unit tests: deterministic policies, schemas, reducers, ranking logic, state transitions, and argument validators.
- Contract tests: tool request/response schemas, error categories, authorization, and compatibility.
- Integration tests: model/provider adapters, retrieval, persistence, tool server, and checkpoint store.
- Trajectory tests: expected tool sequence or allowed path, state changes, approvals, and terminal conditions.
- Evaluation tests: semantic success, groundedness, tool choice, and rubric-based behavior on a versioned dataset.
- Fault-injection tests: timeout, rate limit, malformed output, unavailable tool, partial side effect, cancellation, and resume.
- Security tests: prompt injection, unauthorized data access, cross-tenant leakage, excessive agency, and approval bypass.
Workflow
- Define observable acceptance criteria and forbidden outcomes.
- Use fake tools and deterministic fixtures for fast tests; reserve real external integrations for selected integration tests.
- Stub or record model calls where appropriate, but keep a smaller live-model evaluation suite for behavior changes that mocks cannot reveal.
- Assert final state and side effects, not only text output.
- Make tests isolated, resettable, and safe to rerun. Use unique test identities and idempotency keys.
- Keep flaky tests visible; do not hide failures with unbounded retries.
- Store representative failed trajectories as regression fixtures, with sensitive data removed.
- Run tests at the right gates: quick checks locally, broader suites in CI, and live smoke tests after deployment.
Determinism and tolerance
Model outputs can vary. Prefer invariant-based assertions, schema checks, and allowed tool-path sets over brittle exact-text matches. For semantic graders, use calibrated thresholds and human spot checks. Do not loosen assertions merely to make a broken behavior pass.
Deliverable
Provide test layers, fixtures, environment controls, side-effect assertions, fault matrix, CI gates, and coverage gaps.
References
- OpenAI evals: https://platform.openai.com/docs/guides/evals
- LangGraph testing: https://docs.langchain.com/oss/python/langgraph/test
- Superpowers testing methodology: https://github.com/obra/superpowers
In this repository28 other files it gives its agents.
- agent-architecture-designSkill
- agent-evaluationSkill
- agent-harness-engineeringSkill
- agent-memory-designSkill
- browser-computer-useSkill
- durable-agent-workflowsSkill
- human-in-the-loopSkill
- mcp-developmentSkill
- mcp-server-developmentSkill
- multi-agent-orchestrationSkill
- tool-calling-designSkill
- advanced-retrieval-rerankingSkill
- context-engineeringSkill
- embedding-model-engineeringSkill
- llm-evaluationSkill
- llm-observabilitySkill
- model-serving-inferenceSkill
- multimodal-ai-pipelinesSkill
- prompt-engineeringSkill
- rag-engineeringSkill
- rag-pipeline-engineeringSkill
- structured-outputsSkill
- agentic-securitySkill
- security-reviewSkill
- code-reviewSkill
- debuggingSkill
- test-engineeringSkill
- example-skillSkill
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
Reviews can't be read right now.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.