agent-testing-patterns
Skill17 starsChanged 5 months agoApache-2.0Archived repositoryView on GitHub
Test AI agent systems including tool use, multi-turn conversations, error recovery, and non-deterministic outputs. Covers mock strategies, evaluation metrics, and regression testing for agent workflows. Triggers on AI agent testing, LLM evaluation, or agent quality assurance requests.
1 other project uses a version of this.
Other versions1 project uses a version of this.
Word for word · 1
- organvm-iv-taxis/a-i--skills★ 17100% same
Edited · 0
None found.
Wrote their own version · 0
None found.
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository170 other files it gives its agents.
- AGENTS.mdAGENTS.md
- docs/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- .github/copilot-instructions.mdCopilot instructions
- algorithmic-artSkill
- audio-engineering-patternsSkill
- canvas-designSkill
- creative-writing-craftSkill
- generative-art-algorithmsSkill
- generative-art-deploymentSkill
- generative-music-composerSkill
- interactive-theatre-designerSkill
- modular-synthesis-philosophySkill
- movement-notation-systemsSkill
- narratological-algorithmsSkill
- natural-center-extractionSkill
- react-three-fiber-patternsSkill
- reality-tv-narrative-analyzerSkill
- script-analysis-dramaturgicalSkill
- theme-factorySkill
- three-js-interactive-builderSkill
- corpus-persona-extractionSkill
- data-ingestion-pipelineSkill
- data-pipeline-architectSkill
- data-storytelling-analystSkill
- market-gap-analysisSkill
- ml-experiment-trackerSkill
- prompt-atom-formalizationSkill
- sql-query-optimizerSkill
- systemic-product-analystSkill
- time-series-analystSkill
- accessibility-patternsSkill
- api-design-patternsSkill
- artifacts-builderSkill
- backend-implementation-patternsSkill
- browser-extension-patternsSkill
- cli-tool-designSkill
- code-refactoring-patternsSkill
- code-review-checklistSkill
- coding-standards-enforcerSkill
- community-platform-patternsSkill
- configuration-managementSkill
- continuous-learning-agentSkill
- data-backup-patternsSkill
- database-migration-patternsSkill
- deployment-cicdSkill
- docker-containerizationSkill
- dotfile-systems-architectSkill
- error-handling-logging-patternsSkill
- fastapi-patternsSkill
- feature-workflow-orchestratorSkill
- frontend-design-systemsSkill
- fullstack-starter-packSkill
- gcp-resource-optimizerSkill
- htmx-interaction-patternsSkill
- iterative-code-explorationSkill
- json-schema-designSkill
- mcp-builderSkill
- mcp-server-orchestratorSkill
- mobile-platform-architectSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 1 is distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 22
Test AI agent systems that use tools, make decisions, and produce non-deterministic outputs.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Agent Testing Patterns
Test AI agent systems that use tools, make decisions, and produce non-deterministic outputs.
Testing Challenges
| Challenge | Cause | Strategy |
| Non-deterministic output | LLM randomness | Assert on structure, not exact text |
| Tool use sequences | Agent autonomy | Verify tool calls, not call order |
| Multi-turn state | Conversation context | Snapshot-based assertions |
| Cost | API calls | Mock LLM in unit tests |
| Latency | API round-trips | Parallel test execution |
| Flakiness | Model updates | Semantic assertions, not string matches |
Test Pyramid for Agents
╱╲
╱ ╲ E2E Agent Tests (few, expensive)
╱────╲ Full agent loop with real LLM
╱ ╲
╱────────╲ Integration Tests (moderate)
╱ ╲ Tool execution, state management
╱────────────╲
╱ Unit Tests ╲ Tool implementations, parsers, validators
╱────────────────╲Unit Testing (No LLM)
Tool Implementation Tests
import pytest
def test_file_read_tool():
tool = FileReadTool()
result = tool.execute({"path": "test.txt"})
assert result["content"] == "expected content"
assert result["success"] is True
def test_file_read_tool_missing_file():
tool = FileReadTool()
result = tool.execute({"path": "nonexistent.txt"})
assert result["success"] is False
assert "not found" in result["error"].lower()
def test_tool_input_validation():
tool = FileReadTool()
with pytest.raises(ValueError, match="path is required"):
tool.execute({})Response Parser Tests
def test_parse_tool_call():
raw = '{"tool": "search", "args": {"query": "python"}}'
result = parse_tool_call(raw)
assert result.tool == "search"
assert result.args == {"query": "python"}
def test_parse_malformed_tool_call():
raw = "not json at all"
result = parse_tool_call(raw)
assert result is NoneIntegration Testing (Mocked LLM)
Mock LLM Client
class MockLLMClient:
def __init__(self, responses: list[dict]):
self.responses = iter(responses)
self.calls: list[dict] = []
async def generate(self, messages: list[dict], tools: list[dict] = None) -> dict:
self.calls.append({"messages": messages, "tools": tools})
return next(self.responses)
@pytest.fixture
def mock_agent():
client = MockLLMClient(responses=[
{"content": None, "tool_calls": [{"name": "search", "args": {"query": "python packaging"}}]},
{"content": "Based on the search results, here's how to package Python..."},
])
return Agent(llm=client, tools=[SearchTool(), FileReadTool()])Tool Execution Sequence Tests
@pytest.mark.asyncio
async def test_agent_uses_search_then_responds(mock_agent):
result = await mock_agent.run("How do I package a Python project?")
# Verify tool was called
assert len(mock_agent.tool_history) == 1
assert mock_agent.tool_history[0].tool_name == "search"
assert "python" in mock_agent.tool_history[0].args["query"].lower()
# Verify final response exists
assert result.content is not None
assert len(result.content) > 0State Management Tests
@pytest.mark.asyncio
async def test_session_preserves_context(mock_agent):
await mock_agent.run("My name is Alice")
result = await mock_agent.run("What's my name?")
# Verify conversation history maintained
assert len(mock_agent.messages) == 4 # 2 user + 2 assistantE2E Testing (Real LLM)
Structural Assertions
@pytest.mark.e2e
@pytest.mark.asyncio
async def test_agent_creates_file(real_agent, tmp_path):
result = await real_agent.run(f"Create a Python hello world script at {tmp_path}/hello.py")
# Assert on outcome, not exact content
hello_file = tmp_path / "hello.py"
assert hello_file.exists()
content = hello_file.read_text()
assert "print" in content # Must use print
assert content.strip() # Non-empty
# Verify it's valid Python
compile(content, "hello.py", "exec")Semantic Assertions
@pytest.mark.e2e
@pytest.mark.asyncio
async def test_agent_explains_concept(real_agent):
result = await real_agent.run("Explain what a circuit breaker pattern is in 2-3 sentences")
# Semantic checks (not exact string matching)
assert len(result.content) > 50
assert len(result.content) < 1000
assert any(term in result.content.lower() for term in ["fault", "failure", "threshold", "open", "closed"])Evaluation Metrics
@dataclass
class AgentEvalResult:
task_completed: bool
tool_calls_count: int
tokens_used: int
latency_ms: float
error_recovery_count: int
async def evaluate_agent(agent, test_cases: list[dict]) -> list[AgentEvalResult]:
results = []
for case in test_cases:
start = time.perf_counter()
try:
result = await agent.run(case["prompt"])
completed = case["validator"](result)
except Exception:
completed = False
latency = (time.perf_counter() - start) * 1000
results.append(AgentEvalResult(
task_completed=completed,
tool_calls_count=len(agent.tool_history),
tokens_used=agent.total_tokens,
latency_ms=latency,
error_recovery_count=agent.error_count,
))
return resultsRegression Testing
Golden File Tests
def test_tool_call_format_regression():
"""Ensure tool call format hasn't changed."""
response = agent.format_tool_call("search", {"query": "test"})
expected = load_golden("tool_call_format.json")
assert response == expectedBenchmark Suite
BENCHMARK_CASES = [
{"prompt": "List all Python files in the project", "expected_tools": ["glob"], "max_tokens": 500},
{"prompt": "Fix the syntax error in app.py", "expected_tools": ["read", "edit"], "max_tokens": 2000},
]
async def run_benchmark(agent):
for case in BENCHMARK_CASES:
result = await agent.run(case["prompt"])
tools_used = {t.tool_name for t in agent.tool_history}
assert tools_used.issubset(set(case["expected_tools"] + ["think"]))
assert agent.total_tokens <= case["max_tokens"]Anti-Patterns
- Asserting exact LLM output — Models change; assert structure and semantics
- No mocking in unit tests — Real API calls make tests slow, expensive, and flaky
- Testing only happy path — Test error recovery, malformed responses, tool failures
- No cost tracking — Monitor token usage in E2E tests to catch regressions
- Ignoring non-determinism — Run E2E tests multiple times; set pass threshold (e.g., 4/5)
- Testing agent internals — Test outcomes and tool call patterns, not internal state
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

