harness-eval
Skill9 starsChanged 7 months agoMITView on GitHub
This skill should be used when the user asks to "test the harness", "run integration tests", "validate features with real API", "test with real model calls", "run agent loop tests", "verify end-to-end", or needs to verify Nexus features on a real codebase with actual LLM calls.
- Reads credentials
- Installs packages
5 other projects use a version of this, 1 of them edited.
Adopted by
Most recently taken up by
- baekenough/oh-my-customcodetheir own version, first seen Nov 14, 2026
- Ronnasayd/aipptheir own version, first seen Oct 20, 2026
- MarlonAzevedo/claude-skillstheir own version, first seen Oct 11, 2026
- HKUDS/OpenHarnessedited, first seen Oct 8, 2026
- HanksChen0110/skillstheir own version, first seen Oct 3, 2026
Other versions5 projects use a version of this.
Word for word · 0
None found.
Edited · 1
- HKUDS/OpenHarness★ 16k95% same
Wrote their own version · 4
- baekenough/oh-my-customcode★ 42same name
- HanksChen0110/skills★ 2same name
- Ronnasayd/aipp★ 1same name
- MarlonAzevedo/claude-skills★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repositoryOne other file it gives its agents.
- pr-mergeSkill
The file
We split this skill into the rules it gives an agent
11 rules. 0 are common, 7 are distinctive, and 4 are very specific to this project.
0
Common
Most of the 5 edited versions we read keep these.
7
Distinctive
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 41
If the task is validating sandbox behavior, install and verify the actual runtime before running agent loops:
- Distinctive: line 65
If sandbox dependencies are missing, treat that as an environment/setup failure, not a feature regression.
- Very specific: line 88
Use a real external workspace such as AutoAgent
- Distinctive: line 89
Use real provider credentials and the actual target model
- Very specific: line 91
Use per-prompt timeouts large enough for real exploration, such as 240-600s
- Distinctive: line 93
Verify both text quality and tool traces
- Distinctive: line 94
Keep polling long-running sessions until they finish.
- Very specific: line 107
Verify the model adapts with glob/grep/read_file
- Distinctive: line 115
When a scenario fails, classify it before changing code:
- Very specific: line 152
If max_turns=200 and the run still fails, the next suspect is wall-clock timeout, not turn count
- Distinctive: line 175
Not checking tool call list — model may claim tool use without calling it
“Common” is measured: kept by at least 6 in 10 of the 5 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Harness Eval — End-to-End Feature Validation
Validate Nexus features by running real agent loops against an unfamiliar codebase with actual LLM API calls. Every test exercises the full stack: API client → model → tool calls → execution → result.
Core Principles
- Test on an unfamiliar project — never test on Nexus itself (the agent modifies its own code). Clone a real project as the workspace.
- Use real API calls — no mocks. Configure a real LLM endpoint.
- Multi-turn conversations — always test 2+ turns where the model needs prior context.
- Combine features — test hooks+skills+agent loop together, not in isolation.
- Verify tool execution — inspect tool call lists and output files, not just model text.
Workflow
1. Prepare Workspace
Clone an unfamiliar project (do not use Nexus):
git clone https://github.com/HKUDS/AutoAgent /tmp/eval-workspace2. Configure Environment
export ANTHROPIC_API_KEY=sk-xxx
export ANTHROPIC_BASE_URL=https://api.moonshot.cn/anthropic # or any provider
export ANTHROPIC_MODEL=kimi-k2.5For long-running real evals, do not artificially lower max_turns. Use the product default (200) unless the user explicitly wants a tighter bound.
3. Prepare Real Sandbox Runtime When Relevant
If the task is validating sandbox behavior, install and verify the actual runtime before running agent loops:
npm install -g @anthropic-ai/sandbox-runtime
sudo apt-get update
sudo apt-get install -y bubblewrap ripgrep
which srt
which bwrap
which rg
srt --versionThen run a minimal smoke check through Nexus, not just raw srt, so you verify the real adapter path:
from pathlib import Path
from nexus.config.settings import Settings, SandboxSettings, save_settings
from nexus.tools.bash_tool import BashTool
cfg = Path("/tmp/nexus-sandbox-settings.json")
save_settings(Settings(sandbox=SandboxSettings(enabled=True, fail_if_unavailable=True)), cfg)
# Point config loader at this file, then run BashTool on a tiny command such as `pwd`.If sandbox dependencies are missing, treat that as an environment/setup failure, not a feature regression.
4. Design Tests
Each test follows this pattern:
engine = make_engine(system_prompt="...", cwd=UNFAMILIAR_PROJECT)
evs1 = [ev async for ev in engine.submit_message("Read X, analyze Y")]
r1 = collect(evs1) # text, tools, turns, tokens
evs2 = [ev async for ev in engine.submit_message("Based on what you found...")]
r2 = collect(evs2)
assert "grep" in r1["tools"] # verify tools ranFor detailed code templates and the make_engine/collect helpers, consult references/test-patterns.md.
5. Prefer Long-Horizon, Real Agent Loops
For meaningful end-to-end validation, prefer unfamiliar-repo tasks that force multiple turns, context reuse, and mixed tool usage.
Recommended pattern:
- Use a real external workspace such as
AutoAgent - Use real provider credentials and the actual target model
- Keep
max_turns=200 - Use per-prompt timeouts large enough for real exploration, such as
240-600s - Require at least 2 turns per scenario
- Verify both text quality and tool traces
- Keep polling long-running sessions until they finish; do not abandon a run after the first long pause
Recommended long-horizon scenarios:
architecture_multiturn- Turn 1: map architecture, shell/subprocess surfaces, and test entrypoints
- Turn 2: identify top risks and propose refactors
- Turn 3: condense into onboarding or remediation actions
- Success:
bash,glob,grep,read_fileall appear; no timeout; noMaxTurnsExceeded
hook_block_and_recover- Force the model to try
bash - Block it with a real pre-tool hook
- Verify the model adapts with
glob/grep/read_file
- Force the model to try
sandbox_multiturn- Enable real sandbox settings with
fail_if_unavailable=true - First prompt must start with exactly one shell command such as
pwd && ls -la - Second prompt must explicitly reuse the prior shell findings
- Success:
bashexecutes via sandbox, non-shell tools continue the task, and the agent recovers from incidental repo errors
- Enable real sandbox settings with
When a scenario fails, classify it before changing code:
MaxTurnsExceeded: likely eval harness misconfiguration ifmax_turnswas manually loweredtimeout: task is too broad or per-prompt timeout is too small- sandbox unavailable: environment missing
srt,bwrap, orrg - tool error with task still completed: feature may still be healthy; inspect recovery behavior
6. Run Tests
python tests/test_merged_prs_on_autoagent.py # PR feature tests
python tests/test_real_large_tasks.py # large multi-step tasks
python tests/test_hooks_skills_plugins_real.py # hooks/skills/plugins
python -m pytest tests/ -q -k "not autoagent" # unit tests (no API)For ad hoc long-horizon validation, it is acceptable to run a temporary Python driver script as long as it:
- uses real Nexus engine/tool objects
- targets an unfamiliar repository
- prints per-scenario JSON summaries
- records tools, errors, turns, and token usage
- stays attached until completion
7. Interpret Results
| Result | Meaning | Action |
| PASS with tool calls | Feature works end-to-end | Done |
| PASS without tool calls | Model answered from knowledge | Rewrite prompt to force tool use |
| FAIL with exception | Code bug | Read traceback |
| FAIL with wrong output | Model behavior issue | Check system prompt and tool schemas |
| Timeout | Task too complex | Increase max_turns or simplify prompt |
For long-running real evals, refine the timeout guidance:
- First check whether
max_turnswas manually set too low - If
max_turns=200and the run still fails, the next suspect is wall-clock timeout, not turn count - Distinguish environment failures from product failures
- Example: missing dependency in the unfamiliar target repo is not automatically an Nexus regression
- Example: missing
srt/bwrap/rgis an eval environment issue
Feature Coverage Checklist
- Engine: multi-turn memory, tool chaining, parallel tools, error recovery, auto-compaction
- Swarm: InProcessBackend lifecycle, concurrent teammates, coordinator+notifications
- Hooks: pre_tool_use blocking → model adapts, post_tool_use firing
- Skills: skill tool invocation → model follows instructions
- Plugins: plugin-provided skill loaded and used in agent loop
- Memory: YAML frontmatter parsing, body content search, context injection
- Session: save → load → resume with context preserved
- Providers: Anthropic client, OpenAI client (with reasoning_content), multi-turn
- Cost: token accumulation across turns
Common Pitfalls
- Testing on Nexus itself — agent modifies its own running code
- Using mocks — misses serialization and API compatibility bugs
- Single-turn only — misses context accumulation and compaction bugs
- Artificially lowering
max_turnsduring real evals — can create false failures that do not reflect product defaults - Not checking tool call list — model may claim tool use without calling it
- Hardcoding paths — use
WORKSPACEvariable, skip in CI withpytest.mark.skipif - Declaring sandbox “tested” after only checking raw
srt— verify the Nexus adapter path too - Abandoning long tasks too early — some real tasks pause for minutes before the next event arrives
Additional Resources
Reference Files
references/test-patterns.md— Complete code templates formake_engine,collect, and each feature categoryreferences/feature-matrix.md— Detailed test cases for every Nexus module
Existing Test Files
Working test suites in the repo:
tests/test_merged_prs_on_autoagent.py— PR feature validationtests/test_real_large_tasks.py— Large multi-step taskstests/test_hooks_skills_plugins_real.py— Hooks/skills/plugins in agent loopstests/test_untested_features.py— Module-level integration tests
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

