verify-agent-server
Skill1.4k starsChanged 8 days agoMITView on GitHub
This skill should be used to "verify Agent Server APIs", "test the REST or WebSocket API like an agent would", "drive the agent-server", "check an API change against a real server", "create or update the agent-server feature map", or "run the weekly API audit". Ships control-agent-server (launch, doctor, api, ws, llm, conversations, fixtures, exec, assert, evidence, map) and the maintained map of every agent-facing feature.
- Reads credentials
Trending
More talk lately.about the whole repository
Who's talking about OpenHands/software-agent-sdk
- NousResearch/hermes-agent“Feature: OpenHands Coding Agent Skill — Model-Agnostic Sandboxed Code Agent Delegation”GitHub · Mar 6, 2026 · 4 reactions
- A Hacker News story“Show HN: OpenHands Software Agent SDK”Hacker News · Nov 19, 2025 · 2 points
- OpenHands/docs“docs(sdk): explain recording outcome selection and restart provenance”GitHub · Nov 12, 2026 · 1 reactions
- OpenHands/enterprise“feat: tell the agent whether each secret is personal or org-shared”GitHub · Nov 9, 2026 · 1 reactions
- shanemort1982/software-agent-sdk“fix(sdk): re-check cancellation after acquiring resource lock [REVIEW COPY for upstream #4866]”GitHub · Sep 5, 2026 · 1 reactions
The biggest 5 of 51 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 51
Talked about on GitHub and Hacker News (51 times).
On GitHub · 50
- Mar 6, 2026Feature: OpenHands Coding Agent Skill — Model-Agnostic Sandboxed Code Agent Delegation — NousResearch/hermes-agent4 reactions · 1 comment
- Nov 12, 2026docs(sdk): explain recording outcome selection and restart provenance — OpenHands/docs1 reaction · 1 comment
- Nov 9, 2026feat: tell the agent whether each secret is personal or org-shared — OpenHands/enterprise1 reaction · 2 comments
- Sep 5, 2026fix(sdk): re-check cancellation after acquiring resource lock [REVIEW COPY for upstream #4866] — shanemort1982/software-agent-sdk1 reaction · 9 comments
- Jul 20, 2026Support host-managed ChatGPT tokens through Codex app-server external auth — agentclientprotocol/codex-acp1 reaction
45 more on GitHub
- Jun 9, 2026refactor(llm): clean up reverse-mapping OpenHands provider models — OpenHands/OpenHands1 reaction · 19 comments
- Apr 25, 2026Errors when developing agent-server-gui on OpenHands Cloud — OpenHands/agent-canvas1 reaction · 4 comments
- Mar 21, 2026[Feature]: Add Support for Switching Models via llama.cpp Compatible API — OpenHands/OpenHands1 reaction · 13 comments
- Mar 6, 2026Add release notes generator plugin — OpenHands/extensions1 reaction · 19 comments
- Mar 6, 2026[RFC] CLOUD - OpenHands Automations — Scheduled & Event-Driven Conversations — OpenHands/OpenHands1 reaction · 21 comments
- Feb 18, 2026Package PR review as a plugin — OpenHands/extensions1 reaction · 2 comments
- Feb 13, 2026Make the timeout of `Converstaion.run` configurable — OpenHands/benchmarks1 reaction · 2 comments
- Jan 26, 2026Implement the cluster admin — rbren/cluster-admin-agent1 reaction · 2 comments
- Jan 15, 2026Add Critic Result Display in TUI — OpenHands/OpenHands-CLI1 reaction · 13 comments
- Dec 18, 2025feat: Support custom security policy templates — OpenHands/OpenHands-CLI1 reaction · 3 comments
- Dec 15, 2025PR information is not current, even after pressing refresh — OpenHands/community-pr-dashboard1 reaction · 3 comments
- Dec 8, 2025SWE-Bench evaluation discrepancy: 30/50 vs 39/50 resolved instances on same execution data — OpenHands/benchmarks1 reaction · 8 comments
- Nov 3, 2025Add auto-generated Python API reference documentation from software-agent-sdk — OpenHands/docs1 reaction · 5 comments
- Nov 11, 2026docs(sdk): explain native tool schema constraints — OpenHands/docs0 reactions · 1 comment
- Nov 6, 2026docs(acp): describe explicit permission and session-scope controls — OpenHands/docs0 reactions · 1 comment
- Nov 6, 2026docs: explain remote conversation WebSocket message limits — OpenHands/docs0 reactions · 4 comments
- Nov 5, 2026docs: explain trusted saved-profile agent selection — OpenHands/docs0 reactions · 1 comment
- Oct 30, 2026docs(sdk): explain automatic conversation title prompts — OpenHands/docs0 reactions · 1 comment
- Oct 29, 2026docs(sdk): explain conversation affinity through OpenRouter proxies — OpenHands/docs0 reactions · 1 comment
- Oct 27, 2026docs: explain validation-loop detection and repair behavior — OpenHands/docs0 reactions · 1 comment
- Oct 27, 2026docs(sdk): explain provider retry delays and wait limits — OpenHands/docs0 reactions · 1 comment
- Oct 26, 2026docs(sdk): clarify comparison of invalid tool calls — OpenHands/docs0 reactions · 1 comment
- Oct 26, 2026docs(sdk): explain the independent error-retry threshold — OpenHands/docs0 reactions · 1 comment
- Oct 26, 2026docs(sdk): clarify stuck-detector action comparison — OpenHands/docs0 reactions · 1 comment
- Oct 23, 2026docs: document child waits, typed mailbox delivery and SDK tools — OpenHands/docs0 reactions · 1 comment
- Oct 19, 2026Document SDK capability resolution and image-delivery contracts — OpenHands/docs0 reactions
- Oct 19, 2026build(deps-dev): update openhands-sdk requirement from ~=1.49.0 to >=1.49,<1.56 in the openhands group across 1 directory — jpshackelford/oh-gemini-tool0 reactions · 1 comment
- Oct 18, 2026[OSS→OHE parity] Atomic secret/MCP consumers and truthful ownership/probe states — OpenHands/OpenHands0 reactions
- Oct 18, 2026[OSS→OHE parity] Hosted routing, launch override and profile-auth capability consumers — OpenHands/OpenHands0 reactions
- Oct 18, 2026[OSS→OHE parity] [Design/implementation] Durable tenant installed-plugin lifecycle — OpenHands/enterprise0 reactions · 3 comments
- Oct 18, 2026[OSS→OHE parity] Atomic hosted MCP mutations and truthful sandbox probe capabilities — OpenHands/enterprise0 reactions · 1 comment
- Oct 18, 2026[OSS→OHE parity] Atomic owned-secret mutation, overwrite consent and edit capabilities — OpenHands/enterprise0 reactions
- Oct 18, 2026[OSS→OHE parity] [Design/validation] Hosted subscription-auth ownership and lifecycle — OpenHands/enterprise0 reactions · 3 comments
- Oct 18, 2026[OSS→OHE parity] Bounded hosted pre-save LLM connection validation — OpenHands/enterprise0 reactions
- Oct 18, 2026[OSS→OHE parity] Preserve file bytes, MIME and errors through tenant file transport — OpenHands/enterprise0 reactions
- Oct 18, 2026[OSS→OHE parity] Managed event-index conversation forks through Enterprise lifecycle — OpenHands/enterprise0 reactions
- Oct 18, 2026docs(sdk): explain per-run conversation cost limits — OpenHands/docs0 reactions · 1 comment
- Oct 16, 2026docs(sdk): explain capability resolution and image delivery contracts — OpenHands/docs0 reactions · 2 comments
- Oct 16, 2026docs(sdk): document Jev threat classification and confirmation — OpenHands/docs0 reactions
- Oct 15, 2026Fix OpenHands SDK installation and verify framework imports — LLoliwe/agent-spend-proof0 reactions
- Oct 14, 2026Document runtime ACP permission handlers — OpenHands/docs0 reactions
- Oct 11, 2026docs(sdk): clarify system prompt reuse on conversation restore — OpenHands/docs0 reactions
- Oct 10, 2026docs: document skipped agent definitions — OpenHands/docs0 reactions
- Oct 9, 2026docs(harness-principles): add 2 principles from OpenHands SDK — markhazlett/agent-harness0 reactions
- Oct 6, 2026Document history retrieval and model-directed context resets — OpenHands/docs0 reactions
On Hacker News · 1
- Nov 19, 2025Show HN: OpenHands Software Agent SDK2 points
In this repository8 other files it gives its agents.
- AGENTS.mdAGENTS.md
- openhands-agent-server/AGENTS.mdAGENTS.md
- cross-repo-testingSkill
- debug-test-examples-workflowSkill
- feature-release-rolloutSkill
- manage-evalsSkill
- pr-design-docSkill
- sdk-releaseSkill
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 1 is distinctive, and 7 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
7
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 40
Put it on PATH and read its help before anything else:
- Very specific: line 61
If an agent-facing path cannot be driven with the CLI, that is a harness gap: extend scripts/control_agent_server.py (document the verb in …
- Very specific: line 64
Never work around a gap with an untracked one-off script the next agent cannot rerun.
- Very specific: line 169
Use deepseek-flash (DeepSeek V4.1 Flash) for everything that needs a model.
- Very specific: line 207
Create or extend the map: follow references/mapping.md.
- Very specific: line 210
Re-prove the map: every family's recipes are fenced sh blocks whose commands assert their own expectations, so control-agent-server map run …
- Very specific: line 218
Run the daily pass: follow references/daily.md: static checks (map check, map coverage), the families changed since the Maintenance …
- Distinctive: line 227
Maintain the map (periodic or weekly): follow references/maintenance.md: a frozen BASE/TARGET, a route diff, a PR-intent ledger, a source …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Verify the Agent Server through its real API
Prove what an API consumer observes, not that a handler returned or CI passed.
The consumers are agents and the programs that carry them: the SDK's
RemoteConversation/RemoteWorkspace, the TypeScript client, Agent Canvas and
automations. Three parts work together:
control-agent-server(scripts/) is the lever. It launches this checkout as an isolated real server (privateHOME, persistence dir, config file, session key, secret key, port), and turns every API step into a command you can rerun: REST calls, WebSocket captures, LLM-backed conversations, fixtures, SDK programs.- The feature map (references/feature-map/)
lists every agent-facing behavior with stable IDs, the routes each family
owns, entry points (REST, WebSocket, SDK and TypeScript client methods),
exact
control-agent-serverrecipes, observable results and gotchas. - Evidence: saved request/response exchanges, WebSocket frames, webhook deliveries and program transcripts, plus a pass/fail/blocked/not-run ledger that survives cleanup.
Neither this skill nor the map authorizes external writes, paid models beyond the budget you were given, or product fixes the user did not ask for.
The CLI comes first
Put it on PATH and read its help before anything else:
export PATH="$PWD/.agents/skills/verify-agent-server/scripts:$PATH"
control-agent-server --help # then: control-agent-server <command> --helpRequirements: make build (or uv sync --dev) in the repository root; the
CLI runs on the repository's .venv and needs nothing else. git is needed
for fixtures, tmux for bash and terminal tools, and a DeepSeek key for
model-backed recipes. Every command prints one JSON object; exit 0 ok,
1 action failed, 2 usage, 3 environment.
In a container whose PID 1 does not reap children, run the entire
map run --fresh or test command under a subreaper such as tini -s --
(Docker's docker-init -s -- is also suitable). For manual runs, wrap the
shell containing launch, drive and stop; wrapping launch alone exits the
supervisor too early. Otherwise exited descendants can remain zombies, causing cleanup
checks to mistake them for live work. Keep the original failure evidence and
report this environment difference; do not weaken the cleanup assertions.
If an agent-facing path cannot be driven with the CLI, that is a harness
gap: extend scripts/control_agent_server.py (document the verb in its
--help with examples and in this file), prove it live, then write the recipe.
Never work around a gap with an untracked one-off script the next agent cannot
rerun. api and ws reach every route, so most gaps are about waiting,
fixtures or observing side effects, not about calling an endpoint. Other agents
may be running the same script: edit a copy, run its tests and
map check, then move it into place in one step.
Launch → doctor → drive → evidence → cleanup
export AGENT_SERVER_VERIFY_RUN=$(control-agent-server launch --new --print-run)
control-agent-server doctor # read-only; must be ok before driving
control-agent-server llm preset deepseek # deepseek-flash (active) + deepseek-pro from $DEEPSEEK_API_KEY
CID=$(control-agent-server conversation start --prompt 'Create hello.txt containing hi' --wait --print-id)
control-agent-server conversation events "$CID" --kinds ObservationEvent --contains hello.txt
control-agent-server api GET "/api/conversations/$CID" --expect 200 --save F03.get/after-run
control-agent-server evidence add --feature F03.get --result pass --entry "GET /api/conversations/{id}" \
--expected "execution_status finished" --actual "finished" --artifact evidence/F03.get/after-run.json
control-agent-server stop # stops only this run; evidence stays- Which run. Commands use
--run DIR(anywhere on the line), else$AGENT_SERVER_VERIFY_RUN, else the only live run. With several live runs they refuse to guess, so exportAGENT_SERVER_VERIFY_RUNin every shell when other agents share the machine: an agent that drives someone else's run corrupts both evidence ledgers. Neverpkill -forkillallby pattern;control-agent-server stopends only your run's process groups. - Launch runs
python -m openhands.agent_serverfrom the checkout in its own process group, with cwd,HOME,OH_PERSISTENCE_DIR, the config file (OPENHANDS_AGENT_SERVER_CONFIG_PATH), tmux dir, worktree root, session key (OH_SESSION_API_KEYS_0) and secret key (OH_SECRET_KEY) all private to the run. The caller'sOH_*, LLM and cloud variables are not inherited; pass what a recipe needs with--env K=Vor--pass-env NAME. Useful variants:--no-auth(unauthenticated mode),--deferred-init(dormant untilPOST /api/init),--webhook-sink(a recording webhook receiver wired into the config),--config-json '{...}'(any other config field),--vscode,--preload-tools, and--checkout PATH(another worktree, for a baseline). Nested launches in a map recipe inherit the selected checkout; an explicit--checkoutoverrides it.attach --url ... --key-env VARdrives a server you did not launch (a Docker image, the PyInstaller binary, a remote sandbox);stopnever kills it. Launch refuses to start with less than about 1 GB free: a run with a model conversation holds 300 to 600 MB. - Doctor checks the process group, that this run owns its port,
/alive,/health,/ready,/server_info(version, and thatbuild_git_shais the checkout's HEAD), that a missing key is rejected (401) and the run's key accepted, that a rotated-out key is rejected, the OpenAPI document, WebSocket first-frame auth (bad key closes with 4001), and tracebacks in the log. Run it first, after every surprising failure, and before blaming the product. - Drive with
api METHOD PATH(any REST route,OPTIONSandHEADincluded;--auth none|bad|previous|bearer|bearer-bad|initfor auth cases,--expectfor the status,--check FIELD OP VALUEand--check-headerfor the body and headers,--fieldto chain values into shell variables,--jarfor cookies,--until-okto poll,--save FEATURE/NAMEfor evidence),ws listen|start|read|stop(any WebSocket;--auth first-frame|query|header|none|bad|bad-query|bad-header|previous|key,--expect-rejectfor close codes, replay with--query resend_mode=all, background captures that span other commands,--waitonread|stopto poll until the expectations hold),conversation start|wait|send|events(the essential agent pathways),sink read(what a webhook, telemetry or credential sink received),state ls|cat|grep(what the server persisted, and that secrets are not in it),configandcapabilities, andexec -- <program>(an SDK or TypeScript-client program withAGENT_SERVER_URLandSESSION_API_KEYinjected).logs --grepreads the server log with secrets redacted. - Arrange, don't fake.
llm,fixtureand setup calls throughapiexist to reach preconditions (a configured profile, a git repository, a skill, a stdio MCP server, an HTTP sink). They never count as proof of the feature that consumes them; the map says which steps are proof. Never patch the server or mock a server response to make a live check pass. The one stand-in isfixture llm-stub: an OpenAI-compatible provider on loopback that replays scripted steps (reply,tool,status,hang) through the server's real LLM client. Use it only for provider behavior a real model cannot produce on demand (a 401, a 500, a hang, an exact tool call such asfinishor a risky command); every happy path runs on a real model. A status step can include a provider error body, for example--step 'status:429:{"error":{"message":"Quota exceeded","type":"insufficient_quota","code":"insufficient_quota"}}', so quota exhaustion and transient rate limiting can be distinguished. A recipe-local process that later bullets use (an HTTP fixture server, a proxy) runs underhelper start --name NAME -- CMD, notCMD &:map runends each bullet's process group when the bullet ends. - State control reaches states a happy path never shows, without mocks:
restart(same state after a restart: persistence, restore, reconnection),restart --hard(SIGKILL: crash recovery),restart --rotate-key(stale session key),launch --share-conversations-from RUN(a second instance on the same conversations: leases and takeover),restart --env K=Vor--config-json(config changes such asmax_concurrent_runsor idle TTL),launch --deferred-init,launch --no-auth,conversation start --no-run, andws startbefore an action so its events are captured live. - Evidence goes under
<run>/evidence/<feature-id>/and the append-only ledger<run>/evidence/ledger.jsonl. Nothing is overwritten: a repeated name is saved as<name>-2.json, so cite the path the command prints;evidence reportrenders the report contract. The CLI redacts the run's keys,$DEEPSEEK_API_KEYand cipher-encrypted values from output and evidence, including--fieldand--print-header. Onlyapi --unsafe-unredactedwith either flag returns raw secret values for a round trip; capture those directly in variables and keep them out of reports. Evidence remains redacted. Keys, logs and state stay in<run>/private/. Evidence is not automatically public: read it before publishing it. - Cleanup with
control-agent-server stop(add--purge-privateto delete keys, state and fixtures once the evidence is checked). It signals only the process groups this run started (server, HTTP sinks, background captures, helpers) and verifies the port closed.
LLM budget
Use deepseek-flash (DeepSeek V4.1 Flash) for everything that needs a model;
switch to deepseek-pro only for checks that need a second profile or a
stronger model. Keep prompts tiny, deterministic and confined to the
conversation's workspace ("Create hello.txt containing exactly: hi"), and
assert on tool observations and files, never on the model's wording. Stop or
delete conversations a recipe no longer needs. Without a key, run every
credential-free recipe and record model-dependent ones as blocked with the
missing prerequisite; never substitute a mock and call it a pass.
For critic checks, use an available, authorized OpenHands (openhands/…) or
compatible OpenHands LLM-proxy profile, regardless of the main profile used
for the pass. The SDK reuses its LLM key when verification.critic_api_key
is unset. Restore the previous profile afterwards. Do not declare the critic
blocked merely because the main model uses another provider. This does not
authorize new credentials, additional budget, or sending private test data
to another service; require real scored events before recording a pass.
Choose the job
- Verify a change or a PR: map the changed paths to feature IDs with
control-agent-server map owners --changed origin/main...HEAD, drive every entry point those features list (REST, WebSocket, SDK, TypeScript client), and report with the report contract. A behavior change needs a regression test as well (usually intests/cross/test_remote_conversation_live_server.py); the recipe is the live proof, not a substitute for that test. - Verify an existing fix (an open PR or a merged commit that claims to fix
a
known bugbullet): reproduce on the baseline twice and the fix twice, with the same flags and fixtures:control-agent-server map run --file Fnn --only Fnn.x --fresh --repeat 2 --checkout <base worktree>must report the bulletxfailin both attempts, and the same command without--checkout(the fix)xpassin both.xfailmeans the marked command failed with the failure its# bug: PREDICATEdocuments (see the marker contract); any other failure on the baseline is afail, so the proof cannot rest on an unrelated error. Then drop the marker in the same PR. Baseline not reproducing twice means "inconclusive", never "fixed".--onlyrejects unknown IDs and also runs each selected recipe's transitiveRequires:prerequisites in family order. - Create or extend the map: follow references/mapping.md.
control-agent-server map coveragemeasures which routes no family owns or drives. - Re-prove the map: every family's recipes are fenced
shblocks whose commands assert their own expectations, socontrol-agent-server map run --file Fnn --fresh --recordreplays a family on its own fresh run (with the family'sLaunch:flags), requires a passingdoctorbefore the first recipe, and writes ledger rows, andcontrol-agent-server map run --all --fresh --recordreplays the whole map. Only failing bullets need judgment;known bugbullets report as expected failures until the bug is fixed. - Run the daily pass: follow references/daily.md:
static checks (
map check,map coverage), the families changed since theMaintenance baselineline of the map index (map baseline;map ownerswith no arguments diffs from it), and nothing else: families nothing in the range touched wait for the weekly pass, and a pass with no new commits onmainstops early. A pass that finished its changed families proposesTARGETas the next baseline withmap baseline --set "$TARGET"in its PR; compare ledgers withevidence report --baselinewhen yesterday's is at hand. - Maintain the map (periodic or weekly): follow references/maintenance.md: a frozen BASE/TARGET, a route diff, a PR-intent ledger, a source wave, one live pass over every family, and at most one PR of proven corrections.
Triage what you find
- Map drift: the map describes something the server no longer does by design. Fix the entry with current source and live evidence.
- Harness gap: the server works but the CLI cannot drive or observe it. Fix the CLI, re-drive.
- Product bug: the server is broken. Keep the evidence, search existing
issues, and report it separately: Agent Server, SDK and the TypeScript client
here (
OpenHands/agent-sdk), Agent Canvas UI inOpenHands/OpenHands, scheduling and dispatch inOpenHands/automation. Never rewrite an expected result to bless broken behavior. - Blocked: name the missing prerequisite (key, account, Docker, binary, OS)
and the route you attempted. Unreachable is never a pass.
control-agent-server capabilitiessays what this machine can drive and which families'Needs:are unmet. A family'sNeeds:line gates the whole family under--all; a recipe's ownNeeds:line blocks only that recipe and the ones that require it, so the rest of the family still runs on a machine without it.
How this fits with the tests
Unit tests (tests/agent_server/), the in-process live-server tests
(tests/cross/test_remote_conversation_live_server.py), the stress suite, the
REST breakage check (oasdiff against the last release) and the TypeScript
client's endpoint audit prove contracts in isolation, with scripted LLMs. This
skill proves the assembled server as agents use it: one real process, real
persistence and restarts, real sockets and webhooks, and a real model. It does
not replace any of them.
See references/adaptation.md for where these ideas come from and what was deliberately left out.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

