agentleFS
OpenHandsOpenHands/software-agent-sdk/.agents/skills/verify-agent-server/SKILL.md

verify-agent-server

Skill1.4k starsChanged 8 days agoMITView on GitHub

This skill should be used to "verify Agent Server APIs", "test the REST or WebSocket API like an agent would", "drive the agent-server", "check an API change against a real server", "create or update the agent-server feature map", or "run the weekly API audit". Ships control-agent-server (launch, doctor, api, ws, llm, conversations, fixtures, exec, assert, evidence, map) and the maintained map of every agent-facing feature.

  • Reads credentials

Trending

More talk lately.about the whole repository

Dec 2025: 3 mentionsJan 2026: 2 mentionsFeb 2026: 2 mentionsMar 2026: 4 mentionsApr 2026: 1 mentionMay 2026: 0 mentionsJun 2026: 1 mentionJul 2026: 1 mentionAug 2026: 0 mentionsSep 2026: 1 mentionOct 2026: 28 mentionsNov 2026: 6 mentions
Dec 2025Jun 2026Nov 2026
Mentions a month of the OpenHands/software-agent-sdk repository that our weekly crawl found. There is no adoption data for this file yet.

Who's talking about OpenHands/software-agent-sdk

The biggest 5 of 51 mentions linking to the repository that our crawl found. Each line is their own headline.

See all 51

Talked about on GitHub and Hacker News (51 times).

On GitHub · 50

45 more on GitHub

On Hacker News · 1

The file

We split this skill into the rules it gives an agent

8 rules. About 0 are common, 1 is distinctive, and 7 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 7

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Put it on PATH and read its help before anything else:

    line 40
  2. Very specific:

    If an agent-facing path cannot be driven with the CLI, that is a harness gap: extend scripts/control_agent_server.py (document the verb in …

    line 61
  3. Very specific:

    Never work around a gap with an untracked one-off script the next agent cannot rerun.

    line 64
  4. Very specific:

    Use deepseek-flash (DeepSeek V4.1 Flash) for everything that needs a model.

    line 169
  5. Very specific:

    Create or extend the map: follow references/mapping.md.

    line 207
  6. Very specific:

    Re-prove the map: every family's recipes are fenced sh blocks whose commands assert their own expectations, so control-agent-server map run …

    line 210
  7. Very specific:

    Run the daily pass: follow references/daily.md: static checks (map check, map coverage), the families changed since the Maintenance …

    line 218
  8. Distinctive:

    Maintain the map (periodic or weekly): follow references/maintenance.md: a frozen BASE/TARGET, a route diff, a PR-intent ledger, a source …

    line 227

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Verify the Agent Server through its real API

Prove what an API consumer observes, not that a handler returned or CI passed. The consumers are agents and the programs that carry them: the SDK's RemoteConversation/RemoteWorkspace, the TypeScript client, Agent Canvas and automations. Three parts work together:

  1. control-agent-server (scripts/) is the lever. It launches this checkout as an isolated real server (private HOME, persistence dir, config file, session key, secret key, port), and turns every API step into a command you can rerun: REST calls, WebSocket captures, LLM-backed conversations, fixtures, SDK programs.
  2. The feature map (references/feature-map/) lists every agent-facing behavior with stable IDs, the routes each family owns, entry points (REST, WebSocket, SDK and TypeScript client methods), exact control-agent-server recipes, observable results and gotchas.
  3. Evidence: saved request/response exchanges, WebSocket frames, webhook deliveries and program transcripts, plus a pass/fail/blocked/not-run ledger that survives cleanup.

Neither this skill nor the map authorizes external writes, paid models beyond the budget you were given, or product fixes the user did not ask for.

The CLI comes first

Put it on PATH and read its help before anything else:

export PATH="$PWD/.agents/skills/verify-agent-server/scripts:$PATH"
control-agent-server --help              # then: control-agent-server <command> --help

Requirements: make build (or uv sync --dev) in the repository root; the CLI runs on the repository's .venv and needs nothing else. git is needed for fixtures, tmux for bash and terminal tools, and a DeepSeek key for model-backed recipes. Every command prints one JSON object; exit 0 ok, 1 action failed, 2 usage, 3 environment.

In a container whose PID 1 does not reap children, run the entire map run --fresh or test command under a subreaper such as tini -s -- (Docker's docker-init -s -- is also suitable). For manual runs, wrap the shell containing launch, drive and stop; wrapping launch alone exits the supervisor too early. Otherwise exited descendants can remain zombies, causing cleanup checks to mistake them for live work. Keep the original failure evidence and report this environment difference; do not weaken the cleanup assertions.

If an agent-facing path cannot be driven with the CLI, that is a harness gap: extend scripts/control_agent_server.py (document the verb in its --help with examples and in this file), prove it live, then write the recipe. Never work around a gap with an untracked one-off script the next agent cannot rerun. api and ws reach every route, so most gaps are about waiting, fixtures or observing side effects, not about calling an endpoint. Other agents may be running the same script: edit a copy, run its tests and map check, then move it into place in one step.

Launch → doctor → drive → evidence → cleanup

export AGENT_SERVER_VERIFY_RUN=$(control-agent-server launch --new --print-run)
control-agent-server doctor                     # read-only; must be ok before driving
control-agent-server llm preset deepseek        # deepseek-flash (active) + deepseek-pro from $DEEPSEEK_API_KEY
CID=$(control-agent-server conversation start --prompt 'Create hello.txt containing hi' --wait --print-id)
control-agent-server conversation events "$CID" --kinds ObservationEvent --contains hello.txt
control-agent-server api GET "/api/conversations/$CID" --expect 200 --save F03.get/after-run
control-agent-server evidence add --feature F03.get --result pass --entry "GET /api/conversations/{id}" \
  --expected "execution_status finished" --actual "finished" --artifact evidence/F03.get/after-run.json
control-agent-server stop                       # stops only this run; evidence stays
  • Which run. Commands use --run DIR (anywhere on the line), else $AGENT_SERVER_VERIFY_RUN, else the only live run. With several live runs they refuse to guess, so export AGENT_SERVER_VERIFY_RUN in every shell when other agents share the machine: an agent that drives someone else's run corrupts both evidence ledgers. Never pkill -f or killall by pattern; control-agent-server stop ends only your run's process groups.
  • Launch runs python -m openhands.agent_server from the checkout in its own process group, with cwd, HOME, OH_PERSISTENCE_DIR, the config file (OPENHANDS_AGENT_SERVER_CONFIG_PATH), tmux dir, worktree root, session key (OH_SESSION_API_KEYS_0) and secret key (OH_SECRET_KEY) all private to the run. The caller's OH_*, LLM and cloud variables are not inherited; pass what a recipe needs with --env K=V or --pass-env NAME. Useful variants: --no-auth (unauthenticated mode), --deferred-init (dormant until POST /api/init), --webhook-sink (a recording webhook receiver wired into the config), --config-json '{...}' (any other config field), --vscode, --preload-tools, and --checkout PATH (another worktree, for a baseline). Nested launches in a map recipe inherit the selected checkout; an explicit --checkout overrides it. attach --url ... --key-env VAR drives a server you did not launch (a Docker image, the PyInstaller binary, a remote sandbox); stop never kills it. Launch refuses to start with less than about 1 GB free: a run with a model conversation holds 300 to 600 MB.
  • Doctor checks the process group, that this run owns its port, /alive, /health, /ready, /server_info (version, and that build_git_sha is the checkout's HEAD), that a missing key is rejected (401) and the run's key accepted, that a rotated-out key is rejected, the OpenAPI document, WebSocket first-frame auth (bad key closes with 4001), and tracebacks in the log. Run it first, after every surprising failure, and before blaming the product.
  • Drive with api METHOD PATH (any REST route, OPTIONS and HEAD included; --auth none|bad|previous|bearer|bearer-bad|init for auth cases, --expect for the status, --check FIELD OP VALUE and --check-header for the body and headers, --field to chain values into shell variables, --jar for cookies, --until-ok to poll, --save FEATURE/NAME for evidence), ws listen|start|read|stop (any WebSocket; --auth first-frame|query|header|none|bad|bad-query|bad-header|previous|key, --expect-reject for close codes, replay with --query resend_mode=all, background captures that span other commands, --wait on read|stop to poll until the expectations hold), conversation start|wait|send|events (the essential agent pathways), sink read (what a webhook, telemetry or credential sink received), state ls|cat|grep (what the server persisted, and that secrets are not in it), config and capabilities, and exec -- <program> (an SDK or TypeScript-client program with AGENT_SERVER_URL and SESSION_API_KEY injected). logs --grep reads the server log with secrets redacted.
  • Arrange, don't fake. llm, fixture and setup calls through api exist to reach preconditions (a configured profile, a git repository, a skill, a stdio MCP server, an HTTP sink). They never count as proof of the feature that consumes them; the map says which steps are proof. Never patch the server or mock a server response to make a live check pass. The one stand-in is fixture llm-stub: an OpenAI-compatible provider on loopback that replays scripted steps (reply, tool, status, hang) through the server's real LLM client. Use it only for provider behavior a real model cannot produce on demand (a 401, a 500, a hang, an exact tool call such as finish or a risky command); every happy path runs on a real model. A status step can include a provider error body, for example --step 'status:429:{"error":{"message":"Quota exceeded","type":"insufficient_quota","code":"insufficient_quota"}}', so quota exhaustion and transient rate limiting can be distinguished. A recipe-local process that later bullets use (an HTTP fixture server, a proxy) runs under helper start --name NAME -- CMD, not CMD &: map run ends each bullet's process group when the bullet ends.
  • State control reaches states a happy path never shows, without mocks: restart (same state after a restart: persistence, restore, reconnection), restart --hard (SIGKILL: crash recovery), restart --rotate-key (stale session key), launch --share-conversations-from RUN (a second instance on the same conversations: leases and takeover), restart --env K=V or --config-json (config changes such as max_concurrent_runs or idle TTL), launch --deferred-init, launch --no-auth, conversation start --no-run, and ws start before an action so its events are captured live.
  • Evidence goes under <run>/evidence/<feature-id>/ and the append-only ledger <run>/evidence/ledger.jsonl. Nothing is overwritten: a repeated name is saved as <name>-2.json, so cite the path the command prints; evidence report renders the report contract. The CLI redacts the run's keys, $DEEPSEEK_API_KEY and cipher-encrypted values from output and evidence, including --field and --print-header. Only api --unsafe-unredacted with either flag returns raw secret values for a round trip; capture those directly in variables and keep them out of reports. Evidence remains redacted. Keys, logs and state stay in <run>/private/. Evidence is not automatically public: read it before publishing it.
  • Cleanup with control-agent-server stop (add --purge-private to delete keys, state and fixtures once the evidence is checked). It signals only the process groups this run started (server, HTTP sinks, background captures, helpers) and verifies the port closed.

LLM budget

Use deepseek-flash (DeepSeek V4.1 Flash) for everything that needs a model; switch to deepseek-pro only for checks that need a second profile or a stronger model. Keep prompts tiny, deterministic and confined to the conversation's workspace ("Create hello.txt containing exactly: hi"), and assert on tool observations and files, never on the model's wording. Stop or delete conversations a recipe no longer needs. Without a key, run every credential-free recipe and record model-dependent ones as blocked with the missing prerequisite; never substitute a mock and call it a pass.

For critic checks, use an available, authorized OpenHands (openhands/…) or compatible OpenHands LLM-proxy profile, regardless of the main profile used for the pass. The SDK reuses its LLM key when verification.critic_api_key is unset. Restore the previous profile afterwards. Do not declare the critic blocked merely because the main model uses another provider. This does not authorize new credentials, additional budget, or sending private test data to another service; require real scored events before recording a pass.

Choose the job

  • Verify a change or a PR: map the changed paths to feature IDs with control-agent-server map owners --changed origin/main...HEAD, drive every entry point those features list (REST, WebSocket, SDK, TypeScript client), and report with the report contract. A behavior change needs a regression test as well (usually in tests/cross/test_remote_conversation_live_server.py); the recipe is the live proof, not a substitute for that test.
  • Verify an existing fix (an open PR or a merged commit that claims to fix a known bug bullet): reproduce on the baseline twice and the fix twice, with the same flags and fixtures: control-agent-server map run --file Fnn --only Fnn.x --fresh --repeat 2 --checkout <base worktree> must report the bullet xfail in both attempts, and the same command without --checkout (the fix) xpass in both. xfail means the marked command failed with the failure its # bug: PREDICATE documents (see the marker contract); any other failure on the baseline is a fail, so the proof cannot rest on an unrelated error. Then drop the marker in the same PR. Baseline not reproducing twice means "inconclusive", never "fixed". --only rejects unknown IDs and also runs each selected recipe's transitive Requires: prerequisites in family order.
  • Create or extend the map: follow references/mapping.md. control-agent-server map coverage measures which routes no family owns or drives.
  • Re-prove the map: every family's recipes are fenced sh blocks whose commands assert their own expectations, so control-agent-server map run --file Fnn --fresh --record replays a family on its own fresh run (with the family's Launch: flags), requires a passing doctor before the first recipe, and writes ledger rows, and control-agent-server map run --all --fresh --record replays the whole map. Only failing bullets need judgment; known bug bullets report as expected failures until the bug is fixed.
  • Run the daily pass: follow references/daily.md: static checks (map check, map coverage), the families changed since the Maintenance baseline line of the map index (map baseline; map owners with no arguments diffs from it), and nothing else: families nothing in the range touched wait for the weekly pass, and a pass with no new commits on main stops early. A pass that finished its changed families proposes TARGET as the next baseline with map baseline --set "$TARGET" in its PR; compare ledgers with evidence report --baseline when yesterday's is at hand.
  • Maintain the map (periodic or weekly): follow references/maintenance.md: a frozen BASE/TARGET, a route diff, a PR-intent ledger, a source wave, one live pass over every family, and at most one PR of proven corrections.

Triage what you find

  • Map drift: the map describes something the server no longer does by design. Fix the entry with current source and live evidence.
  • Harness gap: the server works but the CLI cannot drive or observe it. Fix the CLI, re-drive.
  • Product bug: the server is broken. Keep the evidence, search existing issues, and report it separately: Agent Server, SDK and the TypeScript client here (OpenHands/agent-sdk), Agent Canvas UI in OpenHands/OpenHands, scheduling and dispatch in OpenHands/automation. Never rewrite an expected result to bless broken behavior.
  • Blocked: name the missing prerequisite (key, account, Docker, binary, OS) and the route you attempted. Unreachable is never a pass. control-agent-server capabilities says what this machine can drive and which families' Needs: are unmet. A family's Needs: line gates the whole family under --all; a recipe's own Needs: line blocks only that recipe and the ones that require it, so the rest of the family still runs on a machine without it.

How this fits with the tests

Unit tests (tests/agent_server/), the in-process live-server tests (tests/cross/test_remote_conversation_live_server.py), the stress suite, the REST breakage check (oasdiff against the last release) and the TypeScript client's endpoint audit prove contracts in isolation, with scripted LLMs. This skill proves the assembled server as agents use it: one real process, real persistence and restarts, real sockets and webhooks, and a real model. It does not replace any of them.

See references/adaptation.md for where these ideas come from and what was deliberately left out.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.