Benchmark Video Search

Instructions

Follow the routing table, then the numbered steps. Execute each step in order on a first run; for a repeat run against an already-indexed deployment run Step 3, then go to Step 5. Reference material is in references/, the runner is in scripts/.

Purpose

Answers two questions about a search profile deployment: does retrieval find the right footage, and where does a query's time go. It drives the vss CLI — the same path the product uses, since the agent adapter shells out to vss search run <mode> --raw for every search — so the numbers describe what ships rather than a REST endpoint that is being retired.

Searches go through vss. To know when ingestion is done and when a wipe has finished, the runner also reads Elasticsearch and RT-CV's stream list directly, read-only, through the deployment's :7777 ingress (derived from --endpoint). It never writes to either. It changes deployment state only through the agent API: it deletes file sources and uploads videos.

When not to use this skill

• For a single video-search question, use vss-search-archive.
• For an end-to-end benchmark of the OpenClaw chat route, this runner does not measure that path; do not present CLI timings as chat timings.
• For a historical benchmark of the retired agent REST search endpoint, use the external legacy run_eval.py noted in references/troubleshooting.md.

Routing

The default run

Unless the user says otherwise, this is the run. Ask which dataset to use and, if missing, which deployment endpoint to target.

uv run --with requests python3 scripts/run_eval_flows.py --endpoint http://HOST:8000 \
    --data-dir /path/to/datasets --dataset DATASET \
    --skip-download --name RUN_NAME

Five defaults, each load-bearing:

• Every run starts from an empty index. Before uploading, the runner deletes every uploaded file source in VST, waits for the cleanup webhooks to empty the indexes a video_file search reads, then ingests the whole dataset. Anything left indexed would be searched too, and every hit on it would score as a false positive. RTSP cameras are never deleted; the run lists the ones it kept. Report what was deleted (flow.cleared). The deployment may be shared, but nobody else may upload to it while an eval runs. For what is checked and the aborts, see "Before ingest: the wipe" in references/troubleshooting.md.

  WARNING: The wipe is irreversible. Deleting a source through the agent also deletes its VST storage, including other users' uploaded files on a shared deployment. There is no prompt. Confirm the target host with the user before a run that is not --skip-ingest. The runner aborts if --vst-url, --ingress-url and --endpoint are on different hosts; that catches a host mismatch, not two deployments on one host.

  --skip-ingest is the only way to keep what is there. It deletes nothing and uploads nothing, and it aborts unless every dataset video is already registered. Passing a removed cleanup flag (--only-dataset, --clear, --confirm-delete, --skip-existing) is a usage error (exit 2).

• Live decomposition is on by default. The LLM origin is derived from --endpoint (same host, port 30081), so there is no flag to forget. Every query is decomposed the way the deployed agent decomposes it, and routed the way the agent routes it (references/dataset-format.md). Override with --llm-url / --llm-port; turn it off with --no-decompose (replays stored routes) or --fixed-search-path (ignores stored routes for a single-path baseline).

  If the NIM is unreachable the run replays dataset routes when present; otherwise it falls back to embed, which needs nothing but the query text. The fallback records flow.query.live_decomposition.fell_back_to in the result file. Check that field and Search paths: before quoting numbers.

  If decomposition fails for one query mid-run, that query is searched on embed with its original text, as the agent does. It is not marked unanswered. The result records decompose_failed on that query and summary.decomposition.failed, and the summary prints N queries fell back to embed: decomposition failed. More than 20% of queries falling back aborts the run (--max-decompose-failed-pct): the numbers would measure embed, not routing. The run stops as soon as failures pass the cap across the whole dataset, and still writes the result file — with aborted set, each query's decompose_failed error, and no summary. Report the count next to any number you quote.

  --no-decompose also replays dataset routes when present; it does not force one path. Use --fixed-search-path only for a deliberately single-path baseline. Nothing else is inferred: a devset_provenance.json sidecar is never loaded automatically. To use it as an answer key, pass its full path with --decompositions; scoring against perfect routing is a different experiment from scoring the live decomposer.

• The pre-decomposition question is sent with every query as --original-query. Retrieval ignores it; the critic verifies against it instead of a question the library rebuilds out of --query and --attribute. Without it the CLI's critic is asked a different question from the REST flow's, so the two flows' rejection rates are not comparable — and on the object path there is no question at all, so verification is skipped outright. Turn it off only with --no-original-query, and only to reproduce a result file captured before this existed.

  A vss older than the flag is detected by one --help probe at startup and the run continues without it, with a warning. Do not quote critic-filtered metrics from a run that printed that warning.

• Ingest is vst-direct: upload to VIOS, let its webhooks drive perception. That is what the UI does — it stopped calling the agent's ingest API — and it is the only flow that also triggers RT-VLM tagging. The timestamp anchor rides in the upload metadata, not in anything /complete does, so scores stay comparable with agent-3step baselines.

  What agent-3step gave for free was proof: /complete returns chunks_processed, and a zero failed the upload. vst-direct has no such step, so an ingest gate replaces it: the runner reads Elasticsearch and RT-CV through the :7777 ingress until every video is complete, then queries. Embeddings and tags are checked against chunk targets from the video's duration. Raw frames and behavior are done once RT-CV has released the stream and the counts stop changing. RT-CV writes raw only for frames with detections, so sparse footage legitimately ends far below its frame count. If any video is incomplete at the deadline it aborts with each index's count against its target. The one exception is an index no video ever gets: after --ingest-absent-grace-s (default 60 s) tags, or raw with behavior, are treated as not produced by this deployment, and the run confirms as degraded and queries without them (--ingest-strict aborts instead). One searchable hit arrives ~20 s after upload; RT-CV is still writing for minutes after that.

  If the gate reports RT-CV never listed the stream, check the deployed VIOS notification config and that RTVI_EMBED_MODEL matches the webhook's model string — RT-Embed answers a mismatch with HTTP 200 and inference: false. The Docker and Helm search profiles enable webhooks; generic VIOS chart defaults may not. If the deployed webhooks are disabled, use --ingest-flow agent-3step. For a profile without RT-VLM or RT-CV, narrow --ingest-require. Never pass --skip-index-probe or --legacy-index-probe on a run whose numbers you intend to quote.

• Concurrency stays at 1 (the script's default). Concurrent queries contend for the same VLM and embedding services, so per-stage latencies inflate and stop describing a single query.

Deviate only on request: --skip-ingest to reuse what is there, --subset to narrow the slice, --concurrency N to trade latency fidelity for wall-clock, --no-decompose to replay stored routes, or --fixed-search-path for a single-path baseline.

Never add either decomposition override to a run the user called a live-routing eval. Both measure a different flow; the result records the routing mode.

Prerequisites

Agent /health confirms the process is reachable, not that VIOS and the search indices are ready. Step 3 and the ingest gate are the retrieval readiness gates.

Step 1 — Configure the CLI

The CLI reads ~/.vss/config.json, which is separate state from --endpoint and points at a different port: it discovers services by path prefix on the unified origin, which the agent port does not route. Getting this wrong makes every query exit 4.

vss configure --base-url http://HOST:7777
vss configure show

localhost is valid here when the runner runs on the deployment host. This URL controls where the CLI reaches the services; it does not determine the clip URL sent to RT-VLM. If verification fails, inspect the VST-returned videoUrl and test that exact address from the RT-VLM container.

Step 2 — Dry run

Reports what would happen and contacts nothing.

uv run --with requests python3 scripts/run_eval_flows.py --endpoint http://HOST:8000 \
    --data-dir /path/to/datasets --dataset DATASET --dry-run

Step 3 — Verify the deployment is actually indexed

Registration with VST is not ingestion. A deployment can list every sensor and still have an empty Elasticsearch, in which case every query returns zero hits and every metric reads 0.0000 — which looks like catastrophic retrieval quality and is not.

uv run --with requests python3 scripts/run_eval_flows.py --endpoint http://HOST:8000 --check-indices

It counts the documents in each searched index through the same ingress the wipe and the ingest gate read: the --endpoint host on port 7777 (--vss-origin-port), or --ingress-url when given. It is read-only, contacts nothing else, and exits 1 when the embed index is empty or unreachable.

Expect mdx-embed-filtered-*, mdx-behavior-* and mdx-raw-* with non-zero counts. The embed index must have documents: embed is the default --search-path and the path the post-ingest index probe searches. mdx-behavior-* is what the attribute and fusion paths read; if it is empty, only embed can score, and the runner warns. If the embed index is empty, go to Step 4.

Step 4 — Ingest

The default vst-direct flow uploads to VIOS and lets its webhooks trigger perception. It does not call the agent's /complete endpoint or receive a chunks_processed count. The runner waits for VST registration, then waits on the ingest gate until every index is complete. Use --ingest-flow agent-3step only when the deployment cannot run the webhook flow; see references/troubleshooting.md.

uv run --with requests python3 scripts/run_eval_flows.py --endpoint http://HOST:8000 \
    --data-dir /path/to/datasets --dataset DATASET --skip-download

Expect this to be slow while the webhook pipeline indexes the videos:

• A registered source is not necessarily indexed. The ingest gate must confirm every video before scoring; a 210 s video takes about 220 s. It prints an INGESTION SUMMARY with embedding-done and full-ingestion times per video.
• The wipe runs first and prints what it found and deleted (Not clean: ..., then Index empty after Ns). A clean index prints Index is clean.
• Documents that reappear under a name about to be uploaded abort the run before upload (stale documents for X): something wrote them after the wipe, and they would satisfy the gate instantly.
• If webhooks are disabled, select --ingest-flow agent-3step explicitly. That flow calls /complete, which can return a transient 502 and is retried.

Then re-run Step 3. Indices are lazy; they appear after webhook processing, not immediately after upload. See references/flags.md for ingest options.

Step 5 — Run the benchmark

uv run --with requests python3 scripts/run_eval_flows.py --endpoint http://HOST:8000 \
    --data-dir /path/to/datasets --dataset DATASET --subset SUBSET \
    --skip-download --skip-ingest --name RUN_NAME

--skip-ingest reuses an already-indexed dataset without wiping it. It aborts if a dataset video is not registered, and warns about sources outside the dataset. See references/flags.md before changing flags that affect run meaning.

Decomposition needs no flag. An LLM turns the sentence into {query, attributes, has_action, ...} and that decides which path runs — the same call the deployed agent makes. The script derives the NIM origin from --endpoint; if it cannot reach it the run falls back rather than stopping, and says which fallback it took.

fusion cannot be a fallback. It needs an attribute per query and there is nothing to supply one without decomposition — and feeding it the whole query as its attribute is the failure this eval already measured: mAP −25%, HIT@1 halved, latency +59%.

Leave --concurrency alone unless asked. It defaults to 1 because concurrent queries contend for the same VLM and embedding services, which inflates every per-stage latency in the report.

--name makes the result file findable; without it the name is <dataset>_<subset>_<timestamp>.

Step 6 — Report

Results land in scripts/cli_eval_result/<name>.json.

Report Recall and HIT@k as the quality signal. Precision and mAP are dominated by --top-k — five results against roughly one relevant segment keeps them low however good retrieval is — so they compare runs; they do not grade a deployment.

State these alongside any number, because each one changes what it means:

• Which paths ran. Search paths: in the summary. A run that was all embed says nothing about attribute.
• Whether decomposition was live. If so, give its share of query time, what it Routed to: — a routing shift changes what was measured — and how many queries fell back to embed because it failed.
• Whether the critic ran. Critic-filtered metrics print NA when the response carried no verification block. NA is not 0.
• Whether ingest was degraded. summary.ingest_degraded lists the indexes the deployment never produced and the executed paths that read them. Scores on those paths measure a missing index, not retrieval; do not quote them.

Read references/reading-results.md when interpreting stage latencies or metrics. Read references/flags.md when a flag might change comparability. Read references/troubleshooting.md when ingest fails or the CLI exits nonzero. Read references/dataset-format.md when validating a dataset's layout.

Error handling

A non-zero CLI exit aborts the run. A missing result envelope marks that query unanswered and aborts by default; --tolerate-unanswered permits an explicit partial run that excludes those queries from scoring. Never count them as misses. Diagnose an all-zero run with Step 3 before reporting quality. Critic-filtered NA means verification was not available, not zero quality. Use references/troubleshooting.md for the specific failure and recovery steps.

Conventions

• Never report 0.0000 as a quality result without checking Step 3 first. An unindexed deployment and a broken retriever look identical in the metrics and are not the same finding.
• A non-zero CLI exit aborts the run rather than scoring 0.0. A stale virtualenv exits 1 on every invocation, and scoring that as "no results" would report a broken environment as an accuracy regression.
• Queries merge adjacent windows by default, as the agent does. Merging is also what makes the search fetch the agent's doubled candidate pool; unmerged runs search a smaller pool and score lower recall. Result files captured before this default were unmerged. Their precision and mAP are not comparable with a merged run, because merged windows span more segments. Check config.merge_adjacent before comparing two runs.
• The critic is an LLM and is not deterministic. Raw metrics repeat exactly; critic-filtered ones move a few points between runs. Gate on raw.
• Live decomposition is not deterministic either. The same query can route differently between runs, so compare Routed to: before comparing scores.