Run AgentX

Setup

Resolve the endpoint URL, served model name, matching tokenizer, and requested concurrency from the task. Reuse the model checkpoint. If serving is needed, follow deployment to download missing weights and launch it. Enable prefix caching, streaming usage, and cached-token reporting. For vLLM, use --enable-prefix-caching --enable-prompt-tokens-details. Verify the context limit fits the traces with the model's tokenizer; never silently truncate requests.

Start each benchmark campaign with the latest harness and compatible serving-stack releases unless the user requests specific versions. Record and pin the resolved harness and other code revisions, dataset/model/tokenizer revisions, serving framework (e.g. vLLM), and client/server dependencies or container digests in the run artifacts. Keep them fixed across sweep points, cache formats, and repeats unless they are the variable under test. Record local patches; treat upgrades as a new comparison baseline.

Install the latest AgentX harness in a separate client environment. Set AGENTX_PYTHON to a Python executable satisfying the latest release's Requires-Python metadata on PyPI. Installing upstream aiperf alone is insufficient.

set -euo pipefail
export AGENTX_WORKDIR="${AGENTX_WORKDIR:-$PWD/agentx-work}"
"${AGENTX_PYTHON:?}" -m venv "$AGENTX_WORKDIR/venv"
source "$AGENTX_WORKDIR/venv/bin/activate"
python -m pip install --upgrade agentx-harness

Set AGENTX_DATASET to an accepted with-subagents alias from the AgentX tutorial. Check the current tutorial for supported CLI options and scenario defaults. The public loader downloads and caches the traces and prompt reconstruction assets. An alias, even a dated one, does not select an immutable Hugging Face revision. Resolve the alias's repository to a commit SHA and record both with every run. Before benchmarking, verify that the loader passes that SHA as revision to Hugging Face load_dataset. Without a revision flag, set the selected Weka loader's hf_revision to that SHA in the campaign's client installation.

Prefer the deployed checkpoint's local tokenizer. If it requires custom code, add --tokenizer-trust-remote-code only after the user explicitly trusts its repository.

Choose concurrency

Sweep concurrency upward until KV-cache usage nears capacity, then refine around that point. Resweep when the model, hardware, or cache budget changes. Publish the full sweep and repeat promising points before claiming an advantage.

Run

In each shell, set AGENTX_WORKDIR to the setup directory and set AGENTX_DATASET, AGENTX_URL, AGENTX_MODEL, AGENTX_TOKENIZER, AGENTX_MAX_CONTEXT_LENGTH, and AGENTX_CONCURRENCY from the deployment and sweep. Use the server's base URL and actual context limit. Fix AGENTX_SEED across comparisons. First check endpoint health and repeat a long prompt to verify nonzero cached-token usage. Keep the scenario's default trajectory window and duration. Run each concurrency separately with a fresh artifact directory:

set -euo pipefail
source "${AGENTX_WORKDIR:?}/venv/bin/activate"
export HF_HOME="${HF_HOME:-$AGENTX_WORKDIR/hf-cache}"
export AIPERF_DATASET_MMAP_CACHE_DIR="$AGENTX_WORKDIR/dataset-cache"
mkdir -p "$AGENTX_WORKDIR/results"
AGENTX_RUN_DIR=$(mktemp -d "$AGENTX_WORKDIR/results/run-XXXXXX")
python -m pip freeze > "$AGENTX_RUN_DIR/client-packages.txt"
aiperf profile \
  --scenario inferencex-agentx-mvp \
  --url "${AGENTX_URL:?}" --model "${AGENTX_MODEL:?}" \
  --tokenizer "${AGENTX_TOKENIZER:?}" --endpoint-type chat \
  --public-dataset "${AGENTX_DATASET:?}" \
  --max-context-length "${AGENTX_MAX_CONTEXT_LENGTH:?}" \
  --concurrency "${AGENTX_CONCURRENCY:?}" \
  --random-seed "${AGENTX_SEED:?}" \
  --streaming --use-server-token-count --extra-inputs ignore_eos:true \
  --artifact-dir "$AGENTX_RUN_DIR" --ui simple \
  2>&1 | tee "$AGENTX_RUN_DIR/client.log"

Compare and report

Use the benchmarking guide for sweep isolation, performance metrics, and comparison controls. For AgentX:

• Concurrency counts session trees. Report actual overlapping HTTP requests separately because child sessions can overlap.
• Hold KV-cache bytes fixed when comparing KV formats. Keep the dataset, seed, context filter, and warmup fixed, and start a fresh server per point.
• Report token-weighted cache hits from server-reported usage, cache usage, and preemptions. Missing usage means cache hits are unknown.
• Report metadata.submission_valid and any metadata.submission_invalid_reasons from the result export. Flag invalid or missing validity instead of treating the run as a valid comparison.
• Include request errors, unfinished requests, and warmup failures. Synthetic prompts measure serving performance, not model quality; shortened smoke runs are not benchmark results.