Throughput & Cost Skill

Use this skill to answer a serving-economics question, not a quality one: how much agentic-coding load can this model sustain on this exact EC2 instance, and therefore what does a task really cost? It sweeps concurrency, measures throughput from vLLM's own counters, and turns the fixed instance $/hr into a defensible cost per token and cost per task — replacing the fictional total_cost_usd the quality harness records for self-hosted models (which has no per-token bill).

This is deliberately separate from the /benchmark (quality) skill. Quality runs one agentic session per task and scores the artifacts; throughput holds N sessions in flight and measures the server. They share the same building blocks but answer different questions, so they are different scripts.

The cost model

cost_per_output_token = ($/hr ÷ 3600) ÷ sustained_output_tokens_per_second
cost_per_task         = cost_per_output_token × output_tokens_per_task

sustained_output_tokens_per_second comes from vLLM's generation_tokens_total counter delta over each concurrency level's window (server-measured — a session need not finish to have generated its tokens). output_tokens_per_task comes from the model's real /swe runs (metrics.json).

Why it works this way (read before changing it)

• Real agentic load, server-side measurement. Each in-flight unit is a genuine /swe session against a real cloned repo (large read-heavy prompts, short outputs — the true workload). But throughput is read from vLLM's DuckDB counters, not from completed sessions: an agentic session can take ~30 min, far longer than a throughput window, so sessions are cut off at the window (their claude -p timeout is bounded to the remaining window). The tokens they generated while running are already in the counter delta.
• Concurrency is the controlled variable. The harness holds exactly N sessions in flight, refilling a slot as soon as one is cut off/finishes, for a fixed wall-clock window per level.
• One named DuckDB session per level ({model}_c{N}) so each level is sliceable.

The three inputs

1. model — the served-model-name already running on the local vLLM server (e.g. gemma-4-31b). This skill does not start the server; serve it first (use the /vllm-setup or /benchmark flow, or vllm-serve.sh).
2. instance + $/hr — the EC2 instance type (for provenance) and its on-demand hourly price. Default reference: g6e.12xlarge at $10.49/hr (us-east-1 on-demand). Use the user's actual spot/reserved/negotiated rate if they give one.
3. concurrency levels + window — default sweep 1 2 5 7 10 15 20, --duration-seconds 600 (10 min) per level (~60-70 min total) for good coverage. Shorten (e.g. 300) for a quick look.

Workflow

Step 1 — Confirm the server and inputs

curl -s http://127.0.0.1:8000/v1/models | python3 -c "import sys,json;print([m['id'] for m in json.load(sys.stdin).get('data',[])])"

Confirm it serves {model}. If not, stop and tell the user to serve it first. Restate the plan (model, instance $/hr, levels, window) and where results land, then proceed.

Step 2 — Derive the per-task output-token figure (for cost-per-task)

If the model has quality-run artifacts, use its mean output tokens per task so cost-per-task reflects real agentic sessions:

cd benchmarks
python3 - <<'PY'
import json, glob
tot=n=0
for f in glob.glob("swe-benchmark-data/{model-slug}/*/*/metrics.json"):
    ot=json.load(open(f)).get("metrics_that_matter",{}).get("output_tokens")
    if ot: tot+=ot; n+=1
print("mean output tokens/task:", round(tot/n) if n else "unknown (pass a value)")
PY

Pass that as --output-tokens-per-task in Step 4. If unknown, skip it (cost-per-1M-tokens is still produced).

Step 3 — Run the sweep

The sweep drives real /swe2 load at each concurrency level and captures the vLLM server metrics into one shared DuckDB (one named session per level). It does not score anything.

By default it uses dataset/multi-repo-throughput.yaml (25 tasks across 25 different repos in six languages). The harness fills each in-flight slot with a distinct task (random draw without replacement), so at concurrency N <= 25 the N slots each clone and reason over a different repo — simulating N developers on their own projects rather than N copies of one task. Use --dataset dataset/mcp-gateway-registry.yaml only if you deliberately want single-repo load.

cd self-hosted/vllm
./scripts/run-throughput-sweep.sh --model {model} \
  --concurrencies "1 2 5 7 10 15 20" --duration-seconds 600 [--context-window N] [--endpoint URL]

Tell the user, before it runs:

• It holds N real agentic sessions in flight per level for the window; sessions still running at window close are cut off (expected — throughput is server-measured, not per-completed-session).
• Watch the collector + a level in flight:

  tail -f self-hosted/vllm/benchmark-output/throughput/{model}/collector-c*.log
  tail -f self-hosted/vllm/logs/vllm-serve.log

• A slow/dense model at high concurrency may saturate KV cache — which is itself the finding.

Step 4 — Build the performance summary + dashboard

cd self-hosted/vllm
DB=benchmark-output/throughput/{model}/throughput-metrics.duckdb
uv run python -m clients.build_performance_summary \
  --model {model} --db "$DB" --instance-type g6e.12xlarge \
  --dollars-per-hour 10.49 --output-tokens-per-task {N-from-step-2}
uv run python -m clients.build_performance_dashboard \
  --summary benchmark-output/throughput/{model}/performance-summary.json

This writes performance-summary.json (machine-readable: per-level throughput, TTFT/TPOT, KV-cache/running/waiting, cost/token, cost/task; plus peak throughput and cheapest $/1M) and performance-dashboard.html (self-contained, offline).

Step 5 — Report

• Headline: peak sustained output tokens/s and the concurrency it occurs at; cheapest $/1M output tokens; cost per task.
• The saturation story: where KV-cache % and waiting-requests climb (the practical concurrency ceiling), and how TTFT/TPOT degrade with load — the "acceptable latency" knee that anchors the cost figure.
• Point the user at the dashboard HTML and the JSON.

Cost an arbitrary task (outside the harness)

The per-token cost in the summary is a property of the model + hardware + load, not of the tasks the harness ran. So any task on the same served model can be costed from just its input/output token counts — a /benchmark quality run, a production trace, a hypothetical workload:

cd self-hosted/vllm
uv run python -m clients.cost_for_task \
  --summary benchmark-output/throughput/{model}/performance-summary.json \
  --input-tokens {N} --output-tokens {M}   # [--concurrency 20] [--json]

It reports both lenses (blended and lab-split) at every concurrency level plus the cheapest point, reusing the exact per-token costs build_performance_summary.py derived. The dashboard's interactive N:M calculator does the same thing in the browser.

Notes

• This skill does not manage the vLLM server. Serve the model first; the sweep only checks it is up and fails loudly if not.
• Cost is hardware-derived, not a bill. The number is instance $/hr ÷ measured tokens/hr — real and defensible, unlike the token-priced estimate quality runs record for self-hosted models.
• Every script takes --help.