Artificial Analysis: pick the right model

Live benchmark data from https://artificialanalysis.ai — intelligence index (10 evals), coding index, math index, output speed, time-to-first-token, pricing, and context-window, for ~400+ LLMs plus media-model ELO leaderboards.

Your job: when the user needs a model recommendation, run recommend.sh with the right profile — do not guess from memory. Model rankings change weekly; never answer from training data alone.

Prerequisites

1. AA_API_KEY must be exported. Free key (1,000 req/day) at https://artificialanalysis.ai/documentation. If missing, scripts prompt on TTY or fail with a clear pointer.
2. curl and jq installed.

Responses cached 24h at ~/.cache/artificial-analysis/ — repeated calls are free of rate-limit cost.

Decision flow — ALWAYS follow this for "which model" questions

1. Classify the user's need → pick ONE profile (see table below).
2. Extract hard constraints: budget, min context, required provider, required quality floor.
3. Run:  recommend.sh <profile> [--budget $/M] [--min-context N] [--min-intel N] [--creator X] --top 5
4. Report the top pick with its trade-offs vs. #2 and #3.
5. If the user names two or more specific models, follow with: compare.sh <slug> <slug> ...

Do NOT skip step 3. Benchmark numbers change; the API is the source of truth.

Profile selection

When in doubt, use balanced. If the user has two conflicting hints (e.g. "fast and smart"), pick the dominant one and say so.

Hard constraints → flags

• Budget → --budget 5 means ≤ $5/M tokens blended (3:1 input:output).
• Min context → --min-context 200000.
• Quality floor → --min-intel 60 (intelligence_index is ~0–100-ish).
• Specific vendor → --creator anthropic (matches slug or name, case-insensitive substring).

Core scripts

recommend.sh — the one you'll use most

# Pick the best general model
"${CLAUDE_SKILL_DIR}/scripts/recommend.sh" balanced --top 5

# Cheapest decent model for batch processing
"${CLAUDE_SKILL_DIR}/scripts/recommend.sh" cheap --min-intel 55

# Fast Anthropic model for streaming UI
"${CLAUDE_SKILL_DIR}/scripts/recommend.sh" fast --creator anthropic

# Smart model that fits a 500k-token document
"${CLAUDE_SKILL_DIR}/scripts/recommend.sh" long-ctx --min-context 500000

# Best coding model under $10/M
"${CLAUDE_SKILL_DIR}/scripts/recommend.sh" coding --budget 10

Output: ranked table with score, slug, creator, intel/code/math, t/s, TTFT, $/M, ctx.

models.sh — leaderboard

# Top 10 by raw intelligence
"${CLAUDE_SKILL_DIR}/scripts/models.sh" --top 10 --sort-by evaluations.artificial_analysis_intelligence_index

# Just the slugs (for piping into compare.sh)
"${CLAUDE_SKILL_DIR}/scripts/models.sh" --top 20 --slugs

# Inspect available fields on the first model
"${CLAUDE_SKILL_DIR}/scripts/models.sh" --path

Sort keys containing latency, price, ttft, or time_to_first are treated as lower-is-better automatically.

compare.sh — head-to-head

"${CLAUDE_SKILL_DIR}/scripts/compare.sh" claude-4-5-sonnet gpt-5-xhigh gemini-3-1-pro-preview

Prints rows for creator, intel, coding, math, t/s, TTFT, blended / input / output price, and context window.

media.sh — image / audio / video leaderboards

"${CLAUDE_SKILL_DIR}/scripts/media.sh" image --top 15
"${CLAUDE_SKILL_DIR}/scripts/media.sh" edit  --top 10
"${CLAUDE_SKILL_DIR}/scripts/media.sh" tts   --top 10
"${CLAUDE_SKILL_DIR}/scripts/media.sh" ttv   --categories --top 10
"${CLAUDE_SKILL_DIR}/scripts/media.sh" i2v   --categories --top 10

Ranked by ELO rating.

fetch.sh — raw access (for custom jq)

"${CLAUDE_SKILL_DIR}/scripts/fetch.sh" data/llms/models --raw \
  | jq '[.data[] | select(.context_window >= 1000000)] | length'

Guidance when reporting to the user

1. Name the top pick by slug, link to its AA page (https://artificialanalysis.ai/models/<slug>).
2. Justify with the two or three metrics that drove the choice — e.g. "intelligence 72, $3.20/M blended, 190 t/s".
3. Mention the runner-up in one line with the tradeoff ("If you want lower latency, try X at $/M but intel Y").
4. Attribute: say "per Artificial Analysis benchmarks" — that's the data source and their ToS expects it.
5. If slugs are unfamiliar to the user, offer compare.sh as a follow-up.

Failure handling

• 401 → remind user about AA_API_KEY and link to https://artificialanalysis.ai/documentation.
• 429 → the script auto-falls-back to stale cache; warn the user it's stale.
• Empty filter result → loosen a constraint (usually --budget or --min-intel).
• Unknown slug in compare.sh → it warns on stderr; list candidates with models.sh --slugs | grep -i <partial>.

Notes on authentication

The Artificial Analysis API requires an API key. There is no unauthenticated public endpoint — the website renders data server-side through authenticated calls. A free key (1,000 req/day) is explicitly offered for this use case; the skill caches aggressively to stay under that limit.