LLM Inference Serving

The bottleneck is never just compute — it shifts between memory bandwidth, inter-GPU communication, CPU overhead, and KV cache capacity depending on model architecture, request profile, and hardware topology.

Every serving decision is a trade-off between latency, throughput, cost, and accuracy. The right configuration depends on your workload.

Parallelism Strategy — When to Use What

Two questions drive the choice: (1) does the model fit on one GPU, and (2) what is the dominant bottleneck?

Decision rules:

• Single NVLink domain (8 GPUs): TP is default. Add EP for MoE models.
• Cross-node (>8 GPUs): PP for dense models, EP for MoE. Avoid cross-node TP — bandwidth is ~5% of intra-node NVLink.
• GB300 NVL72 rack: 72 GPUs at 1,800 GB/s each via NVLink. Treat entire rack as one NVLink domain for EP. 130 TB/s total all-to-all bandwidth eliminates MoE dispatch bottleneck.
• Non-standard architectures (parallel track transformers): PP maps tracks to TP axis, stages to PP axis. Reuses existing NVLink communicators. Requires custom executor to avoid compiled DAG deadlocks.

Rule of thumb: Heavy traffic stays inside NVLink domain. Cross-domain hops carry only lightweight data (activations for PP, expert dispatch for EP). Never route bulk collectives over Ethernet/InfiniBand when NVLink is available.

Speculative Decoding — Decision Tree

Speculative decoding is the single most impactful lossless optimization for decode-heavy workloads. It predicts K tokens with a cheap drafter; the target model verifies all K in one forward pass via rejection sampling. Output distribution is mathematically identical.

When it helps:

• Decode-bound workloads (output >> input tokens)
• Structured/repetitive output (JSON, code, tool calls) — high acceptance rates
• Low-throughput regime (few concurrent requests per replica)

When it hurts or is marginal:

• Prefill-heavy workloads (long input, short output)
• Very high concurrency (GPU already saturated; drafter adds overhead)
• Low acceptance rate on diverse/creative output

Algorithm selection

Optimization techniques for speculative decoding

• Slim Spec (low-rank LM head): The LM head (embedding→vocabulary projection) can be 50–60% of drafter latency. Insert a low-rank bottleneck: at 1/8 embedding size, get 5× speedup on that layer with 0.99 acceptance length retention. ~10% real production speedup.
• Confidence head + prefix tree trimming: Skip verification of low-confidence draft tokens to save verifier time.
• Shared suffix tree across replicas: Store suffix tree in a dedicated Ray actor. Eliminates cold-start penalty when new replicas scale up. Improves acceptance rate ~35%.
• Custom drafter training: Train speculators on customer-specific data for higher acceptance length. Best DSpark recipe: 48 GB300 GPUs, 12 nodes, TP8 extractors with hidden-state streaming via MoonCake connector. Best acceptance length: 4.15.

KV Cache Management

KV cache grows with context length × batch size. For Llama 405B at 128K context, prefill computation cost grows near-quadratically. This is the dominant memory pressure in agentic workloads.

Memory hierarchy (Dynamo tiers)

Key patterns

• KV cache offload: 20× improvement in time-to-first-token vs. recomputation from scratch. 90% GPU time savings. At 100K GPU cluster scale, ~$66M savings over 3 years.
• Cache-aware routing: Route requests to replicas that already hold relevant KV cache. Critical for agentic (95%+ cache reuse) and chat (70–75% reuse) workloads. Integrated via Dynamo KV indexer + Ray Serve.
• FP4/FP8 KV cache quantization: DeepSeek V4 uses CSSA/HCA compression — 90% KV savings vs. V3.2. For sliding window attention layers, skip FP8 quantization when quant/dequant overhead exceeds memory bandwidth benefit.
• Long context control: Cap max_model_len to prevent individual requests from hogging resources. A single 1M-token needle-in-haystack test can consume ~60 seconds of GPU time.
• Data reduction: 1.4:1 compression ratio on KV cache offloaded to enterprise storage. Translates to extended sessions, more users, higher cache hit rates.

Serving Architecture Patterns

Prefill-decode disaggregation

Prefill is compute-bound (algorithm intensity >100). Decode is memory-bandwidth-bound (algorithm intensity ~1.3). Separating them lets each pool be independently sized and optimized.

• Same-node slicing (B200): Split GPU resources on one node. Avoids cross-node latency for activation transfer.
• Dedicated pools: Separate prefill and decode pools with a router. Adds latency from the routing hop. Best when prefills are long and would stall decode.
• Disaggregated quantization: Use high precision (FP16) for prefill KVs (reused by every decode token). Use low precision (W4A4/W4A16) for decode. 2.3–2.5× decode speedup with no accuracy loss.

Request routing

• HAProxy replacement for Ray Serve proxy: Bypasses Python GIL; true multi-threading. 9× streaming throughput gain. 24× throughput for decode-heavy workloads. Bundled with Ray ≥ 2.57.
• Split control/data plane: Move routing decisions to a one-time control plane query. Response tokens stream directly from replica to client, bypassing intermediate deployment. Eliminates per-token TPOT tax.
• Inference Gateway (GKE): Multi-cluster, multi-region routing via Kubernetes Gateway API. Body-based routing extracts model ID from OpenAI request body. Rate limiting, auth, and safety filtering via routing extensions.

Controller scaling (Ray Serve)

• Ceiling: 1K → 4K replicas (4× improvement). Control loop 32× faster. RPCs 400× fresher.
• Key optimizations: Pydantic v1→v2 (8× for object mutations), Cython autoscaler (13×), zlib-compressed metrics (bounded <2s vs. 80s snowball), sliding-window health checks.
• Path to 8K: Push-based health checks, node-level RPC aggregation.

Agentic Inference Considerations

Agentic workloads generate 10–15× more tokens than single-turn chat. Each iteration adds tool calls, retrieved documents, and reasoning — compounding context across turns.

Key pressures:

1. Expanding context → memory pressure. KV cache grows with every turn. Cache-aware routing and offload are mandatory, not optional.
2. Latency multiplication. 10–20 inference calls per prompt iteration. Single-digit ms overhead compounds to seconds. Every hop in the serving stack matters.
3. Concurrent orchestration. Multiple agents sharing a box. Tool calls are CPU-bound (file system, retrieval, parsing). Offloading tool calls to Ray workers for GIL-bypassed parallelism gives 1.6–1.7× speedup.
4. CPU-GPU ratio. For a 70B model coding task: ~50/50 CPU/GPU time split. Smaller models (8B) produce tokens faster → CPU becomes bottleneck → need more CPUs. Larger models → fewer CPUs needed. Running SLMs on head-node CPUs can increase per-box user capacity by 1.44×.

Production patterns:

• Separate Ray clusters per use case even when serving the same model. Different SLAs (TPS, P99, prompt length) require different vLLM configs.
• Reasoning effort tuning. Setting reasoning level to "low" for classification/tool-call tasks: 930ms → 450ms latency reduction.
• Fast tokenization. Replace default tokenizer with Rust-based fast tokenizer. Gains scale with prompt size — critical for agentic prompts with full conversation history + tool definitions.
• Shared suffix tree for JSON tool responses. Agentic output is highly structured — suffix-based speculative decoding with shared state across replicas is a strong fit.

Cost Optimization Levers

Hardware Selection Quick Reference