AI Latency Optimizer

Purpose

Reduce AI inference latency: pipeline profiling, parallelization, caching, model selection, and streaming strategies to meet SLA targets.

Input: Workflow spec or architecture, latency SLA (p50/p95), current latency baseline (optional), quality constraints
Output: Latency Optimization Plan with bottleneck analysis, optimizations ranked by impact, and SLA compliance projection Examples: See examples.md for worked input/output.

Workflow

Step 1: Profile latency budget

Decompose end-to-end latency:

Identify bottleneck stage (>40% of p95 or exceeding individual SLA).

Step 2: Set SLA targets

Distinguish streaming (TTFT matters) vs batch (total time matters).

Step 3: Apply latency levers

Step 4: Design parallel execution

For independent stages:

Input → [Retrieval || Tool prefetch || Classify] → Merge → Generate

Document synchronization point and timeout for slowest branch.

Step 5: Plan streaming strategy

Step 6: Validate against SLA

Project p50/p95 after optimizations. Confirm quality metrics unchanged via eval gates.

Run Validation checklist.

Decision Rules

Validation

• Latency profile with per-stage p50/p95
• Bottleneck stage identified
• SLA targets documented (TTFT and e2e)
• ≥3 optimizations evaluated with impact estimates
• Parallel execution plan for independent stages
• Streaming strategy defined for user-facing paths
• Projected p95 meets SLA after changes
• Quality impact assessed (no eval regression)

Anti-patterns

• Optimizing non-bottleneck — shaving 10ms off postprocess when LLM takes 3s.
• Cold start ignored — measuring warm-cache latency only.
• Sequential by default — chaining retrieval and tools that could run in parallel.
• Fake streaming — buffering full response then simulating stream.
• Latency without monitoring — no production p95 tracking after deploy.

Best Practices

• Measure client-side latency, not just server-side model time.
• Set per-stage timeouts to fail fast and degrade gracefully.
• Use regional inference endpoints close to users.
• Pre-warm models/instances before traffic spikes.
• Pair with ai-cost-optimizer when latency fixes increase spend.

Output Structure

# Latency Optimization Plan: [System Name]

## Current Profile
| Stage | p50 | p95 | % total |
|-------|-----|-----|---------|

## SLA Targets
| Metric | Target | Current | Gap |
|--------|--------|---------|-----|

## Bottleneck Analysis
[Root cause and contributing factors]

## Optimizations
| Lever | Est. p95 reduction | Trade-off | Priority |
|-------|-------------------|-----------|----------|

## Parallel Execution
[Diagram and sync points]

## Streaming Plan
[What streams, what waits]

## Projected Performance
| Metric | Before | After | Meets SLA |
|--------|--------|-------|-----------|

## Monitoring
| Metric | Source | Alert |
|--------|--------|-------|

Next Skills