AI Cost Optimizer

Purpose

Optimize LLM inference costs: token budget allocation, model routing, caching strategy, and batching without breaching quality thresholds.

Input: AI architecture or workflow spec, usage volume projections, quality thresholds, current cost baseline (optional)
Output: Cost Optimization Plan with savings estimates, trade-off analysis, and implementation priorities Examples: See examples.md for worked input/output.

Workflow

Step 1: Establish cost baseline

Identify top 3 cost drivers by spend share.

Step 2: Audit token usage

Analyze per stage:

• Redundant context (duplicate documents, full history)
• Over-sized system prompts
• Verbose output formats
• Unnecessary model tier for task complexity

Step 3: Evaluate optimization levers

Step 4: Design model routing

Request → Classifier → [Simple path: small model] / [Complex path: frontier model]

Define routing criteria with fallback when classifier uncertain.

Step 5: Plan caching layer

Document cache hit rate target and monitoring.

Step 6: Quantify savings and validate

Project monthly savings per lever. Confirm no quality metric drops below threshold from eval plan.

Run Validation checklist.

Decision Rules

Validation

• Cost baseline with per-component breakdown
• Top 3 cost drivers identified
• ≥3 optimization levers evaluated with trade-offs
• Savings projected with assumptions stated
• Quality impact assessed against eval thresholds
• Model routing criteria defined (if applicable)
• Caching strategy with invalidation rules
• Implementation priority ranked by ROI

Anti-patterns

• Blind downgrade — switching to cheaper model without eval comparison.
• Cache without invalidation — serving stale answers after knowledge update.
• Token counting ignorance — optimizing output while input is 90% of cost.
• Premature fine-tuning — expensive training before prompt optimization exhausted.
• Hidden retry costs — aggressive retries multiplying spend silently.

Best Practices

• Measure cost per successful task, not per API call.
• Track token usage by workflow stage in production.
• A/B test cost optimizations against quality metrics.
• Set budget alerts at 80% and 100% of monthly cap.
• Pair with ai-latency-optimizer when both NFRs conflict.

Output Structure

# Cost Optimization Plan: [System Name]

## Baseline
| Component | Cost/request | Monthly | Share % |
|-----------|--------------|---------|---------|

## Findings
| Issue | Waste estimate | Recommended fix |
|-------|----------------|-----------------|

## Optimization Levers
| Lever | Est. savings | Quality risk | Priority |
|-------|--------------|--------------|----------|

## Model Routing
[Classifier criteria and model mapping]

## Caching Strategy
[Type, key, TTL, invalidation]

## Projected Savings
| Scenario | Monthly cost | vs baseline |
|----------|--------------|-------------|

## Implementation Roadmap
| Phase | Changes | Expected savings |
|-------|---------|------------------|

Next Skills