Token economics

Optimize cost per successful task, not tokens per request. One user task can include several inferences (tool search, tool calls, follow-ups, retries); a lever that shrinks the first request but adds a second inference, more reasoning, or a lower success rate can lose money. Measure, change one lever, measure again.

The cost model

For every inference in a task:

cost = U·p_in + C·p_cached (+ W·p_write on GPT-5.6+) + O·p_out        (× tier multiplier)
U = uncached input tokens   C = cached input tokens   W = cache-write tokens
O = output tokens, INCLUDING hidden reasoning tokens

Three independent budgets are in play, and levers trade them against each other:

Never hardcode prices. Use the current Azure OpenAI pricing for the exact model, deployment type and region.

Measure first (usage fields)

Sum these per task across all inferences. Record counts, model/deployment, tool-catalog version and timings; never log prompts, tool arguments or outputs for cost diagnosis. For production telemetry use foundry-observability; for spend dashboards use foundry-cost-monitoring.

The optimization ladder

Work top-down. Earlier rungs are cheaper to adopt and rarely reduce quality. Judge every rung by cost per successful task (money and latency), never by raw input_tokens alone.

Deferred tool loading: tool_search + defer_loading

Sources: Azure OpenAI tool search, OpenAI tool search.

What it is

Without it, every tool's name, description and full JSON schema is sent on every inference. With it, the model initially sees only searchable surfaces and loads full definitions on demand. Loaded tools are injected at the end of the context, so the earlier prefix stays cache-stable.

Models are primarily trained to search namespaces and MCP servers. Keep each namespace under 10 functions, give it a short purpose-level description, and put rich detail in the deferred function descriptions (loaded only when needed).

Support and failure mode (verified)

• Supported on gpt-5.4 and later in the Responses API (Azure and OpenAI). Verified live on Azure (2026-10-14): gpt-5.4, gpt-5.4-mini, gpt-5.5, gpt-5.6-terra, gpt-5.6-luna, gpt-6-sol, gpt-6-luna, gpt-6.1-sol and gpt-6-astra.
• Earlier models fail hard: HTTP 400 — Tool 'tool_search' is not supported with gpt-5-mini-2025-08-07 (also observed for gpt-4.1). There is no silent fallback — gate the feature on the deployment's model family, do not try/except into a different tool list at runtime without logging it.
• Verified on Azure: hosted search over namespaces, deferred MCP servers ("type": "mcp" + defer_loading: true) and client-executed search.
• defer_loading on a namespace member applies to the function, not the namespace. A namespace may mix eager (always callable) and deferred functions.

Hosted vs client-executed search

Minimal hosted request (Entra ID; full runnable version is the measurement script):

response = client.responses.create(
    model="<gpt-5.4-or-later-deployment>",
    input="List open orders for customer CUST-12345.",
    tools=[
        {"type": "namespace", "name": "crm",
         "description": "CRM tools for customer lookup and order management.",
         "tools": [{"type": "function", "name": "list_open_orders",
                    "description": "List open orders for a customer ID.",
                    "defer_loading": True,
                    "parameters": {"type": "object",
                                   "properties": {"customer_id": {"type": "string"}},
                                   "required": ["customer_id"],
                                   "additionalProperties": False}}]},
        {"type": "tool_search"},
    ],
    parallel_tool_calls=False,
)

Rules that keep the savings

1. Return complete, trusted definitions only in client mode. Tools absent from tool_search_output.tools are not callable. Never echo definitions sourced from user input or untrusted registries — it is a tool-injection path.
2. Keep loaded tools stable. In client mode a tool loaded once stays callable in later turns; do not reload it. Removing or changing a loaded tool breaks the cache from that point forward.
3. Round-trip output items verbatim (including tool_search_call/_output and, with store=False, encrypted reasoning via include=["reasoning.encrypted_content"]). For tools added outside the search flow use an additional_tools input item and preserve its position.
4. Authorization is separate. Deferring a tool hides its schema, not its existence or callability. Do not declare tools the caller is not allowed to use.
5. Make required parameters honest. Newer models (observed on gpt-6-astra and gpt-6.1-sol) load the right deferred tool, then ask the user for missing required fields instead of inventing them; with eager loading the same models sometimes called write tools anyway. Mark only truly mandatory fields as required and treat "found but asked" separately from "not found" in evals.
6. Keep always-needed tools eager. A tool used on most turns is cheaper as an eager definition than as a search every time.
7. Verify the wire request through frameworks. Agent frameworks and SDK wrappers may not serialize namespace, defer_loading or tool_search, or may drop unknown fields; deferral then silently becomes eager (or a 400). Check one real request or the tool_search_call items in the output before trusting savings.
8. Different features: Foundry Toolbox tool search (foundry-toolbox) discovers tools in a versioned Toolbox; the OpenAI Agents API auto-defers MCP tools. The rules here apply to declared tools in Responses API requests.

Measured evidence (2026-10-14, Sweden Central, Global Standard)

Synthetic catalogs (30 functions in 5 namespaces ≈ 3.4k tokens; 160 in 20 ≈ 22k), the six demo cases, complete tool loops, reasoning.effort=low, store=False. Money = input-equivalent cost U + c·C + w·W + r·O with each model's retail Global Standard short-context ratios that day. Values are the deferred-vs-eager cost change (negative = deferral cheaper). Warm = the eager prefix was already cached (steady traffic); cold = first use, the eager catalog is written to cache. Indep. = each case is a new conversation; session = the six cases are turns of one conversation.

c, w, r = cached, cache-write and output price relative to uncached input. Raw input fell 57–94% in every row. Tool selection was 6/6 for both variants on all models with the final cases. † Cold-run cases did not yet give every required field; on write cases gpt-6-astra and gpt-6.1-sol loaded the right tool but asked for the missing values instead of calling (4/6 and 5/6), which lowers deferred cost. Single runs, synthetic tools: treat ±10–15 points as noise.

Also verified: a deferred MCP server (Microsoft Learn MCP, 3 tools) cut input 9,508 → 6,878 (−28%) with the same answer on gpt-5.4; mcp_list_tools still runs. Client-executed search worked end to end.

What the numbers say:

• Raw input is the wrong metric. Deferral always cut input by >55%, yet on a warm small catalog it raised cost on every model: the eager prefix is ~98% cached (billed at 5–10%), while deferral adds search output (billed 5–6×) and a shorter, partly uncached prefix.
• Large catalogs save on every current family (−37% to −68%) except gpt-5.5 in warm independent traffic (+1%): its deferred prefixes got no cache hits. GPT-5.6+/6.x cached 5–15k tokens of the deferred prefix.
• Cheaper cache reads shrink the win. gpt-6.1-sol reads cache at 0.05×, so a warm 160-function eager catalog costs little and deferral saved only 12% there.
• Cold starts favour deferral on GPT-5.6+ because the eager catalog must first be written at 1.25×: small catalogs are roughly neutral (−5% to +21%), large ones save 62–74%. Bursty or per-tenant catalogs live mostly in this regime.
• Latency had no consistent winner on GPT-5.6+/6.x; small catalogs were usually slower deferred. Single requests occasionally took 3–6 minutes on both variants — set explicit client timeouts.
• Searches may run twice per task (paths, then query).

Decide with your own numbers

Per task, deferral pays when

  T_cat · n · (1 − h_E·(1 − c))  +  T_cat · m · (w − 1)   >   ΔO · r  +  Δcache
  T_cat  = catalog tokens removed per inference   n = inferences per task
  h_E    = eager cache-hit ratio                   m = eager cache writes of the catalog
  w      = cache-write ratio (1.25 on GPT-5.6+)    ΔO = extra output (search + reasoning)
  Δcache = cost of cache hits the shorter deferred prefix loses

With a warm cache (h_E ≈ 0.98, c = 0.05–0.1) each removed catalog token is worth only 0.07–0.12 uncached tokens, so a small T_cat cannot pay for search output priced at r = 5–6×. Every cold write of the eager catalog (m) adds w − 1 in deferral's favour. Rules of thumb from the data, not guarantees:

• Defer when the catalog is a large share of the request (≈10k+ tokens), the tool set varies per user/tenant (eager cache already broken), traffic is bursty (the in-memory cache usually expires after 5–10 idle minutes), TPM 429s hurt, or selection accuracy drops with catalog size.
• Stay eager for catalogs of a few thousand tokens with steady traffic, or tools used on most turns — on every current model family. On gpt-5.5 prefer eager unless the catalog is large and conversations are long. To restrict tools per turn without breaking the cache, use tool_choice: allowed_tools (below) instead of editing the tools array.
• Run the script with --cases and your real catalog and price ratios; it prints input_equivalent_cost for both variants. Then compare the eager cached_tokens ratio with production telemetry, not a warm benchmark loop.

Measure before and after

MUST: Copy verbatim from references/python/measure_tool_search.py. Do NOT redefine inline. It runs the same task eager vs deferred as a complete tool turn and prints usage counters only.

pip install "openai>=3.26,<4" "azure-identity>=1.25,<2"
python references/python/measure_tool_search.py \
  --endpoint https://<resource>.openai.azure.com \
  --model <gpt-5.4-or-later-deployment> \
  [--tenant <tenant-id>] [--tools-file tools.json] \
  [--cases references/demo_cases.jsonl] [--session] \
  [--cached-ratio 0.1 --output-ratio 6 --cache-write-ratio 1.25]

• --endpoint also accepts a Foundry project endpoint (…/api/projects/<p>).
• Identity needs Cognitive Services OpenAI User on the resource; token scope is https://ai.azure.com/.default.
• --tools-file takes your real catalog (a JSON array of Responses tools) and derives the deferred variant (defer_loading on functions, namespace members and MCP servers, plus tool_search). Supply --expect-tool <name> or --cases.
• --cases (JSON Lines {"prompt", "expect_tool"}) measures selection success per variant; --session chains the cases as turns of one conversation; --cached-ratio/--output-ratio/--cache-write-ratio (from the current price sheet; write ratio 1.0 where writes are not billed separately) add an input_equivalent_cost comparison. Latency is reported as elapsed_s.
• found_not_called counts cases where the expected tool was loaded by search but not called — usually the model asking for missing required inputs, not a discovery failure. Inspect those before calling it a selection regression.
• Run warm and cold: the first run against a new catalog/deployment pays cache writes on GPT-5.6+; repeated runs show steady state. Report both.
• Exit 0 = deferred variant called every expected tool, 1 = it missed one (selection regression), 2 = API/config/input error (e.g. the HTTP 400 above).
• Run each variant several times and report distributions. There is no usable pre-flight token counter on Azure: responses.input_tokens.count returned HTTP 400 (This model is not supported by Responses API) for gpt-5.4, gpt-5.4-mini and gpt-5-mini — measure with real usage.

Other levers (concise)

Prompt cache. Stable content first, dynamic content last, append-only history, consistent prompt_cache_key per shared prefix (≈15 rpm per prefix+key before misses; partition hot prefixes across keys). Structured-output schemas and tool definitions are part of the cached prefix — keep their order deterministic. GPT-5.6+ adds billable cache writes and explicit breakpoints (prompt_cache_options, prompt_cache_breakpoint); earlier models return HTTP 400 for those fields. Measured on gpt-5.6-luna and gpt-6-luna with a new 2k-token prefix: the default implicit mode wrote 2,009 tokens on first use, then re-wrote the latest message on every request (11 tokens here); explicit mode with a breakpoint after the stable prefix wrote once (1,995) and then only read. For long, variable user turns the implicit per-request write is billed at 1.25× — prefer explicit breakpoints once the stable boundary is known. Details: prompt caching and foundry-hosted-agents references/prompt-caching.md.

Restrict tools per turn without breaking the cache. Keep the full tools array stable and pass tool_choice={"type": "allowed_tools", "mode": "auto", "tools": [{"type": "function", "name": "..."}]}. Measured on gpt-5.4: a different allowed subset on the next request still read 4,736 of 4,899 input tokens from cache. It accepts top-level functions only — namespace members and namespaces were rejected with HTTP 400 — so it complements, not combines with, namespaced deferral.

Fewer round trips. Every inference re-sends the whole prefix and history. Let independent tool calls run in parallel (parallel_tool_calls, default true), return compact tool results, cap built-in tool loops (max_tool_calls), limit retrieval results, and use detail: "low" for images that do not need fine detail.

Server-side state is not a token discount. previous_response_id and stored conversations save bandwidth and code, but all prior input in the chain is still billed as input (source). Only caching, compaction and smaller context reduce the bill.

Reasoning and output. reasoning={"effort": ...} (allowed values vary by model — check the reasoning models table); text={"verbosity": "low"} for terse answers; set max_output_tokens close to the real need: Azure's admission estimate counts it in full against TPM, and rejected (400) requests can count too (source). For classification/extraction try the lowest supported effort first and accept only if evals hold.

Compaction. Server-side: context_management=[{"type": "compaction", "compact_threshold": <tokens>}] on responses.create (works with store=False); or explicit client.responses.compact(...). With stateless chaining you may drop items before the latest compaction item; with previous_response_id do not prune. See Responses compaction.

Service tiers. service_tier="flex" per request on Global Standard (preview, 50% off, variable latency, HTTP 429 when capacity is unavailable, no automatic Standard fallback — retry explicitly with default if acceptable). Batch: separate deployment, 50% off, 24-hour target. Priority: lower, steadier latency at a premium. See Flex processing. PTU vs PAYG economics: paygo-ptu-cost-analyzer.

Anti-patterns

• Padding prompts to reach the 1,024-token caching threshold.
• Judging a lever by one request's input_tokens instead of cost per successful task.
• Assuming tool_search always saves money: on a small, warm catalog it measured +44–72% cost and slower turns.
• Editing the tools array per turn to restrict tools (breaks the cache) instead of allowed_tools.
• Believing previous_response_id makes earlier turns free.
• Enabling tool_search on a pre-5.4 deployment and catching the 400 silently.
• Injecting timestamps, request IDs or per-user data at the top of instructions.
• Dumping full API payloads as tool outputs; they are re-sent on every later turn.
• Setting max_output_tokens to the model maximum "just in case".
• Logging prompts or tool payloads to diagnose cost.