Modal Endpoints

Use this skill when deploying Auto Endpoints: managed, OpenAI API-compatible LLM inference on Modal with minimal configuration (SDK/CLI 1.5.1+).

Endpoints are built on Modal Servers. For custom server code, see modal-servers. For hand-rolled vLLM/SGLang Functions, see modal-web-serving / modal-scaling-performance.

When to Use This Skill

• Need a production LLM API with one CLI command
• Serving catalog models (Qwen, Kimi, Gemma, DeepSeek, Nemotron, GPT-OSS, GLM, …)
• Serving fine-tunes from Hugging Face or a Modal Volume
• Want scale-to-zero, usage-based GPU billing, and tuned engine recipes
• Prefer managed speculative decoding / recipes over writing server code first

Prefer a custom @app.server when you need full control of engine flags, multi-model routers, or non-OpenAI protocols.

Quick Start

# Proxy token (required unless --unauthenticated)
modal workspace proxy-tokens create
# With RBAC:
modal workspace proxy-tokens allow wk-... main

modal endpoint create --model Qwen/Qwen3.5-4B
# Optional explicit name; default derived from model id
modal endpoint create --name agent --model zai-org/GLM-5.2-FP8

Dashboard: Endpoints tab mirrors CLI options.

Calling the API

OpenAI Chat Completions under /v1. Model name = base catalog repo id (or custom HF repo id).

curl "<endpoint-url>/v1/models" \
  -H "Modal-Key: $MODAL_PROXY_TOKEN_ID" \
  -H "Modal-Secret: $MODAL_PROXY_TOKEN_SECRET"

# Or for quick tests (auth injected; higher latency):
modal curl "<endpoint-url>/v1/models"

Find URLs: dashboard or modal endpoint list.

Custom Weights

Always pass a catalog --model (recipe base), then custom weights:

# Hugging Face fine-tune
modal endpoint create \
  --name my-ft \
  --model Qwen/Qwen3.6-27B \
  --custom-hf-repo org/my-ft \
  --custom-hf-revision <sha> \
  --custom-hf-token "$HF_TOKEN"   # if gated/private

# Modal Volume (directory must contain config.json)
modal endpoint create \
  --name my-volume-ft \
  --model Qwen/Qwen3.5-4B \
  --custom-volume-name my-volume \
  --custom-volume-path /checkpoints/1234

Placement

modal endpoint create \
  --model Qwen/Qwen3.5-4B \
  --routing-region us-east \
  --compute-region us-west

Lifecycle Management

modal endpoint list --env prod
modal endpoint list --env prod --json
modal endpoint stop qwen3-5-4b --env prod

Stop tears down containers and stops compute billing. Autoscaling overrides are adjustable in the UI. Idle scale-to-zero → no compute charge while idle.

Endpoints vs Custom Servers vs Web Functions

Inspect generated Server source: Endpoint dashboard Source panel.

Symptom Triage

401 on requests

• Create proxy token; pass both headers; with RBAC allow the Environment
• Or recreate with --unauthenticated if public is intentional

Endpoint slow to become ready

• Large models / weight pulls — watch dashboard provisioning
• Prefer Volume-cached weights for custom checkpoints

Wrong model name in API calls

• Use the base/catalog repo id (or custom HF id), not the Endpoint display name

Reference Map

• references/endpoint-cli.md — create/list/stop and auth flags
• references/custom-weights-placement.md — HF/Volume weights and regions
• references/when-to-use-endpoints.md — product boundaries

Guardrails

• Requires client/CLI with Endpoints support (1.5.1+)
• Secrets for proxy tokens are shown once at creation — store safely
• Region pinning multiplies cost — mention when recommending --compute-region
• Endpoints bill standard GPU/CPU rates while containers run

Key References

• https://modal.com/docs/guide/endpoints
• https://modal.com/blog/introducing-auto-endpoints
• https://modal.com/docs/guide/servers
• https://modal.com/docs/sdk/py/changelog (1.5.1+)