Modal Servers

Use this skill when deploying a process that listens on a port inside Modal with the Server primitive (SDK 1.5.1+), optimized for ultra-low-latency HTTP between clients and containers.

For Function-based web endpoints (fastapi_endpoint / asgi_app / wsgi_app / web_server), see modal-web-serving. For one-command LLM inference products, see modal-endpoints.

When to Use This Skill

• Serving vLLM, SGLang, or any HTTP server with minimal proxy latency
• Need regional request routing (routing_region) close to clients
• Want sticky sessions via Modal-Session-ID
• Migrating from @modal.web_server to @app.server
• Need default authenticated ingress (proxy tokens) or explicit unauthenticated=True
• Building custom inference behind the same infrastructure as Auto Endpoints

Quick Start

import modal
import subprocess

app = modal.App("my-server")

@app.server(port=8000, unauthenticated=True)
class FileServer:
    @modal.enter()
    def start(self):
        # Must bind 0.0.0.0 and listen on the configured port
        self.proc = subprocess.Popen(
            ["python", "-m", "http.server", "8000", "--bind", "0.0.0.0"]
        )

    @modal.exit()
    def stop(self):
        self.proc.terminate()

Deploy: modal deploy server_app.py

URL shape: https://<workspace>--<app>-<server>.<routing_region>.modal.direct/...

Retrieve programmatically: modal.Server.get_url() (see reference).

Servers vs Functions vs Web Decorators

Core Configuration

@app.server(
    image=image,
    gpu="H100",
    port=8000,
    startup_timeout=30 * 60,
    target_concurrency=32,       # soft target for autoscaling
    min_containers=0,
    max_containers=10,
    buffer_containers=1,
    scaledown_window=10 * 60,
    exit_grace_period=30,        # finish in-flight before SIGTERM
    routing_region="us-east",    # us-east | us-west | ca-central | eu-west | ap-south
    compute_region="us",         # optional: pin container placement (cost multiplier)
    unauthenticated=False,       # default: proxy-token auth required
    h2_enabled=False,
    secrets=[...],
    volumes={...},
    enable_memory_snapshot=False,
)
class InferenceServer:
    @modal.enter()
    def start(self):
        ...

Autoscaling notes

• Omit target_concurrency → no concurrency-based scale-out (still distributes if min_containers > 1).
• Prefer unset target_concurrency over max_containers=1 for singletons (allows rolling replacement).
• Dynamic tuning: modal.Server.update_autoscaler(...) (reset on next deploy).
• Containers are not ready until the port is listening; failed readiness within startup_timeout → failed container.
• Health checks verify the port stays up; repeated failures replace the container.

Zero-to-one

When no containers are warm, requests are rejected with 503 (they do not queue like Function inputs). Clients must retry. The first request still triggers a cold start.

Authentication

Default: require headers:

Modal-Key: wk-...
Modal-Secret: ws-...

Create tokens:

modal workspace proxy-tokens create
# With RBAC, scope to the Server's environment:
modal workspace proxy-tokens allow wk-... main

Testing helper (adds auth for you; higher latency): modal curl <url>/path

Public traffic: set unauthenticated=True.

RBAC: tokens must be scoped to the Environment or requests get 403.

Sticky Sessions

Send header Modal-Session-ID: <any-string> so related requests hit the same container (useful for WebSockets / conversational state).

Lifecycle

1. @modal.enter() starts the process binding 0.0.0.0:<port>
2. Modal waits until the port accepts connections
3. Traffic flows through the regional routing plane
4. Scale-down: stop new requests → exit_grace_period for in-flight → SIGTERM → @modal.exit() → hard kill after grace

Do not use @modal.method() or Function web decorators on a Server class. Request handling is entirely the child process’s job.

Symptom Triage

503 on first traffic

• Expected at zero containers; keep min_containers>=1 or retry with backoff
• Check startup_timeout and that the process binds the right port

401 / 403

• Missing proxy token headers, or token not allowed for this Environment
• Accidentally left unauthenticated=False for public clients

High latency still

• Move routing_region (and optionally compute_region / colocate) nearer clients
• Ensure app isn’t doing cold model load on every request — use @modal.enter + snapshots if appropriate

Migrating from @modal.web_server

• Move startup into @modal.enter on an @app.server class
• Revisit auth defaults and 503 cold-start behavior
• Drop Function input concurrency decorators; use target_concurrency instead

Reference Map

• references/server-basics.md — definition rules, URL, vs Cls/web_server
• references/autoscaling-auth.md — concurrency, auth, sticky sessions, routing
• references/migration-from-web-server.md — migration checklist

Guardrails

• Requires Modal Python SDK 1.5.1+
• Always bind 0.0.0.0, not 127.0.0.1
• Do not assume request queueing or built-in retries
• Serialization, timeouts, and retries are application concerns
• Mark regional compute pinning cost multipliers when advising users

Key References

• https://modal.com/docs/guide/servers
• https://modal.com/docs/sdk/py/latest/modal.App (.server)
• https://modal.com/blog/serverless-servers
• https://modal.com/docs/sdk/py/changelog (1.5.1+)