agentleFS
botirkhaltaevbotirkhaltaev/botir-skills/skills/modal-storage/SKILL.md

modal-storage

Skill4 starsChanged 3 months agoApache-2.0View on GitHub

Persistent and cloud storage on Modal — Volumes, CloudBucketMount (S3/GCS/R2), and model weight management. Use when storing data, sharing files between Functions, managing ML model weights, mounting cloud buckets, or configuring Volume mount options (read-only / sub_path).

We split this skill into the rules it gives an agent

13 rules. About 0 are common, 5 are distinctive, and 8 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈5

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 8

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Use this skill when working with persistent storage, sharing data between Functions, managing model weights, or mounting external cloud …

    line 8
  2. Very specific:

    Prefer Volumes. NetworkFileSystem remains deprecated — migrate to Volumes.

    line 27
  3. Distinctive:

    When running on AWS, Modal automatically uses S3 Gateway endpoints for zero-cost data transfer.

    line 187
  4. Distinctive:

    Use region-specific or global S3 endpoints (not cross-region)

    line 190
  5. Very specific:

    Check the Volume name matches: modal volume ls vol-name /

    line 216
  6. Very specific:

    Ensure create_if_missing=True on first use

    line 217
  7. Very specific:

    Verify the mount path in volumes= dict

    line 218
  8. Very specific:

    If using sub_path, confirm the subdirectory path

    line 219
  9. Very specific:

    Use @modal.enter() to load once per container

    line 223
  10. Distinctive:

    Verify IAM permissions (read, write, list)

    line 228
  11. Very specific:

    Check AWS_REGION matches bucket region

    line 229
  12. Very specific:

    Use volume.reload() / sb.reload_volumes() in long-running containers to see updates

    line 241
  13. Distinctive:

    Don't store secrets in Volumes — use Modal Secrets instead

    line 244

≈ “Common” is an estimate: at least 1 in 5 of the 308,232 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Modal Storage

Use this skill when working with persistent storage, sharing data between Functions, managing model weights, or mounting external cloud storage.

When to Use This Skill

  • Storing and loading ML model weights
  • Sharing files between Functions
  • Persisting data across container restarts
  • Mounting S3, GCS, or Cloudflare R2 buckets
  • Managing large datasets for batch processing
  • Mounting Volume subdirectories (especially in Sandboxes)

Storage Options

TypeBest ForMutabilityPerformance
VolumeModel weights, shared dataWrite-once, read-manyHigh (distributed, cached)
CloudBucketMountExisting cloud dataRead/write via mountpointDepends on bucket
NetworkFileSystem(deprecated)Read/writeModerate

Prefer Volumes. NetworkFileSystem remains deprecated — migrate to Volumes.

Volumes (Recommended)

Create and use

volume = modal.Volume.from_name("my-data", create_if_missing=True)

@app.function(volumes={"/data": volume})
def process():
    with open("/data/result.txt", "w") as f:
        f.write("hello")

Mount options (with_mount_options)

Configure per-mount options (not stored on the Volume itself). Especially useful for Sandboxes that should only see one subdirectory:

volume = modal.Volume.from_name("my-data")

# Read-only
@app.function(volumes={"/models": volume.with_mount_options(read_only=True)})
def infer():
    ...

# Subdirectory mount (created if missing) — great for multi-tenant Sandboxes
sb = modal.Sandbox.create(
    app=app,
    volumes={
        "/user_data": volume.with_mount_options(sub_path=f"/users/{user_id}"),
        "/models": volume.with_mount_options(read_only=True, sub_path="/models"),
    },
)

sub_path is directories only — you cannot mount a single file.

Volumes v2 sync

On Volumes v2, flush commits from inside a Sandbox or modal shell with:

sync /path/to/mountpoint

Useful for long-running Sandboxes that need intermediate results visible to other containers before terminate. v1 relies on periodic background commits plus a final commit on exit.

CLI management

modal volume create my-vol
modal volume ls my-vol /
modal volume put my-vol local/file remote/path
modal volume get my-vol remote/path local/file

Key characteristics

  • Distributed: accessible from any region
  • Optimized for write-once, read-many
  • Built-in caching and chunking
  • Multiple Functions can mount the same Volume
  • Mount options (read_only, sub_path) apply per container association

Model Weight Pattern

volume = modal.Volume.from_name("model-weights", create_if_missing=True)
MODEL_DIR = "/models"

@app.function(volumes={MODEL_DIR: volume})
def download_model(model_id):
    from huggingface_hub import snapshot_download
    snapshot_download(model_id, local_dir=f"{MODEL_DIR}/{model_id}")

@app.cls(gpu="A100", volumes={MODEL_DIR: volume.with_mount_options(read_only=True)})
class Model:
    @modal.enter()
    def load(self):
        self.model = load_from_disk(f"{MODEL_DIR}/my-model")

    @modal.method()
    def predict(self, prompt):
        return self.model(prompt)

CloudBucketMount (S3/GCS/R2)

@app.function(
    volumes={
        "/bucket": modal.CloudBucketMount(
            "my-bucket",
            secret=modal.Secret.from_name("aws-creds")
        )
    }
)
def read_data():
    import os
    files = os.listdir("/bucket")

Supports AWS S3, Google Cloud Storage, and Cloudflare R2.

Go SDK

mc, _ := modal.NewClient()

volume, _ := mc.Volumes.FromName(ctx, "my-data", &modal.VolumeFromNameParams{
    CreateIfMissing: true,
})

app, _ := mc.Apps.FromName(ctx, "my-app", &modal.AppFromNameParams{CreateIfMissing: true})
image := mc.Images.FromRegistry("alpine:3.21", nil)

readOnly := true
roVol := volume.WithMountOptions(&modal.VolumeMountOptions{ReadOnly: &readOnly})

sb, _ := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{
    Volumes: map[string]*modal.Volume{"/data": volume, "/ro": roVol},
})

p, _ := sb.Exec(ctx, []string{"sh", "-c", "echo 'hello' > /data/msg.txt"}, nil)
p.Wait(ctx)

tmpVol, _ := mc.Volumes.Ephemeral(ctx, nil)
defer tmpVol.CloseEphemeral()

TypeScript SDK

import { ModalClient } from "modal";
const modal = new ModalClient();

const volume = await modal.volumes.fromName("my-data", { createIfMissing: true });

const app = await modal.apps.fromName("my-app", { createIfMissing: true });
const image = modal.images.fromRegistry("python:3.13");

const sb = await modal.sandboxes.create(app, image, {
    volumes: {
        "/data": volume,
        "/user": volume.withMountOptions({ subPath: "/users/u123" }),
        "/ro": volume.withMountOptions({ readOnly: true }),
    },
});

const proc = await sb.exec(["sh", "-c", "echo 'hello' > /data/msg.txt"]);
await proc.wait();

const tmpVol = await modal.volumes.ephemeral();
tmpVol.closeEphemeral();

S3 Gateway Endpoints

When running on AWS, Modal automatically uses S3 Gateway endpoints for zero-cost data transfer. No configuration needed.

Best practices:

  • Use region-specific or global S3 endpoints (not cross-region)
  • Schedule Modal Functions in the same region as your S3 bucket with region=
  • Inter-region traffic will incur charges

Data Transfer Patterns

Upload local data to Volume

volume = modal.Volume.from_name("my-data", create_if_missing=True)

@app.function(volumes={"/data": volume})
def upload(local_path: str):
    import shutil
    shutil.copy(local_path, "/data/")

Batch upload via CLI

modal volume put my-vol ./local-dir/ /remote-dir/ --force

Symptom Triage

"FileNotFoundError on Volume"

  • Check the Volume name matches: modal volume ls vol-name /
  • Ensure create_if_missing=True on first use
  • Verify the mount path in volumes= dict
  • If using sub_path, confirm the subdirectory path

"Slow model loading"

  • Store weights in a Volume (not downloaded at boot time)
  • Use @modal.enter() to load once per container
  • Pre-download to Volume via a separate Function
  • Mount read-only for inference: with_mount_options(read_only=True)

"CloudBucketMount errors"

  • Verify IAM permissions (read, write, list)
  • Check AWS_REGION matches bucket region
  • See references/cloud-bucket-mounts.md for troubleshooting

Reference Map

  • references/volumes.md — Volume lifecycle, mount options, CLI, batch upload, v2 sync
  • references/cloud-bucket-mounts.md — S3, GCS, R2 mounting, OIDC, key_prefix
  • references/model-weights.md — model weight storage patterns and optimization

Guardrails

  • Volumes are optimized for write-once, read-many — not a general-purpose filesystem
  • Use volume.reload() / sb.reload_volumes() in long-running containers to see updates
  • On Volumes v2, use sync /mountpoint inside Sandboxes for explicit commits
  • CloudBucketMount inherits mountpoint-s3 limitations
  • Don't store secrets in Volumes — use Modal Secrets instead
  • NetworkFileSystem is deprecated — use Volumes

Key References

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.