Design workflows with volumes

Use this skill before creating or changing steps that use named volumes or filesystem-backed state. Discover the workflow tools available in the current interface. For an existing workflow, inspect its step graph, source, volume declarations, schedules, recent runs, and volume usage before changing the design. Separate measurements from assumptions.

Users describe the behavior they need, not a storage primitive. Choose file layouts, database formats, volume names, and mount options yourself. Ask only product questions that change the design, such as whether data must survive later Live runs, how fresh an answer must be, or how long records must be retained. Explain the resulting behavior in product terms.

Core model

A volume is a named POSIX directory mounted at /storage/<name>. In Live, every workflow in a space that declares the same name mounts the same committed files. Draft branches have isolated files. scope=run gives one workflow run its own volume, shared by steps in that run that declare the same name and scope. A :ro mount is read-only.

Writable mounts do not serialize executions. Two runs of the same step, different steps, or different Live workflows in the same space can write the same named volume at the same time. Run-scoped volumes are shared only by steps in one run, so separate runs do not write the same run-scoped volume.

A step writes into a private view while it runs. Other steps see the last committed version, never its partial writes. A failed step discards its changes. A successful step publishes each volume separately, so files that must be committed together belong in one volume.

Overlapping runs that write different paths both publish. When overlapping runs change the same file, the run that publishes second fails with a volume conflict that names the volume and the file, and none of its changes to that volume are saved. An embedded database such as SQLite spreads one update across its main file and sidecar files like -wal, so overlapping writers to one database can corrupt it without any conflict. A routed step’s response reaches the caller before its volume changes publish, so a request step that rewrites a shared file can report success and then lose the write. Design every writer so overlapping runs never change the same file, and have request steps only create new, uniquely named files.

Follow the secret-storage prohibition. Use stdin and stdout for small handoffs between adjacent steps. Use volumes when steps need filesystem paths, many related files, random access, or data that survives a step.

Shape the request path

Design the common request around the answer its caller needs. When possible, one entry step validates the request, reads only that subject’s current view, writes any necessary update to a unique path, and responds. Mount only the volumes it uses. A direct response should not also start downstream steps that merely decide there is no work. Keep connector calls, large scans, aggregation, and database updates off a frequent request path when the product permits a short delay.

For concurrent requests, give each delivery, device, customer, or job a distinct record path. Use stable subject keys for direct lookups, unique IDs for new immutable records, and idempotency keys that survive retries. Do not append from many runs to one shared file or update one shared embedded database from request steps. A small read-only file per subject often suffices for point lookups. Use an embedded database when its queries or transactions justify it. Keep lookup work proportional to the subject being requested, not to all retained records.

request → validate → read views/<subject-id>.json
                   → write pending/<bucket>/<unique-id>.json → respond

scheduled work → apply bounded pending batch → refresh affected views → clean up applied input

Only create downstream executions for work that must follow this request. A scheduled or otherwise low-concurrency step can update derived state separately when a brief delay is acceptable. Preserve the required authentication, response signing, freshness, and failure behavior. If the caller needs an immediate result that cannot be computed from current and pending state, keep the necessary work on its response path.

Give shared state an owner

Default to independent files under a writable volume. Runs writing distinct paths can overlap without serializing the whole step. Use read-only mounts for steps that only read.

VOLUME ["incoming"]
VOLUME ["views:ro"]

If an embedded database, report, or index must be updated, give each mutable file group one scheduled or otherwise low-concurrency writer. This applies to SQLite, DuckDB, and similar file-backed databases. Request steps may query a published database in read-only mode, but should write independent pending records instead of changing it. Ensure writer runs cannot overlap on the same database files. A short timeout alone does not guarantee this when starts are late or retries occur. If one writer cannot keep up, partition by a stable key into independently owned files, or use a store built for concurrent updates.

Keep a database and its sidecars together. If pending records, derived state, and affected views are in the same volume, one successful writer step can publish their changes together. If input and derived state are in different volumes, their publications are separate. Record an idempotency key with the durable derived update and confirm it in a later run before deleting the input. Recheck state when applying delayed work and preserve rejected work or its outcome when the product needs it.

Bound background work and storage

Move aggregation, database updates, view refreshes, and cleanup out of frequent entry steps when a short delay is acceptable. Give each run a fixed record and time budget, and prevent overlapping runs that update the same files. When a tick only triggers a scan of durable pending records, skip obsolete ticks instead of replaying them after a queue delay. Apply the bound before reading, parsing, or sorting a growing directory. Use direct paths or bounded time- or key-based buckets so selecting the next batch stays cheap. A schedule must drain sustained input faster than records arrive. Otherwise, its queue and volume usage grow even when each run succeeds.

Remove or compact applied files according to the required retention period. Keep necessary outcomes and idempotency information, but avoid retaining full payloads twice without a product need. Estimate retained logical bytes from input size, arrival rate, derived files, database sidecars, and retention time. All volumes in a space share the space’s storage limit, and new runs in the space are refused once it is reached, so inspect actual volume usage when it may affect the design. Do not silently drop required data to save space.

For source data, inspect metadata and filter before downloading when possible. Fetch only needed files, and use pagination, change feeds, ranges, streams, or batches for larger inputs. If the complete collection is required, plan ingestion, retention, and recovery from interruption rather than relying on one large download.

Check representative requests, duplicates, and sustained traffic. Inspect request latency, waiting executions, background run duration, oldest pending age, pending files and bytes, and volume usage. A successful scheduled run is not enough. Pending work must repeatedly drain under the expected arrival rate.

Lifetime and draft storage

Use VOLUME ["state"] when files should survive and be visible to later Live runs. A read-only step declares VOLUME ["state:ro"]. Every workflow in the space that declares state shares its Live files.

Draft branches start with isolated files for the same name. Whether a draft starts empty, copies Live’s data, or copies the data of the branch it was forked from is decided per draft. When the workflow context describes that decision as pending, follow its instructions and record the answer with seedBranchStorage before running any step that mounts the volume; a copy is a snapshot, so later changes in the draft never affect Live or the source branch. The user can reset a draft’s stored data from the storage browser. Draft files are discarded when the draft is pushed live; only when the workflow has no Live storage yet is the user offered to seed Live with the draft’s data.

Use VOLUME ["work:scope=run"] for files needed only within one workflow run. A reader declares VOLUME ["work:scope=run,ro"]. Every step sharing that run-scoped volume must declare scope=run. VOLUME ["work:ro"] names a different, persistent volume. Scope changes lifetime, not whether writers overlap within a run.

Other useful shapes

• Independent uploads, generated artifacts, and cache entries can use one directory per job or source key. Readers can use those paths directly when no cross-job query is needed.
• SQLite suits indexed mutable state with one writer per database file. DuckDB suits batch analysis over tabular data. Other embedded databases can also work when their read and write behavior fits the workflow. Keep each database and its working files under one writer, and use a remote source when it can answer the required query without a local copy. When a SQLite database grows to hundreds of megabytes read by scattered lookups, PRAGMA page_size = 65536, set before the first table is created or applied once with VACUUM, cuts the reads a lookup costs; a small database or one scanned once per run does best with the defaults.
• Separate volumes when ownership, lifetime, or retention differs. Do not split files that require one atomic publication across volumes. More volumes and mounts also add work to every step that uses them.
• Read-only APIs and dashboards should mount read-only views. If a shared summary is needed, a bounded writer can update it from completed records.

How volume I/O behaves

Volumes are network-backed and read on demand in 64 KiB blocks. Opening a file costs nothing, an unread block costs a network round trip, and data a step has read stays cached while its memory allows, so scattered small reads across a large file are the expensive shape and sequential reads are cheap. A successful step publishes only the blocks it changed; fsync adds nothing, and a file deleted before the step ends costs nothing. /tmp and the working directory are memory-backed, 512 MiB each by default, inside the step’s memory limit, which the CPU count and memory a process reads from the system do not reflect.

This matters only for a step whose time is dominated by one large file. For that step, prefer fewer and larger reads over scattered small ones, stream a scan rather than load the file whole, group writes so each block is dirtied once, and leave rereads to the file cache rather than a large cache inside the process, which counts toward the step’s memory. Keep a change only when a representative run before and after justifies it.

When explaining the finished workflow, describe what the caller sees and when background work becomes visible. Mention storage details only when they affect the product’s behavior or the user asks.