agentleFS
all-source-osall-source-os/all-source/.claude/skills/allsource-data-access/SKILL.md

allsource-data-access

Skill4 starsChanged yesterdayApache-2.0View on GitHub

Understand AllSource storage internals: WAL format, Parquet schema, event flow, and how to read data files without a running server. Triggers on: "read parquet", "read wal", "storage layout", "event schema", "allsource data files", "where are events stored", "parquet schema". Each Parquet file contains events with this Arrow schema: With allsource-inspect (read-only, safe beside a running app): Do not read the Parquet files directly — not with pyarrow, not with duckdb, not with cat/grep over the WAL. It looks like it…

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 3 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    It knows nothing about tenancy or schema: No tenant scoping, no value-object semantics — an answer that mixes tenants reads exactly like a …

    line 72
  2. Distinctive:

    When batch reaches 10,000 or 5-second timeout → flush to Parquet

    line 170
  3. Distinctive:

    Read existing Parquet files → populate DashMap

    line 179

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

AllSource Data Access

Understand AllSource storage internals: WAL format, Parquet schema, event flow, and how to read data files without a running server.

Triggers on: "read parquet", "read wal", "storage layout", "event schema", "allsource data files", "where are events stored", "parquet schema".


Storage Layout

<data_dir>/
├── storage/                    # Parquet files (long-term columnar storage)
│   ├── events-20240115-001.parquet
│   ├── events-20240115-002.parquet
│   └── events-20240116-001.parquet
├── wal/                        # Write-Ahead Log (crash recovery)
│   ├── wal-0000000000000000.log
│   └── wal-0000000000000001.log
└── __system/                   # System metadata (tenants, config, auth)
    └── wal/
        └── wal-0000000000000000.log

Default Data Paths

PlatformAppPath
macOSLonghand~/Library/Application Support/Longhand/allsource/
LinuxLonghand~/.local/share/longhand/allsource/
Docker CoreMounted/app/data/ (via volume)
Embedded (default)In-processdata/mcp-embedded/ or configured via CORE_DATA_DIR

Parquet Schema

Each Parquet file contains events with this Arrow schema:

ColumnArrow TypeNullableDescription
event_idUtf8NoUUID v4 string
event_typeUtf8NoDot-separated type (e.g., workflow_run.started)
entity_idUtf8NoEntity identifier (e.g., workflow:abc-123)
payloadUtf8NoJSON string of event data
timestampTimestamp(Microsecond)NoEvent creation time
metadataUtf8YesOptional JSON string (correlation IDs, source info)

Parquet Configuration

  • Compression: Snappy (default)
  • Batch size: 10,000 events per flush (configurable)
  • Flush timeout: 5 seconds (partial batch written if timeout reached)
  • File naming: events-YYYYMMDD-NNN.parquet (date-partitioned, sequential)
  • High-throughput mode: 50,000 events/batch, 10-second timeout

Reading Parquet Files

With allsource-inspect (read-only, safe beside a running app):

allsource-inspect events --data-dir <path> --format json
allsource-inspect events --data-dir <path> --event-type-prefix "workflow_run." --fields definition_id --format json

Do not read the Parquet files directly — not with pyarrow, not with duckdb, not with cat/grep over the WAL. It looks like it works and gives a wrong answer:

  • It misses the WAL tail. Events that have not been folded into Parquet yet live only in the WAL. A Parquet-only read silently omits the most recent events, which are usually the ones being asked about.
  • It knows nothing about tenancy or schema. No tenant scoping, no value-object semantics — an answer that mixes tenants reads exactly like a correct one.
  • It leaves no tool behind. The next session repeats the same detour.

An agent did exactly this — duckdb over storage/**/*.parquet to find parked workflow runs — and the result was wrong for the first two reasons (#284). If the commands above cannot answer the question, extend the tool rather than going around it.

With the allsource-mcp MCP server (no extra deps):

Use quick_stats to see total events and parquet file count
Use query_events with entity_id to read specific events

WAL Format

The WAL uses JSON Lines format — one JSON object per line, each representing a WALEntry:

{"sequence":1,"wal_timestamp":"2024-01-15T10:00:00Z","event":{...},"checksum":3847291654}

WAL Entry Structure

FieldTypeDescription
sequenceu64Monotonically increasing sequence number
wal_timestampDateTime<Utc>When the entry was written to WAL
eventEventThe full event object (same fields as Parquet)
checksumu32CRC32 checksum of sequence + wal_timestamp + event.id

WAL Configuration

SettingDefaultDescription
max_file_size64 MBRotate WAL file after this size
sync_on_writetruefsync after each write (durability vs throughput)
max_wal_files10Max WAL files before cleanup
compressfalseLZ4 compression (optional)

WAL File Naming

Files are named wal-NNNNNNNNNNNNNNNN.log where N is a zero-padded sequence number.

Reading WAL Files

WAL files are plain text, one JSON object per line:

# Read WAL entries as JSON
cat wal/wal-0000000000000000.log | head -5

# Count entries
wc -l wal/wal-0000000000000000.log

# Filter by entity
grep '"entity_id":"workflow:abc-123"' wal/wal-0000000000000000.log

With allsource-inspect:

allsource-inspect wal --data-dir <path> --format json

Verifying WAL Integrity

Each entry has a CRC32 checksum. Replay verifies it and skips a mismatched entry with a Corrupted WAL entry … (checksum mismatch) warning, so run any read command with tracing on and look for that line. No CLI flag checks it on demand.


Event Flow

Ingest API/Embedded
       │
       ▼
   DashMap (in-memory concurrent map)
       │  ← 11.9μs query latency, 469K events/sec
       │
       ├──▶ WAL (append-only, fsync)
       │       │  ← crash recovery source
       │       │
       │       ▼
       │    Parquet (columnar, Snappy)
       │       ← long-term storage, efficient reads
       │
       ▼
   Query (reads from DashMap + Parquet)

Write Path

  1. Event hits DashMap (immediate availability for queries)
  2. Event appended to WAL (durability guarantee)
  3. When batch reaches 10,000 or 5-second timeout → flush to Parquet
  4. WAL entries for flushed events can be cleaned up

Read Path

  1. Query reads from in-memory DashMap first
  2. Parquet files provide historical data
  3. Combined result returned to caller

Recovery Path (on startup)

  1. Read existing Parquet files → populate DashMap
  2. Replay WAL entries not yet in Parquet → recover unflushed events
  3. System is consistent — no data loss

System Events

System metadata uses a separate WAL in __system/wal/:

DomainEvent TypesPurpose
_system.tenant.*created, updated, suspendedTenant lifecycle
_system.config.*set, deletedKey-value configuration
_system.auth.*key_provisioned, key_revokedAPI key management
_system.schema.*registered, updated, deprecatedEvent schema registry
_system.policy.*created, updated, deletedAccess policies
_system.consumer.*registered, ack_updated, deletedSubscription cursors

System events use the same WAL format but are stored in __system/ to separate metadata from user event data.


Reading Without a Running Server

The allsource-mcp and allsource-inspect tools open the data directory directly using the allsource-core embedded API. No server process needed.

use allsource_core::embedded::{Config, EmbeddedCore, Query};

let core = EmbeddedCore::open(
    Config::builder()
        .data_dir("/path/to/data")
        .read_only(true)
        .build()?
).await?;

let events = core.query(Query::new().entity_id("workflow:abc-123")).await?;
// Do not call core.shutdown() on a read-only core: it flushes storage.

This opens Parquet files and replays the WAL for reads. read_only(true) is required when anything else may own the directory. Without it the open checkpoints the recovered WAL into Parquet and truncates the WAL out from under the owning process, losing the writes it is still appending (#201).

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.