engine9 exports

An export is a self-contained snapshot of one account’s warehouse data; a receiver gets files, not a live database. Typical contents are warehouse table Parquet, idv1 file copies with each store’s metadata.json, and optional segment definition export CSVs. Use this skill to inspect, consume, join, or validate those files; use the export-building documentation when creating or running a package.

Quick reference

Rule: inventory.json5 is authoritative. Do not infer artifact locations from the default directory structure; use its paths and relative_path values.

File format

Default root: {store_path}/{account_id}/exports/{export_id}/{date}/

--export_dir overrides that root and is the only destination: a local path, the token temp, or an s3:// / r2:// / gs:// / gdrive:// URI. temp resolves to {os.tmpdir()}/{account_id}/exports/{export_id} and is not copied to the account store. Use {{account_id}} / {{export_id}} / {{date}} in a path when you want a per-run folder under a customer bucket; otherwise a path is used as-is. The account store is not also written. --credentials (or account settings.file_credentials) is a bootstrap path/URI to the destination key file; it is not part of the package.

Start with inventory.json5. Bundle export writes it last. It is the catalog of what was written; its presence means the run finished. A definition may relocate artifacts with relative_path; trust the inventory, not assumed paths.

Not included

• Raw CSV/JSON from the source input store (even when metadata.json lists them)
• Monthly warehouse statistics — those live on a standalone inventory run, not in the export’s inventory.json5
• The setting table (credentials / config)

inventory.json5 and the export result both carry export_dir (the export root). Strip that prefix from any absolute filename to recover the path under the root.

inventory.json5

Bundle exports write this at the export root (format_version 2) after artifacts. It is the catalog of what was written, not monthly statistics.

directories[].files is the file list (name, filename, records, source_directory), not a count.

Use it to confirm every expected parquet exists, to recover relative paths, and to see why a table or store was omitted.

Concepts

Warehouse tables

Table parquet is a dump of the named warehouse table at export time. Columns match the live table; DESCRIBE / parquet schema is the authority. Bundles choose a subset — a typical dump includes people, messages, source codes, and transactions, not every warehouse table.

One person may have many emails, phones, remotes, and hashes.

Rule: Do not treat email as a person key; resolve it to person_id.

Hash-only accounts typically ship person_hash_email / person_hash_phone and omit plaintext contact tables from the default export when utilities/limited-pii exclude_pii is set (an explicit tables list is an operator override).

Rule: Say transaction, never donation. Revenue questions use transaction and summary tables, not timeline TRANSACTION_* rows.

Rule: Say entry, never event. A timeline or idv1 row is an entry.

Input stores: metadata.json + idv1

Each selected input is one inbound stream — a message, form, ad, SQL extract, or other named source. The export copies:

1. Every .idv1.parquet in that store
2. That store’s metadata.json

Default layout: {input_type}/{input_id}/{filename}. Unknown input_type is stored as timeline. The folder name {input_id} is the same UUID as input.id and as timeline.input_id / transaction.input_id.

metadata.json

Snake_case on disk. Older files may still use camelCase (inputId, pluginId, entryTypes); prefer snake_case.

{
  "input_id": "c8ec58a6-a9d7-5277-a12a-030cc01d037f",
  "plugin_id": "a1b2c3d4-0000-4000-8000-000000000001",
  "input_type": "message",
  "remote_input_id": "msg-123",
  "remote_input_name": "April Appeal",
  "min_timeline_ts": "2024-06-01T10:00:00.000Z",
  "max_timeline_ts": "2024-12-31T23:59:59.000Z",
  "records": 152340,
  "distinct_people": 89000,
  "distinct_records": 152340,
  "files": [
    { "filename": "email-sends.idv1.parquet", "records": 152340 }
  ],
  "entry_types": {
    "EMAIL_SEND": {
      "min_timeline_ts": "2024-06-01T10:00:00.000Z",
      "max_timeline_ts": "2024-12-31T23:59:59.000Z",
      "records": 152340,
      "distinct_people": 89000,
      "distinct_records": 152340
    }
  }
}

entry_types names are exact (EMAIL_OPEN, not EMAIL_*). A bundle can select a store by that map, but message stores selected by input_type and channel may have an empty entry_types. Count entry_type_id in the parquet to see which activity kinds it holds.

records in metadata can be stale. Prefer parquet COUNT(*) when the count matters.

.idv1.parquet rows

These are Timeline ID files: already resolved to person_id and a stable entry id. They are ready to load or query. Shape: inputs/timeline.

Extra columns (URL, amount, user agent, …) are plugin detail fields. They vary by file.

Rule: A click is not automatically an open. Filter on entry_type_id; do not infer one type from another.

Entry types

Parquet stores entry_type_id (integer). metadata.json entry_types uses the string name. Full catalog: e9-timeline.

SOURCE_CODE_OVERRIDE (0) is a dictionary override marker, not a person action.

Email-oriented bundles typically ship stores whose entry_types include EMAIL_*. SMS-oriented bundles ship SMS_*.

Workflow

Join the pieces

plugin.id  =  input.plugin_id
input.id   =  folder {input_id}
           =  metadata.json input_id
           =  timeline.input_id / transaction.input_id
           =  person_remote.source_input_id   (for remotes written by that stream)

person.id  =  person_email.person_id
           =  person_phone.person_id
           =  person_remote.person_id
           =  transaction.person_id
           =  idv1 / timeline.person_id

source_code_dictionary.source_code_id  =  transaction.source_code_id
                                       =  idv1 / timeline.source_code_id

Rule: Plugin scope for remotes is not a column on person_remote. Join person_remote.source_input_id → input.id → input.plugin_id.

A message input’s input.id is the handle for that send. Opens and clicks for that send live in stores whose metadata.json input_id (or entry_types) points at that activity; they share person_id with person / transaction.

Read segment definition export CSVs

A named search export is one CSV plus a sidecar:

• {export_dir}/{export_id}.{export_name}.export.csv (or .export.csv.gz)
• {export_dir}/{export_id}.{export_name}.export.csv.metadata.json5

export_dir defaults to {store_path}/{account_id}/exports/{export_id}/{date} when omitted. The CSV is written in that directory.

Columns are whatever the search and its row transforms selected — not a fixed warehouse schema.

The sidecar records how the file was built:

Examples

Parquet is the native format. For a full walkthrough that loads the export into one DuckDB file ({export_root}/engine9.duckdb) and builds participant, opt-out, and transaction/click lists, see duckdb.md. Ad-hoc reads straight from the files:

-- Catalog
-- (open inventory.json5 next to the files)

SELECT id, given_name, family_name
FROM read_parquet('tables/person.parquet')
LIMIT 20;

SELECT id, ts, person_id, amount, source_code_id
FROM read_parquet('tables/transaction.parquet')
WHERE person_id = 12345
ORDER BY ts DESC
LIMIT 20;

-- One input store (entry_type_id 42 = EMAIL_OPEN)
SELECT id, ts, person_id, entry_type_id
FROM read_parquet('timeline/*/opens.idv1.parquet')
WHERE entry_type_id = 42
LIMIT 20;

Resolve email → person_id via person_email, then join other files on person_id.

Rule: Do not treat timeline id as a person key.

Troubleshooting

Start with inventory.json5: if it is missing the run did not finish. Verify expected artifacts, inspect skipped_tables and skipped_files, and compare listed counts with Parquet counts when metadata may be stale. For creation, execution, and deeper export diagnostics, use the export-building guide.

Related documentation

• Build and debug exports
• Warehouse inventory
• Person identity
• Plugin-scoped remote identities
• Timeline entries and entry types
• Timeline input file shapes
• Producing input files
• Source codes and attribution