Produce engine9 input files

An input is the generic wrapper for one named stream of inbound data owned by one plugin. Almost anything inbound can be an input: a form, an email blast, an ad, a SQL table or named query, a CRM people dump, an advocacy action, a payment extract, or a mixed activity log. Producers write files plus input identity; engine9 then derives input_id, resolves people, and (for activity rows) writes Timeline ID files. Use this skill when an outside agent or plugin must emit that shape.

The common case is timeline activity: each row is an entry with a string entry_type (FORM_SUBMIT, EMAIL_SEND, EMAIL_OPEN, CRM_ORIGIN, …). Those rows are not on the warehouse timeline until people are resolved and the ID files are loaded. Call them entries, never events.

Quick reference

Rule: A submission, payment, click, or lead id is a row id (remote_entry_id), not a person id (remote_person_id).

Concepts

The warehouse handle

The warehouse input row is the handle for the stream. Timeline entries, transactions, and person_remote rows point at input.id. The on-disk input store uses the same identity.

input_type is a label, not a closed enum. Use a conventional value from the table below so stores and exports group cleanly.

Stream identity vs row identity

Rule: Do not put a submission, payment, or click id in remote_person_id. That creates one person per row.

How person_id is chosen from those match keys is e9-person-id. Plugin-scoped vendor person ids land in person_remote via source_input_id → input.id (e9-person-remote).

Timeline entries and entry_type

Most input files are person-activity rows that will become timeline entries:

Rule: Prefer the string entry_type on producer rows. Use numeric entry_type_id only after conversion via TIMELINE_ENTRY_TYPES.

Common names for producers (full catalog: e9-timeline):

Until identity succeeds, a file that only has an email is not a timeline entry yet. After InputWorker.id, the store holds Timeline ID (.idv1.parquet). Loading those files into timeline is optional and separate — large stores often remain as idv1 only (e9-inventory, timeline loading).

Row-level file shapes (Raw vs ID) are inputs/timeline.

When to split vs one extract

If the vendor has a durable object id, that object is the input. Do not dump every form, blast, or ad into one file when those ids exist.

A signup form may also appear as a signup_form message in the message catalog / global_message. That message input is the cataloged send, not the submission stream. Submission rows belong on the form input keyed by the vendor form id.

File format

Producer rows (Timeline Raw)

Emit CSV, JSONL, or parquet. Rows must not include person_id. Required and identity fields:

Typical form row:

{
  ts: "2026-03-01T15:04:05Z",
  entry_type: "FORM_SUBMIT",
  remote_input_id: "1234567890",
  remote_input_name: "Volunteer signup",
  remote_entry_id: "lead-aaa",
  email: "person@example.org",
  phone: "+15555550100",
  given_name: "Ada",
  family_name: "Lovelace",
  campaign_id: "111",
  ad_id: "222"
}

Person-extract row (input_type: person) uses remote_input_id: "remote_person" on the file, not per row. Default entry_type is CRM_ORIGIN. Include remote_person_id when the vendor has a durable constituent id.

Typical email-blast rows share one remote_input_id (the send) and differ by entry_type:

{
  ts: "2026-04-01T12:00:00Z",
  entry_type: "EMAIL_SEND",
  remote_input_id: "msg-123",
  remote_input_name: "April Appeal",
  remote_entry_id: "send-aaa",
  email: "person@example.org"
}

Opens and clicks for that blast keep the same remote_input_id and use EMAIL_OPEN / EMAIL_CLICK with their own remote_entry_id and ts.

Store metadata

Write one store per input:

{account_id}/plugins/{plugin_id}/{input_type}/{input_id.slice(0,4)}/{input_id}/
  metadata.json
  {filename}

metadata.json (snake_case):

{
  "input_id": "<uuid>",
  "plugin_id": "<plugin uuid>",
  "input_type": "form",
  "remote_input_id": "1234567890",
  "remote_input_name": "Volunteer signup"
}

Rule: input_id must equal getInputUUID(plugin_id, remote_input_id). MCP input_id computes the same value. Do not use a random UUID.

After InputWorker.id, the store also has .idv1.parquet (Timeline ID). Export copies of those stores are e9-export.

Handoff: one file per input

A producer returns a list of files (or writes them to the store above). Each item is one input:

Stamp remote_input_id on every row of a mixed stream, then split. Do not concatenate forms, blasts, or ads into one stored file.

Workflow

Map a vendor feed

1. List the durable objects (forms, messages/sends, ads, tables, or a single extract name).
2. Choose input_type and entry_type from the tables above.
3. Map identity: vendor person id → remote_person_id; otherwise email/phone; submission / send / click id → remote_entry_id.
4. Map ts from the vendor event time, not load time.
5. Copy extra fields as detail; do not overload identity columns.
6. Split so each remote_input_id is its own file, then store with metadata.json.

Produce files as an outside agent

1. Resolve plugin_id for the integration (MCP plugin_id or getPluginUUID).
2. For each durable object, set remote_input_id to the vendor object id (or a stable extract name).
3. Compute input_id = getInputUUID(plugin_id, remote_input_id).
4. Write one Timeline Raw file per input plus metadata.json.
5. Run InputWorker.id / idFiles (people + Timeline ID), then loadTimelineTables if warehouse timeline load is in scope.

Delete an input or its store files

A delete covers the input store (metadata.json, raw files, .idv1.parquet, and every other file in that input's directory) and, when the request includes it, the warehouse input row. Timeline or transaction rows that point at the input are a separate delete. They still require this same list, count, and confirmation when the request is to remove the input's files.

1. Resolve each input: account_id, plugin, input_type, input_id, and data_path.
2. List every file the delete would remove. When more than one input is in scope, say which input each file belongs to.
3. State the file count, and the input count when more than one input is in scope. Zero files is still a result: say there are none.
4. Stop. Show that list and those counts. Do not delete in that turn.
5. Delete only after a later message confirms that exact list. The original request to delete is not confirmation. A "yes" or "delete them" sent before the list was shown is not confirmation. A reply that does not refer to the listed set is not confirmation.
6. Remove the confirmed files with one @engine9/plugins/e9workers:FileWorker removeFiles call. Pass filenames (comma-delimited or an array), file_array (path strings or { filename } objects), or options_filename / optionsFilename (a JSON file of those options, including an idFiles options file). Do not schedule one remove task per file. When the confirmed set is every file in one input’s store directory, MCP input deleteFiles is that single call. It resolves the directory from input_id and does not take a path.
7. If the store changes, or the user changes which inputs are in scope, list and count again and wait for a new confirmation.

Rules

Rule: When reading an existing account input store from Cursor, use MCP input with input_id: root files, then files with path for a listed subdirectory (e.g. "batched"), then download / summary with the store-relative name (e.g. batched/foo.batch.csv). Do not use local aws s3. If MCP cannot list or download the file, stop and say so.

Rule: One named remote stream per input. If the vendor has a durable form, message, ad, or table id, that object is the input.

Rule: Call timeline records entries, never events.

Rule: Timeline Raw rows must not include person_id.

Rule: remote_person_id is a person in the vendor system. Submission, send, click, and payment ids go in remote_entry_id.

Rule: Do not set input_id by hand except via getInputUUID(plugin_id, remote_input_id).

Rule: Do not mix input_type values in one file. Split first.

Rule: FORM_SUBMIT is the form-action type. Use SIGNUP / SIGNUP_INITIAL / SIGNUP_SUBSEQUENT only for list-subscription facts, not for “someone filled this form.”

Rule: Leave source_code empty when the vendor has no code; ad ids and campaign ids are detail, not source codes.

Rule: Never delete input store files or an input row in the same turn that lists them. Show the file list and the count, then wait for a later message that confirms that exact list.

Rule: Remove a confirmed set of input store files with one FileWorker.removeFiles call. Do not schedule one remove task per file.

Rule: MCP input deleteFiles removes every file in one input’s store directory, including files inside subdirectories, then removes those subdirectories. Use it only after the user confirms that folder. It does not delete a subset of the folder, the warehouse input row, or timeline rows.

Conductor’s Inputs artifact exposes the same folder delete: the Files card’s Delete button confirms, then calls deleteFiles.

Examples

Split a mixed form stream

// Each row already has remote_input_id = vendor form id
const byRemoteInputId = new Map();
for (const row of rows) {
  const id = String(row.remote_input_id);
  if (!byRemoteInputId.has(id)) byRemoteInputId.set(id, []);
  byRemoteInputId.get(id).push(row);
}
// Write one Timeline Raw file + metadata.json per map entry

Single person extract (do not split)

{
  input_type: 'person',
  remote_input_id: 'remote_person',
  remote_input_name: 'Load People From Remote System'
}

Rows: ts, remote_person_id, email, given_name, family_name. File-level identity; rows do not need remote_input_id.

SQL table extract

{
  input_type: 'sql',
  remote_input_id: 'public.constituents',
  remote_input_name: 'Constituents table'
}

Use a stable table or extract name. Do not hash the SQL text as remote_input_id unless that string is itself durable — a changed query would mint a new input_id.

Lead-gen form with no vendor person id (Facebook)

When the vendor issues a form id and a per-submission id, but no constituent id:

If a row has neither email nor phone, fall back to remote_person_id = submission id so the row can still resolve. That fallback is last-resort only.

Do not key the input by ad, ad set, or campaign when the object people filled in is the form. The ad can be its own message input for spend and catalog; submissions still belong on the form.

Troubleshooting

Related documentation

• Timeline Raw vs Timeline ID
• Timeline model and entry types
• Timeline loading
• Person identity
• Person remotes
• Transaction mapping
• Export input stores
• Inventory (input-store vs warehouse)
• MCP plugin_id / input_id
• @engine9/input-tools getPluginUUID, getInputUUID, TIMELINE_ENTRY_TYPES