engine9 person identity (person_id)

The engine9 person identity framework selects a durable person_id from inbound email, phone, remote, delegate, hash, and existing-person identifiers. Use this skill to understand identity stores, trace the inbound people pipeline, or diagnose deduplication and duplicate-person behavior. Downstream tables such as timeline, transaction, and segments consume this identity rather than creating their own.

Quick reference

Concepts

person.id is person_id: a numeric autoincrement on the person table. Every other person-related table stores that integer as person_id.

Names (given_name, family_name) and addresses are attributes, not match keys. Two people can share a name. Identity matching uses only the identifier types below.

Rule: One person may have many emails and many phones. Do not apply the legacy one-email-equals-one-person assumption to current identity.

Rule: This skill is exclusively about person_id; do not apply these rules to source_code_id, message IDs, timeline entry IDs, or other identifiers.

File format

Two layers: a lookup from match key → person_id, and attribute rows hanging off that person_id.

Identity lookup (match key → person_id)

This is the core lookup. The current optimization is one compact table per identifier type:

Compact schema: value (16-byte hash, primary key) + person_id. No plaintext, no source_input_id. First insert wins (INSERT OR IGNORE / INSERT IGNORE). Tables are created on demand when that id_type is first written.

value is not email_hash_v1 itself. Extract already hashed the email/phone to a hex id_value; the compact store hashes that string again: first 16 bytes of SHA-256(id_value UTF-8). id_type is the table name, not mixed into the hash. Rare collisions are accepted. For remote_person_id the id_value is the plugin-prefixed string, then the same 16-byte truncate.

person_identifier is fundamentally similar but lower performing, so it is being deprecated, but can still be used. Same semantics (id_type + id_value string → person_id), plus source_input_id. MySQL accounts still default to it until migrated. SQLite / D1 default to compact person_id_* and leave person_identifier empty.

Do not assume person_identifier is populated. Check identifier_store_kind (core plugin setting) or which tables have rows. See Identifier stores.

Person attributes (many per person)

Written after person_id is assigned. Unique keys allow multiple emails/phones/remotes on one person.

person_hash_email / person_hash_phone (optional @engine9/schemas/person_hash) store hashes without plaintext. They are not in the default inbound chain.

Downstream consumers

timeline, transaction, person_segment, and similar tables store person_id after the people pipeline has run. They are not identity sources.

Identifier types

Inbound rows do not match on raw email/phone strings in the lookup tables. Extract transforms push { type, value, path } onto row.identifiers. assignPersonIds matches those.

appendPersonId lowercases and strips accents on every identifier value before lookup (NFD + combining marks). Matching is case- and accent-insensitive.

Rule: Never store the blank SHA-256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855. Extract skips it, and assignPersonIds throws if it appears.

Code: @engine9/schemas/person_email|person_phone|person_remote/transforms/inbound/extract_identifiers.js, @engine9/core/lib/id/index.js.

Workflow

Inbound pipeline

Shared by server PersonWorker.loadPeople (streams/files) and core PersonWorker.processPeople (in-memory). Woven by buildInboundTransforms in @engine9/core/lib/peoplePipeline/getInboundTransforms.js from the plugins installed in the account — core has no list of person plugins. Each people schema plugin declares metadata.inbound = { slot: [transformKey] }; the snapshot is stored on plugin.transforms at install.

Slots, in order (assign is core-only):

beforeAll  extra
normalize  person:normalizeFieldNames                       # lowercase keys, strip punctuation
id         person_email / person_phone / person_remote :id   # identifiers[]; person_hash:id when installed
id         core  person.extractDelegateIdentifiers           # core client only
assign     core  appendInputId → appendPersonId → appendEntryTypeId → source code steps
upsert     person / person_address / person_email / person_phone / person_remote :upsert
           (+ person_custom per table_prefix, person_hash when installed)
afterAll   extra

Woven steps in a slot run in path order, then any extraTransforms for that slot. omitTransforms drops a path for one job. A limited-pii account has no person_email / person_phone / person_address rows, so those steps do not exist there.

See the chain for an account: personWorker.getInboundTransforms({ pluginId, describe: true }) prints one slot source path line per step (also logged at debug by loadPeople / processPeople).

doNotUpsert / do_not_upsert: run extracts + lookup only. Existing people resolve; unknown rows stay without person_id; no inserts.

Identity assignment is blocking (lookups and inserts must serialize across threads). InputWorker.id calls loadPeople so Timeline ID files already have person_id before they hit timeline.

Inspect store mode:

SELECT value FROM setting
WHERE plugin_id = '00000000-0000-4000-a000-000000000001'
  AND name = 'identifier_store_kind';
-- compact | legacy | missing (then SQLite/D1 → compact, MySQL → person_identifier)

How assignPersonIds works

@engine9/core/lib/id/index.js. One batch, one store.

1. Refuse blank SHA-256 identifier values.
2. Within-batch person_id: if any row already has person_id (caller-supplied), copy it onto other rows that share an identifier.
3. Within-batch temp_id: rows that share any identifier get the same temp_id, so they become one person even if the store has never seen them.
4. Store lookup of every { id_type, id_value }.
5. Apply hits without overwriting: remote_person_id matches first, then other types. If person_id is already set, a later hit is ignored. First assignment wins for that row.
6. Insert person rows for unmatched temp_ids (unless doNotUpsert). Optional created_at from date_created / remote_date_created / frakture_date_created / created_at.
7. Insert identifier mappings that the store did not already have. Existing keys are skipped (first-wins, never remapped).

Consequences:

• One inbound row with email + phone + remote id attaches all three keys to the same person_id. A later row matching any of those keys resolves to that person (unless a higher-priority already-set person_id / remote_person_id won first).
• Rule: Identifier mappings are immutable and first-wins. A second person created via a new remote_person_id does not steal an email or phone key that already points at someone else. person_email uniqueness is (email, person_id), so plaintext email can appear on more than one person; the lookup key still points at the first person_id.
• Caller-supplied person_id is respected and seeds the store for new keys.

After assignment, upserts write attributes. Email upsert looks up existing person_email by address; same person+email updates (keeps original source_input_id); new pair inserts. Default new subscription is Subscribed; explicit unsubscribe (or EMAIL_UNSUBSCRIBE) updates all person_email rows with that address. Phone upsert is analogous (min 8 digits; default SMS Not Subscribed).

Select an identifier store

createDefaultIdentifierStore (@engine9/core/lib/id/storeKind.js):

1. Durable Object binding (worker.personIds / PERSON_IDS) always wins (Cloudflare).
2. Else setting identifier_store_kind on the core plugin (00000000-0000-4000-a000-000000000001): compact or legacy (person_identifier accepted as legacy).
3. Else dialect default: SQLite / D1 → compact; MySQL → person_identifier.

findPersonIdentifierHashCollisions scans the legacy table for 16-byte hash collisions before migrate. PersonWorker.populatePersonRemoteFromIdentifiers backfills person_remote from person_identifier rows with id_type = 'remote_person_id' (strips the {pluginId}. prefix into person_remote.remote_person_id).

Migrate: PersonWorker.migratePersonIdentifiersToCompact pages person_identifier, writes compact tables (first-wins), renames person_identifier to person_identifier_legacy_<timestamp>, sets identifier_store_kind=compact. dry_run counts only.

Rules

Old identity model (person_metadata, person_id_int, timeline_v3, transaction_metadata)

person_metadata, person_id_int, transaction_metadata, and timeline_v3 all use an OLD identity model that did not account for multiple emails or phones per person.

In that model the person key was a single integer person_id_int, and person_metadata.person_id was often the email string (legacy loaders filter meta.person_id like '%@%'). person_metadata generates person_id_int from that person_id string. person_id_int is not current person.id — do not join them. One email ≈ one person. Phones were not first-class match keys. timeline_v3 joined person_metadata using (person_id_int). Inspect that log through timeline_v3_summary, whose type column is entry_type_label (not current entry_type). transaction_metadata sat in the same integer-person space.

PersonWorker.unwindPersonMetadata is the bridge: copy person_metadata.person_id_int into person.id, then delete those legacy metadata rows. LocalDatabasePersonWorker.importFromTimelineV3 still reads timeline_v3 joined to person_metadata and feeds emails into the current people pipeline.

Rule: When you see person_id_int or person_metadata.person_id as an email, you are in the old model. Do not join it to current person_email as if it were person.id; resolve through the current lookup (person_id_* or person_identifier) and attribute tables.

Legacy source_code_summary.origin_* rollups were computed in that old identity world. Current models write {prefix}_* tables with bigint person_id — e9-model.

Troubleshooting

Prefer attribute tables (plaintext) over compact BLOBs. DESCRIBE first; filter to one example; LIMIT.

-- Email → person_id (current model)
SELECT person_id, email, subscription_status, email_hash_v1
FROM person_email
WHERE LOWER(email) = 'alice@example.com'
LIMIT 20;

-- Phone → person_id (phones stored normalized, often +1…)
SELECT person_id, phone, phone_type, sms_status, phone_hash_v1
FROM person_phone
WHERE phone IN ('+12025550143', '2025550143')
LIMIT 20;

-- Remote CRM id → person_id (value is usually NOT plugin-prefixed in this table)
SELECT pr.person_id, pr.remote_person_id, pr.source_input_id, i.plugin_id
FROM person_remote pr
JOIN input i ON i.id = pr.source_input_id
WHERE LOWER(pr.remote_person_id) = '123425385'
LIMIT 20;

-- Canonical person
SELECT id, given_name, family_name, created_at FROM person WHERE id = ?;

On legacy (person_identifier) accounts:

SELECT id_type, id_value, person_id, source_input_id
FROM person_identifier
WHERE id_type IN ('email_hash_v1', 'phone_hash_v1', 'remote_person_id', 'delegate')
  AND person_id = ?
LIMIT 50;

On compact accounts, person_identifier may be empty or renamed. Lookup tables are person_id_email_hash_v1, person_id_phone_hash_v1, person_id_remote_person_id. The value column is a 16-byte hash of the identifier id_value, not the email string — use person_email / person_phone / person_remote for human debugging.

Duplicate-people symptoms: same email on two person_ids (lookup first-wins; attribute table allows both), missing remote_person_id extract (pluginId required), short phone dropped (< 8 digits), blank hash refused, or still reading person_metadata / timeline_v3 as if they were current.

Related documentation