Lexicon System

Overview

Lexicons define the schema for AT Protocol records. This project has two namespaces:

• pub.leaflet.* - Leaflet-specific lexicons (documents, publications, blocks, etc.)
• site.standard.* - Standard site lexicons for interoperability

The lexicons are defined as TypeScript in lexicons/src/, built to JSON in lexicons/pub/leaflet/ and lexicons/site/standard/, and TypeScript types are generated in lexicons/api/.

Key Files

• lexicons/src/*.ts - Source definitions for pub.leaflet.* lexicons
• lexicons/site/standard/**/*.json - JSON definitions for site.standard.* lexicons (manually maintained)
• lexicons/build.ts - Builds TypeScript sources to JSON
• lexicons/api/ - Generated TypeScript types and client
• package.json - Contains lexgen script

Running Lexicon Generation

npm run lexgen

This runs:

1. tsx ./lexicons/build.ts - Builds pub.leaflet.* JSON from TypeScript
2. lex gen-api - Generates TypeScript types from all JSON lexicons
3. tsx ./lexicons/fix-extensions.ts - Fixes import extensions

Adding a New pub.leaflet Lexicon

1. Create the Source Definition

Create a file in lexicons/src/ (e.g., lexicons/src/myLexicon.ts):

import { LexiconDoc } from "@atproto/lexicon";

export const PubLeafletMyLexicon: LexiconDoc = {
  lexicon: 1,
  id: "pub.leaflet.myLexicon",
  defs: {
    main: {
      type: "record",  // or "object" for non-record types
      key: "tid",
      record: {
        type: "object",
        required: ["field1"],
        properties: {
          field1: { type: "string", maxLength: 1000 },
          field2: { type: "integer", minimum: 0 },
          optionalRef: { type: "ref", ref: "other.lexicon#def" },
        },
      },
    },
    // Additional defs for sub-objects
    subType: {
      type: "object",
      properties: {
        nested: { type: "string" },
      },
    },
  },
};

2. Add to Build

Update lexicons/build.ts:

import { PubLeafletMyLexicon } from "./src/myLexicon";

const lexicons = [
  // ... existing lexicons
  PubLeafletMyLexicon,
];

3. Update lexgen Command (if needed)

If your lexicon is at the top level of pub/leaflet/ (not in a subdirectory), add it to the lexgen script in package.json:

"lexgen": "tsx ./lexicons/build.ts && lex gen-api ./lexicons/api ./lexicons/pub/leaflet/document.json ./lexicons/pub/leaflet/myLexicon.json ./lexicons/pub/leaflet/*/* ..."

Note: Files in subdirectories (pub/leaflet/*/*) are automatically included.

4. Add to authFullPermissions (for record types)

If your lexicon is a record type that users should be able to create/update/delete, add it to the authFullPermissions permission set in lexicons/src/authFullPermissions.ts:

import { PubLeafletMyLexicon } from "./myLexicon";

// In the permissions collection array:
collection: [
  // ... existing lexicons
  PubLeafletMyLexicon.id,
],

5. Regenerate Types

npm run lexgen

6. Use the Generated Types

import { PubLeafletMyLexicon } from "lexicons/api";

// Type for the record
type MyRecord = PubLeafletMyLexicon.Record;

// Validation
const result = PubLeafletMyLexicon.validateRecord(data);
if (result.success) {
  // result.value is typed
}

// Type guard
if (PubLeafletMyLexicon.isRecord(data)) {
  // data is typed as Record
}

Adding a New site.standard Lexicon

1. Create the JSON Definition

Create a file in lexicons/site/standard/ (e.g., lexicons/site/standard/myType.json):

{
  "lexicon": 1,
  "id": "site.standard.myType",
  "defs": {
    "main": {
      "type": "record",
      "key": "tid",
      "record": {
        "type": "object",
        "required": ["field1"],
        "properties": {
          "field1": {
            "type": "string",
            "maxLength": 1000
          }
        }
      }
    }
  }
}

2. Regenerate Types

npm run lexgen

The site/*/* site/*/*/* globs in the lexgen command automatically pick up new files.

Common Lexicon Patterns

Referencing Other Lexicons

// Reference another lexicon's main def
{ type: "ref", ref: "pub.leaflet.publication" }

// Reference a specific def within a lexicon
{ type: "ref", ref: "pub.leaflet.publication#theme" }

// Reference within the same lexicon
{ type: "ref", ref: "#myDef" }

Union Types

{
  type: "union",
  refs: [
    "pub.leaflet.pages.linearDocument",
    "pub.leaflet.pages.canvas",
  ],
}

// Open union (allows unknown types)
{
  type: "union",
  closed: false,  // default is true
  refs: ["pub.leaflet.content"],
}

Blob Types (for images/files)

{
  type: "blob",
  accept: ["image/*"],  // or specific types like ["image/png", "image/jpeg"]
  maxSize: 1000000,  // bytes
}

Color Types

The project has color types defined:

• pub.leaflet.theme.color#rgb / #rgba
• site.standard.theme.color#rgb / #rgba

// In lexicons/src/theme.ts
export const ColorUnion = {
  type: "union",
  refs: [
    "pub.leaflet.theme.color#rgba",
    "pub.leaflet.theme.color#rgb",
  ],
};

Normalization Between Formats

Use lexicons/src/normalize.ts to convert between pub.leaflet and site.standard formats:

import {
  normalizeDocument,
  normalizePublication,
  isLeafletDocument,
  isStandardDocument,
  getDocumentPages,
} from "lexicons/src/normalize";

// Normalize a document from either format
const normalized = normalizeDocument(record);
if (normalized) {
  // normalized is always in site.standard.document format
  console.log(normalized.title, normalized.site);

  // Get pages if content is pub.leaflet.content
  const pages = getDocumentPages(normalized);
}

// Normalize a publication
const pub = normalizePublication(record);
if (pub) {
  console.log(pub.name, pub.url);
}

Handling in Appview (Firehose Consumer)

When processing records from the firehose in appview/index.ts:

import { ids } from "lexicons/api/lexicons";
import { PubLeafletMyLexicon } from "lexicons/api";

// In filterCollections:
filterCollections: [
  ids.PubLeafletMyLexicon,
  // ...
],

// In handleEvent:
if (evt.collection === ids.PubLeafletMyLexicon) {
  if (evt.event === "create" || evt.event === "update") {
    let record = PubLeafletMyLexicon.validateRecord(evt.record);
    if (!record.success) return;

    // Store in database
    await supabase.from("my_table").upsert({
      uri: evt.uri.toString(),
      data: record.value as Json,
    });
  }
  if (evt.event === "delete") {
    await supabase.from("my_table").delete().eq("uri", evt.uri.toString());
  }
}

Publishing Lexicons

To publish lexicons to an AT Protocol PDS:

npm run publish-lexicons

This runs lexicons/publish.ts which publishes lexicons to the configured PDS.