agentleFS
ea-toolkitea-toolkit/architecture-catalog/.claude/skills/crawl-data/SKILL.md

crawl-data

Skill38 starsChanged 9 months agoMITView on GitHub

Scan a target codebase for data model definitions (SQL schemas, ORM models, TypeScript interfaces, Pydantic models) and propose data_concept, data_aggregate, and data_entity registry entries. Presents findings for review before writing files.

The file

We split this skill into the rules it gives an agent

11 rules. About 0 are common, 7 are distinctive, and 4 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈7

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    If no path is provided, ask the user which directory to scan.

    line 19
  2. Very specific:

    Read models/registry-mapping.yaml to find:

    line 25
  3. Distinctive:

    Never hardcode paths: Always derive from the YAML.

    line 31
  4. Distinctive:

    Search the target directory for data model files.

    line 35
  5. Very specific:

    Look in: /models/, /entities/, /schema/

    line 49
  6. Distinctive:

    If unclear, each standalone table/model becomes its own aggregate

    line 89
  7. Distinctive:

    Before proposing entries, check existing registry entries:

    line 97
  8. Very specific:

    Read the _template.md for the target type

    line 136
  9. Distinctive:

    Write to the correct registry folder

    line 142
  10. Very specific:

    Always propose as status: draft — never auto-promote to active

    line 171
  11. Distinctive:

    If a model has no clear parent, make it both a data_concept and data_aggregate (flat hierarchy)

    line 173

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Crawl Data — Discover and Register Data Models

Scan a codebase directory for data model definitions and propose registry entries.

Arguments

  • $1 — Path to scan (required). Absolute or relative path to the codebase to crawl.
  • --domain <name> — Domain to assign discovered models to (optional, will ask if omitted).
  • --write — Write proposed entries to registry immediately (default: preview only).

If no path is provided, ask the user which directory to scan.

Workflow

1. Discover Type-to-Folder Mapping

Read models/registry-mapping.yaml to find:

  • The folder path for data_concept entries
  • The folder path for data_aggregate entries
  • The folder path for data_entity entries
  • The _template.md in each folder for frontmatter structure

Never hardcode paths. Always derive from the YAML.

2. Scan for Data Model Definitions

Search the target directory for data model files. Use these detection patterns:

SQL Schema files:

  • Glob: **/*.sql, **/migrations/**/*.sql, **/schema/**/*.sql
  • Content match: CREATE TABLE, ALTER TABLE
  • Extract: table name, column names/types, constraints, foreign keys

Prisma models:

  • Glob: **/schema.prisma, **/*.prisma
  • Content match: model <Name> {
  • Extract: model name, fields with types, relations (@relation)

TypeORM / Sequelize / Drizzle (TypeScript ORMs):

  • Grep for: @Entity(), @Table, Model.init, pgTable(, mysqlTable(
  • Look in: **/models/**, **/entities/**, **/schema/**
  • Extract: class/table name, decorated columns, relations

Pydantic / dataclass models (Python):

  • Grep for: class.*BaseModel, @dataclass, class.*Model.*models.Model (Django)
  • Look in: **/models/**, **/schemas/**, **/domain/**
  • Extract: class name, field names/types, validators

TypeScript interfaces / types:

  • Grep for: export interface, export type.*=.*{
  • Look in: **/types/**, **/interfaces/**, **/models/**
  • Extract: interface/type name, property names/types

Protobuf messages:

  • Glob: **/*.proto
  • Content match: message <Name> {
  • Extract: message name, fields with types

3. Extract Data Model Information

For each discovered model, extract:

FieldSource
nameTable/class/interface name, converted to Title Case
descriptionJSDoc/docstring/comment above definition, or TBD
entity_typeroot if standalone, child if has foreign key to parent, value-object if embedded
attributesColumn/field definitions with name, type, required status
classificationpii if field names suggest personal data (email, phone, ssn, address), internal otherwise

4. Build Data Hierarchy

Group discovered models into the three-level registry hierarchy:

  1. Data Concept — High-level business concept (e.g., "Customer", "Order", "Payment")
    • Inferred from: table name prefixes, module/folder grouping, foreign key clusters
    • One concept per logical grouping
  2. Data Aggregate — Bounded collection of entities (DDD aggregate root + children)
    • Inferred from: root tables with child tables referencing them
    • If unclear, each standalone table/model becomes its own aggregate
  3. Data Entity — Individual table/model with attributes
    • Direct mapping from discovered models
    • Includes extracted attributes array

5. Check for Duplicates

Before proposing entries, check existing registry entries:

  • List existing data_concept, data_aggregate, and data_entity entries
  • Compare by name (case-insensitive)
  • Flag potential duplicates

6. Present Findings

Show the user a summary:

**Data Model Discovery Results** — scanned: <path>

Found X data models:

**Proposed Hierarchy:**

📦 Customer (Data Concept)
  └─ 🗃️ Customer Aggregate (Data Aggregate)
       ├─ 📝 Customer (root entity, 8 attributes)
       ├─ 📝 Customer Address (child entity, 5 attributes)
       └─ 📝 Customer Preference (value-object, 3 attributes)

📦 Order (Data Concept)
  └─ 🗃️ Order Aggregate (Data Aggregate)
       ├─ 📝 Order (root entity, 12 attributes)
       └─ 📝 Order Line Item (child entity, 6 attributes)

**Duplicates:** (if any)
- "Customer" already exists in registry as data-concepts/customer.md

**Classification hints:**
- Customer → PII detected (email, phone fields)
- Order → internal

7. Write Entries (if --write or user confirms)

For each approved entry:

  1. Generate kebab-case filename from the model name
  2. Read the _template.md for the target type
  3. Fill in discovered fields:
    • name, description, status: draft
    • classification for data_concept
    • entity_type and attributes for data_entity
    • Parent relationships (parent_data_concept, parent_data_aggregate)
  4. Write to the correct registry folder
  5. Report what was written

8. Post-Scan Report

**Written X entries:**
- data_concept: registry-v2/<path>/concept-name.md
- data_aggregate: registry-v2/<path>/aggregate-name.md
- data_entity: registry-v2/<path>/entity-name.md

**Next steps:**
1. Review and fill in TBD fields (owner, description)
2. Verify classification (PII, business-confidential, internal)
3. Wire component relationships: `owned_by_component`
4. Run `/validate` to check model consistency

Detection Priority

  1. SQL CREATE TABLE statements (highest confidence — explicit schema)
  2. Prisma models (high confidence — typed, complete)
  3. ORM entity decorators (high confidence — structured)
  4. Protobuf messages (high confidence — typed)
  5. Pydantic/dataclass models (medium confidence — may be DTOs not domain models)
  6. TypeScript interfaces (lower confidence — may be API shapes not data models)

Notes

  • Always propose as status: draft — never auto-promote to active
  • PII classification is a hint based on field names — human must verify
  • If a model has no clear parent, make it both a data_concept and data_aggregate (flat hierarchy)
  • Skip migration files — focus on current model state, not history
  • Skip test fixtures and mock data files
  • Large codebases: limit scan to first 100 models and suggest narrowing the path

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.