arch-lens-data-lineage
Skill1 starsChanged 8 months agoMITView on GitHub
Create Data Lineage architecture diagram showing information flow, transformations, and storage destinations. Data-centric lens answering "Where is the data?
In this repository59 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- arch-lens-c4-containerSkill
- arch-lens-concurrencySkill
- arch-lens-deploymentSkill
- arch-lens-developmentSkill
- arch-lens-error-resilienceSkill
- arch-lens-module-dependencySkill
- arch-lens-operationalSkill
- arch-lens-process-flowSkill
- arch-lens-repository-accessSkill
- arch-lens-scenariosSkill
- arch-lens-securitySkill
- arch-lens-state-lifecycleSkill
- audit-archSkill
- audit-bugsSkill
- audit-cohesionSkill
- audit-defense-standardsSkill
- audit-frictionSkill
- audit-implSkill
- audit-testsSkill
- design-guardsSkill
- dry-walkthroughSkill
- elaborate-phaseSkill
- exp-lens-benchmark-representativenessSkill
- exp-lens-causal-assumptionsSkill
- exp-lens-comparator-constructionSkill
- exp-lens-error-budgetSkill
- exp-lens-estimand-claritySkill
- exp-lens-exploratory-confirmatorySkill
- exp-lens-fair-comparisonSkill
- exp-lens-governance-riskSkill
- exp-lens-iterative-learningSkill
- exp-lens-measurement-validitySkill
- exp-lens-pipeline-integritySkill
- exp-lens-randomization-blockingSkill
- exp-lens-reproducibility-artifactsSkill
- exp-lens-sensitivity-robustnessSkill
- exp-lens-severity-testingSkill
- exp-lens-unit-interferenceSkill
- exp-lens-validity-threatsSkill
- exp-lens-variance-stabilitySkill
- id-slopSkill
- implement-worktreeSkill
- investigateSkill
- make-arch-diagSkill
- make-experiment-diagSkill
- make-groupsSkill
- make-planSkill
- make-reqSkill
- make-scenariosSkill
- mermaidSkill
- plan-arcSkill
- rectifySkill
- review-approachSkill
- update-architectureSkill
- update-reqsSkill
- update-specsSkill
- verify-diagSkill
- verify-proposed-diagSkill
The file
We split this skill into the rules it gives an agent
12 rules. About 0 are common, 9 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈9
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 38
BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY
- Distinctive: line 51
Look for: CLI args, API requests, file reads, imports, user input, data ingestion
- Distinctive: line 56
Look for: Adapter, Converter, transform, parse, serialize, from_*, to_*, mapping, conversion
- Distinctive: line 61
Look for: schema models, type definitions, serialization, deserialization, format conversion
- Distinctive: line 66
Look for: database operations, persistence, .save(), .create(), .write(), storage
- Distinctive: line 71
Look for: .get(), .query(), .find(), .load(), read operations, data access layer
- Distinctive: line 75
Document the journey of key data entities:
- Distinctive: line 90
Write-only artifacts (debugging, logging) - system writes but never reads back
- Distinctive: line 93
Use different arrow styles:
- Very specific: line 131
Use cylinder shape: [(Label)]
- Very specific: line 136
Write the diagram to: temp/arch-lens-data-lineage/arch_diag_data_lineage_{YYYY-MM-DD_HHMMSS}.md
- Distinctive: line 232
Before creating the diagram, verify:
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Data Lineage Architecture Lens
Cognitive Mode: Data-Centric Primary Question: "Where is the data?" Focus: Information Flow, Transformations, Storage Locations, Format Conversions
When to Use
- Need to understand how data flows through the system
- Documenting data transformations and conversions
- Identifying storage destinations and access patterns
- User invokes
/arch-lens-data-lineageor/make-arch-diag data
Critical Constraints
NEVER:
- Modify any source code files
- Focus on runtime behavior (that's process flow lens)
- Show static structure without data context
ALWAYS:
- Trace data from INPUT to STORAGE
- Show transformation stages and format changes
- Identify the single source of truth
- Distinguish read vs write operations
- BEFORE creating any diagram, LOAD the
/mermaidskill using the Skill tool - this is MANDATORY
Analysis Workflow
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Data Origins (Inputs)
- Find user input handling
- Identify external data sources
- Look for: CLI args, API requests, file reads, imports, user input, data ingestion
Transformation Stages
- Find data conversion/transformation code
- Identify adapters and converters
- Look for: Adapter, Converter, transform, parse, serialize, from_, to_, mapping, conversion
Format Changes
- Find schema definitions and conversions
- Identify format boundaries (JSON, XML, protobuf, etc.)
- Look for: schema models, type definitions, serialization, deserialization, format conversion
Storage Destinations
- Find database operations
- Identify file outputs
- Look for: database operations, persistence, .save(), .create(), .write(), storage
Access Patterns
- Find data retrieval code
- Identify query patterns
- Look for: .get(), .query(), .find(), .load(), read operations, data access layer
Step 2: Map Data Flow
Document the journey of key data entities:
- Origin: Where does it come from?
- Transformations: What changes happen?
- Storage: Where is it persisted?
- Retrieval: How is it accessed later?
CRITICAL - Analyze Read/Write Direction: For EVERY storage location and data flow:
- Read sources (inputs): Components that READ from this location
- Write destinations (outputs): Components that WRITE to this location
- Read-write (primary storage): Both read and written by the system
- Write-only (artifacts): Written but NEVER read back by the system
Clearly distinguish:
- Primary storage (source of truth) - system reads AND writes
- Write-only artifacts (debugging, logging) - system writes but never reads back
- External inputs - system reads only
Use different arrow styles:
- Solid arrows for read/write primary storage
- Dashed arrows for write-only artifacts
Step 3: Identify Conversion Boundaries
Find format changes:
- External format -> Internal format
- Internal format -> Database format
- Database format -> API response
- Note naming convention changes
Step 4: Create the Diagram
Use flowchart with:
Direction: LR (left-to-right) for data flow, or TB for hierarchical
Subgraphs for Stages:
- Input/Origins
- Transformation/Processing
- Storage (primary)
- Artifacts (secondary/write-only)
- External Sync (if applicable)
Node Styling:
cliclass: Data origins, user inputhandlerclass: Transformation, adaptersstateNodeclass: Database tables, primary storageoutputclass: Write-only artifacts, filesintegrationclass: External sync, APIs
Connection Types:
- Solid arrows for primary data flow
- Dashed arrows for write-only/secondary
- Label with operation names
Database Nodes:
- Use cylinder shape:
[(Label)] - Show table relationships
Step 5: Write Output
Write the diagram to: temp/arch-lens-data-lineage/arch_diag_data_lineage_{YYYY-MM-DD_HHMMSS}.md
Output Template
# Data Lineage Diagram: {System Name}
**Lens:** Data Lineage (Data-Centric)
**Question:** Where is the data?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Data Flow Overview
| Stage | Format | Key Transformation |
|-------|--------|-------------------|
| Input | {format} | {description} |
| Processing | {format} | {description} |
| Storage | {format} | {description} |
## Lineage Diagram
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart LR
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Input ["Data Origins"]
USER["User Input<br/>━━━━━━━━━━<br/>Source type<br/>Format"]
end
subgraph Transform ["Transformation"]
direction TB
ADAPTER["Adapter<br/>━━━━━━━━━━<br/>Conversion type"]
end
subgraph Storage ["Primary Storage (Source of Truth)"]
direction TB
DB[("Database Table<br/>━━━━━━━━━━<br/>Key fields")]
end
subgraph Artifacts ["Write-Only Artifacts"]
direction TB
FILE["output.json<br/>━━━━━━━━━━<br/>For debugging"]
end
%% FLOWS %%
USER -->|"input"| ADAPTER
ADAPTER -->|"save()"| DB
DB -.->|"write-only"| FILE
%% CLASS ASSIGNMENTS %%
class USER cli;
class ADAPTER handler;
class DB stateNode;
class FILE output;Color Legend:
| Color | Category | Description |
| Dark Blue | Input | Data origins (user, external) |
| Orange | Transform | Format conversion and adapters |
| Teal | Storage | Primary storage (source of truth) |
| Dark Teal | Artifacts | Write-only outputs |
| Red | Sync | External sync services |
Data Transformation Summary
| Stage | Format | Key Conversion |
| {stage} | {format} | {conversion} |
Storage Destinations
| Entity | Primary Storage | Secondary | Access Pattern |
| {entity} | {location} | {artifact} | {how accessed} |
Critical Design Principle
Source of Truth: {e.g., "Database is single source of truth. File outputs are write-only."}
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/make-arch-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/arch-lens-c4-container` - For container-level storage viewDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

