dagster-orchestration-adapter
Skill2 starsChanged 4 months agoApache-2.0View on GitHub
Run the medallion sequence UNATTENDED / in CI as a Dagster asset graph -- the scheduler sibling of the retail-orchestrate conductor -- running ONLY already-approved steps behind every gate, and recording what each asset DID as DERIVED run-evidence. Use when someone asks to schedule the pipeline, run the medallion in CI, "orchestrate unattended", or wire Dagster in the Seshat BI repo. Dagster RUNS approved steps; Tower BI (the gate exit code + the named human) decides whether a stage passed. This skill READS committed approvals as the GO signal, EXECUTES approved steps behind STOP / HUMAN-SEAM edges, and WRITES run-evidence; it never defines meaning, never moves a stage to pass, never publishes Power BI (it only TRIGGERS F016), and HARD-STOPS at every human judgment call.
- Reads credentials
- Installs packages
In this repository60 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- approval-consoleSkill
- approval-evidence-packSkill
- business-knowledge-interviewSkill
- capabilitiesSkill
- consumer-data-dictionarySkill
- cross-table-lineageSkill
- dashboard-designSkill
- dashboard-intelligenceSkill
- dbt-transformation-adapterSkill
- evidence-pack-generatorSkill
- first-hour-compassSkill
- friendly-pr-reviewerSkill
- grain-confidence-reviewerSkill
- kpi-contract-builderSkill
- pbi-mcp-write-adapterSkill
- pbip-workflowSkill
- pbip-xraySkill
- pbir-authoring-adapterSkill
- portfolio-watchSkill
- powerbi-dashboard-designSkill
- pr-readiness-reviewerSkill
- readiness-viewerSkill
- release-notes-generatorSkill
- report-intent-interviewSkill
- retail-build-warehouseSkill
- retail-control-roomSkill
- retail-discover-portfolioSkill
- retail-governSkill
- retail-initSkill
- retail-onboard-tableSkill
- retail-orchestrateSkill
- retail-scaffoldSkill
- retail-semantic-checkSkill
- retail-validateSkill
- run-next-readinessSkill
- seshat-studioSkill
- showcase-buildSkill
- source-mappingSkill
- speckit-analyzeSkill
- speckit-checklistSkill
- speckit-clarifySkill
- speckit-constitutionSkill
- speckit-git-commitSkill
- speckit-git-featureSkill
- speckit-git-initializeSkill
- speckit-git-remoteSkill
- speckit-git-validateSkill
- speckit-implementSkill
- speckit-planSkill
- speckit-specifySkill
- speckit-tasksSkill
- speckit-taskstoissuesSkill
- bi-analyst-knowledgeSkill
- bi-bigdata-knowledgeSkill
- bi-dax-knowledgeSkill
- bi-python-knowledgeSkill
- bi-sql-knowledgeSkill
- retail-kpi-knowledgeSkill
The file
We split this skill into the rules it gives an agent
11 rules. About 0 are common, 7 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈7
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- could be automatic: a setting could enforce it
- Distinctive: line 117
No other external boundary.
- Distinctive: line 157
MUST NOT define metrics, mappings, semantic logic, grain, rollup, segment, PII disposition, or dashboard design (execution-only).
- Very specific: line 159
MUST NOT create truth or grant approval / move a stage to pass.
- Distinctive: line 162
MUST NOT execute a gated asset when its required approval / evidence is absent -- it fails closed.
- Very specific: line 164
MUST NOT publish a Power BI model or open a Power BI connection.
- Distinctive: line 166
MUST NOT emit a numeric / maturity / confidence score in run evidence (hard rule #9).
- Distinctive: line 167
MUST NOT run a gate around a failed upstream gate (no run-around of a STOP edge).
- Distinctive: line 168
MUST NOT commit real hostnames / DSNs / credentials (Principle IX).
- Very specific: line 172
When the artifact it would execute is undefined, or the gate it is downstream of is not pass, the adapter SURFACES it as a blocker, HALTS …
- Very specific: line 201
Read approval from existing state -- you may READ Gate status.
- Distinctive: line 327
NOT produce a finished, published deliverable -- that is the named-seam design, not a bug.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
dagster-orchestration-adapter
- Authority category (F024): Execution Adapter /
DB-connected(NOT publish-capable; it TRIGGERS F016, it never publishes). The F024 enumerated declaration -- seedocs/architecture/product-modules.md. - Roadmap feature: F030 On-disk spec:
specs/024-dagster-orchestration-adapter(dir 024 == F030; when the dir number and the F-number disagree, the roadmap F-number wins).
Dagster is the unattended / CI RUNTIME for the medallion sequence: a scheduler / pipeline runs
the asset graph so a human does not re-type each step. retail-orchestrate (F005) is the
INTERACTIVE conductor of the SAME sequence; Dagster is its unattended sibling -- same medallion
sequence, same gate-exit authority, same two human seams, neither self-approving. Tower BI (the
gate exit code + the named human) is the brain; Dagster is the runner. This skill is how the
agent reasons about running the medallion unattended without letting the orchestrator become the
brain. You (the agent reading this) ARE the design surface; this skill is procedure, not an
engine -- the engine, when built, is the separate Dagster project. See
docs/decisions/0010-dagster-is-orchestration-adapter.md and
docs/integrations/dagster-adapter.md.
STATUS: the Dagster PROJECT is BUILT (spec 134, activation slice of F030; the dbt engine seam is ACTIVATED per spec 135).
orchestration/dagster/exists --definitions.py, the 11-asset graph,jobs, one STOPPED schedule, one STOPPED sensor,pyproject.tomlpinningdagster==1.13.14and bringingseshat-bi[dbt](thedagster-dbtpin was dropped by spec 135 FR-011: it excludes dbt-core 1.12 and sits on no execution path) -- plus theseshat dagster doctor|run|evidencecontrol layer. The OPERATIONAL procedure:
seshat dagster doctor-- read-only preflight (project, its own venv underorchestration/dagster/.venv, the pinned dagster, the resolved build engine per table, per-table gate state, DSN present/absent). Blockers exit 2; the install remedy iscd orchestration/dagster && uv venv .venv && uv pip install -p .venv -e "../..[dbt]" -e ".[dev]".seshat dagster run --job <full_sequence_job|through_gold_job> [--table <table>]-- executes the graph as a shell-free child process in the orchestration venv. A failed/blocked gate halts downstream assets and exits 3 (the CI signal); report it with the recorded blocking_reason + named owner.seshat dagster evidence [--run-id <id>]-- list runs / render the committed record atorchestration/dagster/run-evidence/<run-id>.md.Without DB credentials the DB-touching assets record a deferred boundary and block fail-closed -- report that truthfully; never fake a run.
Authority declaration (F024) -- the filled adapter contract
This adapter declares EXACTLY ONE of the five F024 authority categories. Quoted verbatim from
docs/architecture/product-modules.md: an Execution Adapter is "a tool that crosses an
external trust/connectivity boundary to MATERIALIZE or PUBLISH an already-approved artifact. An
adapter MUST declare exactly one connectivity level: local-only | DB-connected |
external-service-connected | publish-capable. It is execution-only and gated; it never
defines metrics, mappings, semantic logic, or dashboard design." Dagster's connectivity level is
DB-connected -- the STRONGEST boundary it itself crosses (it loads bronze, runs
migrations, runs retail validate against a live Postgres). It is NOT publish-capable: the
terminal asset TRIGGERS the parked F016 Power BI Execution Adapter and Dagster opens no Power BI
connection itself.
The filled templates/adapter-contract.md declaration follows.
Adapter Contract -- Dagster Orchestration Adapter
- Authority category: Execution Adapter
- Connectivity level:
DB-connected(exactly one -- the STRONGEST it uses; it TRIGGERS the publish-capable F016 but never publishes itself) - Product layer:
1(Layer D / orchestration -- the functional axis; it SEQUENCES all seven readiness stages and DECIDES none) - Roadmap feature:
F030On-disk spec:specs/024-dagster-orchestration-adapter - Owner: orchestration / platform owner (a named human; Dagster never self-approves)
- Status: BUILT (spec 134 activation slice: the
orchestration/dagster/runtime project + theseshat dagstercontrol layer + CI definitions-load smoke; automations ship STOPPED)
What it does (one line)
Runs the already-approved medallion sequence unattended / in CI as an asset graph, behind every gate, and records what each asset DID as derived run-evidence -- deciding no stage and publishing nothing (it only TRIGGERS F016 once
publish_ready = pass).
Gate it is DOWNSTREAM of
An orchestrator runs MANY gated steps, so the gate is PER-ASSET, not one stage. Each gated asset runs only after its own gate is satisfied; the asset fails closed / blocks if it is not. The load-bearing gates:
- Gated on stage (per asset):
silver_tablesis gated on Mapping Ready (the committedGate status: CLEARED+ zero open rows);semantic_modelis gated on the committed semantic-model / metric approval;dashboard_blueprintis gated on Semantic Model Ready;publish_execution_evidenceis gated onpublish_ready = pass. The mechanical STOP nodes (silver_tables,gold_tables,semantic_model) are gated on a literal gate exit 0. - Required approval / evidence: for human-seam assets, the committed approval read from disk
(the
Gate statusfield; theapprovals[]owner+date inmappings/<table>/readiness-status.yaml). For mechanical assets, theseshat check/retail validateexit 0. - Fail-closed behavior: a failed gate asset HALTS and propagates the stop to every downstream asset (a STOP edge); a human-seam asset whose approval is absent BLOCKS and runs nothing. In both cases the run terminates with a non-zero / failed Dagster run status (the CI signal) and records the concrete blocker + named owner. See the asset graph below.
Boundaries it CROSSES (connectivity)
- Opens a connection to a live database to load bronze and run silver/gold migrations or dbt builds (DB-connected).
- Connects to the live database (read-only) to run
retail validate(DB-connected). - TRIGGERS the publish-capable F016 adapter when
publish_ready = pass-- it does NOT itself cross the publish boundary (no Power BI connection; F016 owns publish). - No other external boundary.
Approved artifact it MATERIALIZES / PUBLISHES
The definition MUST already exist in Core Authority. The adapter executes it; it does not author it.
- Materializes silver / gold from the APPROVED
source-map.yaml(after Mapping Ready) -- via the governedseshat.dbtbuild when the dbt engine is selected (spec 135), or via SQL-migration assets otherwise; identical gate semantics either way. - Generates the handoff pack from already-approved upstream evidence.
- Publishes NOTHING itself; the terminal asset TRIGGERS the already-approved F016 publish once
publish_ready = pass.
Derived run-evidence it WRITES
A RUN RECORD (what ran, when, with what result) as derived evidence. Never a new truth or approval.
- A per-run record at
orchestration/dagster/run-evidence/<run-id>.md(a filled copy oftemplates/dagster-run-evidence.md): per-asset gate command + exit code + measured numbers + outcome (materialized/failed/skipped/blocked-- never the readiness tokenpass), timestamps, commit sha, and per blocked/skipped asset the concreteblocking_reason- named owner.
- Those measured results are ALSO surfaced as
evidence[]/blocking_reasons[]on the affected table'smappings/<table>/readiness-status.yaml. Whether that evidence MARKS any stagepassis Core Authority's record, never Dagster's write.
Secrets handling (Principle IX)
- Credentials: reads the git-ignored
.envkeys only -- e.g.DATABASE_URLor theANALYTICS_DB_*set; NEVER inline real values. - Committed example only: a
profiles.example.yml/.env.examplewith placeholder values only -- never a real host / DSN / credential.
Forbidden operations (the matrix says NO)
These hold for EVERY Execution Adapter regardless of connectivity level:
- MUST NOT define metrics, mappings, semantic logic, grain, rollup, segment, PII disposition, or dashboard design (execution-only).
- MUST NOT create truth or grant approval / move a stage to
pass; MUST NOT write a readinessstatusor aGate status: CLEARED; MUST NOT invent a parallel approval marker (named-human / Core Authority only). - MUST NOT execute a gated asset when its required approval / evidence is absent -- it fails closed.
- MUST NOT publish a Power BI model or open a Power BI connection; the connectivity level is
DB-connected, notpublish-capable(it may only TRIGGER F016). - MUST NOT emit a numeric / maturity / confidence score in run evidence (hard rule #9).
- MUST NOT run a gate around a failed upstream gate (no run-around of a STOP edge).
- MUST NOT commit real hostnames / DSNs / credentials (Principle IX).
How it handles a missing definition or approval
When the artifact it would execute is undefined, or the gate it is downstream of is not pass,
the adapter SURFACES it as a blocker, HALTS the affected asset, and terminates the run failed
(the CI signal) -- it never invents the definition, self-approves, or executes past the missing
gate (Principle V; stop-and-ask). A judgment call surfaced mid-run (grain / PII / rollup /
segment / sentinel-vs-null) HALTS the affected asset and escalates to the named owner.
Scope boundary (read first)
Sequence-and-record only. This skill RUNS approved steps behind the gate and records what each
asset did as evidence; it does NOT define source mapping, metric contracts, or semantic logic,
does NOT publish Power BI, and does NOT move a readiness stage to pass. Three non-negotiables:
This adapter is not the owner of generic Dagster competence. Asset, job, resource, component,
schedule, sensor, automation, project-structure, CLI, and troubleshooting guidance belongs to
the official dagster-expert skill. Its activation and discovery must be proven by the
Spec 148 harness probe before routing to it. This skill owns only Seshat's readiness-aware sequencing, named-human
stops, closed execution policy, fail-closed propagation, redaction, evidence normalization,
and cross-tool BI progression around the official Dagster runtime; it must not reproduce
upstream instructions.
- The gate exit code is the SOLE pass authority (Principle I -- agent proposes, gate
disposes). A Dagster asset's success means "the command ran and returned this exit," NEVER
"the stage is now
pass." A greenseshat check/retail validateasset records exit-0 EVIDENCE; Core Authority's process (and a named human at the human-seam stages) records the stagepass. There is NO path by which a green asset alone writespass. - Two hard human seams you MUST halt at (mirrored from
retail-orchestrate):- Mapping gate (Principle IV): no
silver_tablesuntil the map is reviewed and approved. Read approval from existing state -- you may READGate status; you may not writeCLEAREDyourself (approval is the reviewer's action). Do NOT invent a parallel "APPROVED-FOR-SILVER" marker -- that forks an existing field into two sources of truth. - Judgment calls (Principle V): grain, PII publish-safety, business rollup, the authoritative returns column, sentinel-vs-null -- HALT the affected asset and escalate to a named human; do not decide them to make a finding go away.
- Mapping gate (Principle IV): no
- Dagster stops at the publish wall (Principle II). It is
DB-connected, not publish-capable; the terminal asset only TRIGGERS F016 oncepublish_ready = pass. Even whenpublish_ready = passbut F016 is parked / absent, the publish asset FAILS CLOSED -- it recordsblocking_reason"F016 publish adapter not available" with the named owner and publishes nothing as a fallback.
ASCII only, UTF-8 no BOM in everything you author (--, ->; no em-dash / smart-quote /
unicode-arrow).
Run-state: read mappings/<table>/ FIRST (no new state file)
Compute what each asset may do from what is already on disk -- there is NO Dagster orchestration state file to create; the GO signal is the EXISTING committed gate state, exactly as the conductor reads it:
| What you observe | Action |
No mappings/<table>/ dir | The source_map asset is not yet reachable; the front door is retail-onboard-table -> source-mapping. Run no silver. |
Gate status: OPEN (or any open row) in mappings/<table>/unresolved-questions.md | source_map HUMAN SEAM blocks. silver_tables does not materialize; record the open mapping blocker + named owner; write no approval. |
Gate status: CLEARED and zero open rows | The source_map seam is satisfied; silver_tables is PERMITTED to run (the read of committed approval is the only GO signal). |
silver/gold built; a gate asset returns non-zero | STOP edge. The asset is failed; every downstream asset is skipped; the run terminates failed. |
publish_ready not pass in readiness-status.yaml | The publish_execution_evidence asset BLOCKS and triggers nothing; record the missing publish approval. |
You may READ Gate status / approvals[]; you may NOT write CLEARED or a stage pass to any
artifact (that is the named human's / Core Authority's action). Read approvals by path + git ref.
The asset graph (gate semantics, not just names)
The planned graph is eleven assets. Each dependency edge is a STOP edge (a failed gate halts all
downstream assets) or a HUMAN-SEAM edge (reads a committed approval and halts if absent -- never
writes it). The full enumerated graph + project shape lives in
docs/integrations/dagster-adapter.md; the gate semantics:
raw_source_file
-> bronze_<table> (load bronze; DB write of raw landing)
-> source_profile (profile the source; writes profile evidence)
-> source_map [HUMAN SEAM: reads Gate status; HALTS if not CLEARED]
-> silver_tables [STOP edge: blocked until source_map gate CLEARED]
-> gold_tables (Kimball star; mechanical -- writes check evidence)
-> metric_contracts [reads approved contracts; does not author them]
-> semantic_model [HUMAN SEAM: reads semantic-model approval]
-> dashboard_blueprint (design evidence; gated on semantic_model_ready)
-> handoff_pack (generate the BI handoff bundle; writes evidence)
-> publish_execution_evidence [gated on publish_ready = pass; TRIGGERS F016 only]Mechanical STOP nodes (silver_tables, gold_tables, semantic_model) gate on a literal gate
exit 0. HUMAN-SEAM nodes (source_map, semantic_model publish-safety, publish_execution_evidence)
gate on a committed human approval. The terminal asset TRIGGERS the parked F016 adapter; Dagster
never publishes itself.
Fixed sequence (run only behind the gate; the SAME commands CI runs)
| Asset | What it runs | Authority note |
bronze_<table> | Load the raw landing into bronze. | Execute an approved step (DB write). |
source_profile | Profile the source; write profile evidence. | DERIVED evidence only. |
source_map | READ Gate status + approvals[]. If not CLEARED -> HALT, record the open mapping blocker. | HUMAN SEAM (Principle IV); never self-grant. |
silver_tables | Build silver (dbt via the governed seshat.dbt build, or SQL migrations); then seshat check. | STOP edge; gated on source_map CLEARED. |
gold_tables | Build the gold star; then seshat check. | STOP edge; mechanical. |
metric_contracts | READ the approved metric contracts (authors none). | Reads truth; never authors it. |
semantic_model | seshat check + contract-binding read; READ the semantic-model approval. | STOP + HUMAN SEAM. |
dashboard_blueprint | Produce design evidence. | Gated on semantic_model ready. |
handoff_pack | Generate the BI handoff bundle; write evidence. | DERIVED evidence only. |
publish_execution_evidence | READ publish_ready. If pass -> TRIGGER F016. If F016 absent -> FAIL CLOSED. | Publish wall (Principle II); triggers F016 only. |
| (every asset) | Write the run-evidence record; surface results as evidence[] / blocking_reasons[]. | DERIVED evidence; never a pass. |
At every mechanical gate node, run the SAME command CI runs (seshat check, and
retail validate once creds exist) so the unattended run behaves identically to the conductor's
gate. Validate evidence is recorded; a deferred-boundary (no creds / DB unreachable) is
recorded with its timestamp and NEVER fabricated into a pass.
Fail-closed: a halted run also FAILS the Dagster run status (the CI signal)
A failed gate asset HALTS all downstream assets, which are recorded skipped -- not run around.
Beyond writing the run-evidence record, a halted / fail-closed / human-seam-blocked run MUST
itself terminate with a non-zero / failed Dagster run status so an unattended scheduler surfaces
the blocker rather than exiting silently. That failed run status is DERIVED evidence ABOUT the
execution -- it flips NO readiness stage.
HARD-STOP: the Principle V judgment calls (never auto-resolve)
HALT the affected asset and escalate to a named human; record an owner row in
mappings/<table>/unresolved-questions.md -- do not edit around any of these:
- a grain ambiguity, or any step that would change the declared grain.
- a sentinel-vs-null choice; the authoritative returns column.
- a PII publish-safety question on any column.
- a business rollup / segment mapping the approved map does not answer.
- moving Mapping Ready / Silver Ready / Gold Ready / Semantic Model Ready / Publish Ready to
pass-- always a named-human or gate-exit action, never a green Dagster asset. - triggering a publish when
publish_readyis notpass, or publishing yourself when F016 is absent -- FAIL CLOSED. - any
dagsterMAJOR version bump -- a named reviewer approves; no automerge (pin the pair TOGETHER; the shared policy is F031 / F033). - any finding you cannot confidently classify -> default to escalate.
Auto-update posture (Dagster-specific; the shared policy is deferred)
Pin dagster exactly (the dagster-dbt pin was dropped by spec 135 FR-011; the governed dbt runtime arrives via seshat-bi[dbt]); updates via PR only; a
definitions-load smoke test as the minimum CI gate; a small orchestration smoke test once an
implementation exists; NO automerge for Dagster MAJOR versions. The SHARED cross-adapter
update/maturity policy lives in F031 (spec 025, adapter-maintenance-policy) and F033 (spec 027,
release-maturity); this skill states only Dagster's adapter-specific needs and DEFERS the rest.
Seams (deferred by design -- report and park, never fake)
- A live unattended run -- needs the DB + a git-ignored credential set (only
profiles.example.yml/.env.examplewith placeholders is committed; Principle IX). Without it, report the boundary + the enable steps (supply credentials in the git-ignored.env); never traceback, never fake a pass. - The Power BI publish -- F016 (parked, publish-capable, gated on Publish Ready); the
terminal asset TRIGGERS it once
publish_ready = passand never publishes itself. If F016 is absent, FAIL CLOSED. - The dbt build internals -- F029 owns HOW the silver/gold transformations are defined and
validated; this adapter owns the SEQUENCING and gate-respecting execution of them through
the governed
seshat.dbtcontrol layer. Do not redefine dbt's internals here.
At a seam: state plainly what is deferred, what would unblock it, and STOP. One invocation does NOT produce a finished, published deliverable -- that is the named-seam design, not a bug.
See also
- The decision record:
docs/decisions/0010-dagster-is-orchestration-adapter.md. - The integration doc + the enumerated
orchestration/dagster/shape + asset graph:docs/integrations/dagster-adapter.md. - The run-evidence record shape:
templates/dagster-run-evidence.md. - The copy-me adapter declaration:
templates/adapter-contract.md. - The category parent (the five authority categories + the matrix):
docs/architecture/product-modules.md; the decisiondocs/decisions/0008-core-authority-vs-product-modules.md. - The conductor sibling (the gate-read + human-seam posture this mirrors):
.claude/skills/retail-orchestrate/SKILL.md;specs/005-layer-d-orchestration/spec.md. - The transformation adapter it orchestrates through the
seshat.dbtcontrol layer:.claude/skills/dbt-transformation-adapter/SKILL.md;specs/023-dbt-transformation-adapter/(F029). - The verbs it sequences:
.claude/skills/{retail-onboard-table,source-mapping,retail-build-warehouse,retail-validate,retail-semantic-check}/SKILL.md. - The spec + posture:
specs/024-dagster-orchestration-adapter/spec.md;.specify/memory/constitution.mdPrinciples I, II, IV, V, VII, VIII, IX. - The method + exit gates:
docs/medallion-playbook.md. The readiness spine:docs/readiness/readiness-model.md. The replay reference (cited, never inlined): a worked example underdocs/worked-examples/.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

