new-monitor-kind
Skill0 starsChanged 9 days agoMITView on GitHub
Add a new `check.kind` (monitor kind) end-to-end — registry, run-path routing, authoring gates, dimension derivation, dry-run, export/import, MCP, frontend catalog, docs and tests. Walks the exact traversal used by freshness/volume (#426/#437), comparison (ADR 0015, #791–#795), schema_drift (#592) and anomaly (#593, #1117/#1119), with the traps each one hit. All six CHECK_KINDS are shipped — use when adding a genuinely NEW kind (which now always needs a constraint-widening migration first) or when the user says "add a new monitor kind".
In this repository6 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- adr-createSkill
- gh-issue-from-findingSkill
- live-verifySkill
- security-scanSkill
The file
We split this skill into the rules it gives an agent
10 rules. About 0 are common, 2 are distinctive, and 8 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
8
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 15
Nothing is reserved-and-unbuilt: all six CHECK_KINDS are shipped — expectation, freshness/volume (#426/#437), comparison (#791–#795), …
- Very specific: line 19
Read ADR 0012, and the ADR for the kind if one exists (comparison →
- Distinctive: line 20
Read the closest precedent's PR diff end-to-end.
- Very specific: line 74
Add an entry to _BY_KIND in backend/app/services/check_dimension.py only if the kind's dimension is genuinely derivable from its shape:
- Very specific: line 80
If it is not derivable, return None and leave it out.
- Very specific: line 86
Check backend/app/services/suite_io_service.py — the if "dimension" in c branch is exactly this distinction.
- Very specific: line 101
add the catalog spec: type: 'monitor:my_kind', kind: 'my_kind', the input fields, threshold requirements (freshness requires a …
- Distinctive: line 102
check the per-datasource adjustment hook (the flat-file freshness special case) if your kind behaves differently by connection type
- Very specific: line 118
The empty/absent case: no rows, no baseline, a None scalar.
- Very specific: line 125
Run /live-verify against every datasource whose feature-matrix cell you are about to tick.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
new-monitor-kind
Purpose
check.kind is the ADR 0012 monitor-kind seam. Adding one is a wide, ordered traversal — registry → run path → authoring → classification → surfaces → docs — that is easy to half-finish. A half-finished kind is worse than none: it is dispatchable but unauthorable, or authorable but silently unclassified.
This repo has now walked it five times. This skill is that traversal, plus the trap each walk hit.
Nothing is reserved-and-unbuilt: all six CHECK_KINDS are shipped — expectation, freshness/volume (#426/#437), comparison (#791–#795), schema_drift (#592), anomaly (#593).
Before you start
- Read ADR 0012, and the ADR for the kind if one exists (
comparison→ 0014/0015). If the kind has no ADR and is non-trivial, write one first — use/adr-create. - Read the closest precedent's PR diff end-to-end. Pick by shape:
- scalar (one SQL aggregate → a badness number) →
freshness/volume - stateful (needs a stored baseline to compare against) →
schema_drift(#592), oranomaly(#593) — the closer precedent for a LEARNED baseline: it reusesmonitor_baselineswith no migration and returnsskipon cold start (backend/app/services/anomaly.py) - two-dataset (source + target) →
comparison(#791–#795)
- scalar (one SQL aggregate → a badness number) →
- Decide the shape first. It determines whether
build_statementis a real function orNone, and that single choice drives the run-path routing for free.
The traversal
1. It needs a migration — every unclaimed kind is now built
backend/app/db/models.py CHECK_KINDS backs a real DB constraint (_in_check("kind", CHECK_KINDS, "kind_valid") on both checks and monitor_baselines), and all six values are shipped — there is no reserved slot left to claim.
- A new kind therefore always needs an Alembic migration widening the constraint, landing in its own PR before the code that writes the new value (working-agreement #30, two-step). Run
/agents migration-safetyon it.
2. Register the strategy — the one required step
backend/app/datasources/monitors.py:
MY_KIND = "my_kind"
def _validate_my_kind(config: dict[str, Any]) -> None: ...
def _my_kind_outcome(scalar, config, now) -> CheckOutcome: ...
def _my_kind_statement(table, config) -> Select[Any]: ... # or omit for a stateful kind
MONITOR_KIND_REGISTRY[MY_KIND] = MonitorKindStrategy(
MY_KIND, _validate_my_kind, _my_kind_outcome, _my_kind_statement,
)Register in the dict literal, not at runtime. The module comment is explicit: every derived value (MONITOR_KINDS, SCALAR_MONITOR_KINDS, STATEFUL_MONITOR_KINDS, the authoring allowlist, runners' capability sets) snapshots at import. A late registration is half-visible — dispatchable but unauthorable. Tests may monkeypatch; production code must not.
build_statement=None is what routes a kind to the stateful path. You do not write routing code; you choose a shape.
Semantics to lock in the module docstring (this is where the kind's contract lives):
- what the
metric_valueis, in one sentence, and that higher = worse (ADR 0016 bands it) - the exact
configkeys and their units - what happens on the empty/absent case
3. Run path
- Scalar kinds flow through the runners'
run_monitors(run_monitor_specs/run_monitors_over_engine) with no new code — but each runner must advertise the kind:supported_monitor_kindsinsnowflake.py,unity_catalog.py,iceberg.py,flatfile.py. A kind absent from a runner's set is refused at author time for that datasource, which is the correct default — widen deliberately, per datasource, with evidence it works there. - Stateful kinds need an executor the worker injects. See
backend/app/services/schema_drift.pyandbackend/app/services/anomaly.py(each owns its baseline read/write and its session) and their wiring inbackend/app/worker/tasks.py+ the partition doc inbackend/app/services/run_service.py.
4. Authoring gates — backend/app/services/check_service.py
_V1_SUPPORTED_KINDS derives from MONITOR_KINDS, so registration widens it automatically. What is not automatic:
- the per-datasource capability set your kind keys off — there is now a registry,
_CAPABLE_TYPES_BY_KINDincheck_service.py(keyed entries forschema_drift/anomaly,MONITOR_CAPABLE_TYPESas the fallback); a new kind with its own capability set adds an entry there - any config guardrail that needs the DB (a real column, a resolvable target)
- the
expectation_typepairing — usemonitor_expectation_type(MY_KIND)→monitor:my_kind. Never hand-write the string; the author path asserts kind↔type and the frontend catalog mirrors it.
5. Dimension — ADR 0038. The #124 trap lives here.
Add an entry to _BY_KIND in backend/app/services/check_dimension.py only if the kind's dimension is genuinely derivable from its shape:
_BY_KIND = {"freshness": TIMELINESS, "volume": COMPLETENESS, "schema_drift": CONSISTENCY, ...}If it is not derivable, return None and leave it out. None is a real answer that renders as a coverage gap; a plausible-looking guess fills the #889 scorecard with confident nonsense. accuracy and integrity are never derivable.
The trap: #124's export/import path silently classified every check in prod — an ADR violation inside the PR that added the ADR — and the tests missed it by construction. One test POPPED the dimension key instead of setting it to null, and the only null that round-tripped was custom SQL, where derivation also returns None. So:
Write the round-trip test with
dimension: nullpresent in the payload, on a check whose type is derivable. Present-and-null must stay null; absent must derive.
Check backend/app/services/suite_io_service.py — the if "dimension" in c branch is exactly this distinction.
6. Remaining backend surfaces
| Surface | File | Needed? |
| Dry-run preview | services/dryrun_service.py | Only if the kind can be previewed without a run. It currently supports expectation, schema_drift and anomaly; unsupported kinds must raise DryRunUnsupportedError, not fall through. |
| Export / import | services/suite_io_service.py | Validation is atomic — every kind validated before any row is written. Add the kind's guardrail alongside the MONITOR_KINDS / COMPARISON_KIND branches. |
| Check API | api/v1/checks.py | Only for a kind-specific endpoint (e.g. schema_drift's rebaseline). |
| MCP tools | mcp/server.py | kind is a passthrough string — update the tool description so an LLM knows the kind exists and when to pick it. Descriptions are LLM-facing (CLAUDE.md §10). |
7. Frontend
frontend/src/components/checks/expectationCatalog.ts:
- widen the
CheckKindunion - add the catalog spec:
type: 'monitor:my_kind',kind: 'my_kind', the input fields, threshold requirements (freshnessrequires a fail/critical threshold because it has no in-config bound — if your kind is unbounded, it needs the same), and help text stating the metric's direction - check the per-datasource adjustment hook (the flat-file freshness special case) if your kind behaves differently by connection type
frontend/src/api/suites.ts carries kind as a string — update the doc comments so the contract stays readable.
8. Docs
docs/site/reference/feature-matrix.md— a row per datasource. Do not tick a cell you have not run live. #953 was a ✅ that had never once worked.CLAUDE.md§5 — move the kind from "reserved" to shipped, with the PR number.docs/progress.md— the cycle-plan row.- ADR 0012 — amend its status list.
docs/site/guides/datasources-checks.md/docs/site/get-started/concepts.md— user-facing description.
9. Tests
- Unit, in
backend/tests/:validate_configrejects each malformed shape;outcomebands correctly at each threshold boundary;build_statementrenders through the Core layer. - Identifier quoting: assert a mixed-case and an already-quoted column both work, and that the CONNECTION's dialect does the quoting. Never hand-roll
"— Unity Catalog uses backticks (#937). - The empty/absent case: no rows, no baseline, a
Nonescalar. What does the check report? It must not silently pass. - Dimension round-trip: present-and-null vs absent (see §5).
- Adversarial types: the scalar arrives from a driver. Feed
str,Decimal, tz-naive and tz-aware datetimes, Arrow-backed values. Run/agents driver-boundary-guardon the diff — this kind's whole job is to consume a driver value. - Mutation-check the regression tests:
/agents regression-mutation-verifier.
10. Live-verify before you tick anything
Run /live-verify against every datasource whose feature-matrix cell you are about to tick. Unit tests cannot settle a driver-typed value — that is the standing lesson (CLAUDE.md §13), and it is why UC freshness shipped broken for weeks.
Definition of done
- Registered in
MONITOR_KIND_REGISTRYat import time CHECK_KINDSadmits the value (migration filed separately if it did not)- Each runner that should support it advertises it in
supported_monitor_kinds - Authoring gate + capability set updated;
expectation_typeviamonitor_expectation_type - Dimension derived or deliberately left
None, with the present-null round-trip test - Dry-run either supports it or raises
DryRunUnsupportedError - Export/import validates it atomically
- Frontend catalog spec +
CheckKindunion - MCP tool description mentions it
- Docs: feature matrix (live-verified cells only), CLAUDE.md §5, progress.md, ADR 0012
- Tests: config, banding, quoting, empty case, dimension, adversarial types — all mutation-checked
/agents driver-boundary-guard+/agents qa-verifierclean/code-reviewrun on the PR, findings fixed or filed as issues
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

