probe-define
Skill0 starsChanged 2 months agoApache-2.0View on GitHub
Generate probe-skeleton YAML files at `lineage/{repo}/probes/{P-NNN}.yaml` from `feature-flows.yaml`'s uncovered test-matrix cells. Each skeleton carries the feature_id + test_class + cross-references + a TODO-marked arrange/act/observe/assert block the maintainer fills in. Accelerates probe corpus growth — slice 4 of dynamic-verification ADR. Local-only per APPROACH.md section 5 rule 12.
In this repository27 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- alignSkill
- code-walkSkill
- conceptsSkill
- contributeSkill
- coverageSkill
- doc-gap-checkSkill
- enrichSkill
- enumerateSkill
- find-implicit-adrsSkill
- implementSkill
- ingest-docsSkill
- log-issueSkill
- navigateSkill
- next-batchSkill
- orientSkill
- panelSkill
- probe-runSkill
- probeSkill
- reflect-featureSkill
- retrieveSkill
- reviewSkill
- scanSkill
- statusSkill
- streamsSkill
- test-coverageSkill
- triageSkill
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 34
Read lineage/{repo}/feature-flows.yaml.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Generate a probe skeleton from an uncovered cell (slice 4)
Drive the layer-5 probe-corpus growth. Each invocation produces (or refreshes) a probe-skeleton YAML at lineage/{repo}/probes/{P-NNN}.yaml. The skeleton carries enough scaffolding for the maintainer to fill in the arrange / act / observe / assert blocks with the specific behaviour the probe pins; the cross-references + frontmatter are pre-populated from feature-flows.yaml.
This skill is the maintainer-facing entry point for the canonical slice-4 deliverable per adrs/drafts/dynamic-verification-layer.md.
Prerequisite
lineage/{repo}/feature-flows.yamlexists with at least one feature.lineage/{repo}/probes/directory exists (/probe-runcreated it as part of slice 2).python3available (the skill uses a small inline script to compute the next-available probe ID and parse feature-flows.yaml).
Argument forms
| Form | Behaviour |
/probe-define --feature <feature-id> | Generate one skeleton for the first uncovered cell of that feature (lowest test_class index in unit/integration/performance/security order). |
/probe-define --feature <feature-id> --test-class <class> | Generate one skeleton for the specified test_class of that feature. |
/probe-define --feature <feature-id> --all-uncovered | Generate one skeleton per uncovered cell of the feature (up to 4 — one per test_class). |
/probe-define --feature <feature-id> --out-id P-NNN | Override the auto-derived probe ID (default: next-available based on existing probes). |
/probe-define --all | Generate skeletons across every feature × every uncovered cell. Useful for batch corpus growth. |
Protocol
1. Orient (read-only)
- Read
lineage/{repo}/feature-flows.yaml. Capture the feature(s) requested. - For each requested feature, enumerate uncovered cells: those whose
test_matrix.<class>.stateisGAPorPARTIAL. (PROBED-*cells are already covered — skip.) - Find the next available probe ID by scanning
lineage/{repo}/probes/P-*.yaml. Convention:P-NNNwhere NNN starts at 001 and increments. Don't reuse IDs. - Confirm the orchestration plan to the maintainer: list each skeleton that will be created.
2. Generate each skeleton
For each skeleton, write to lineage/{repo}/probes/{P-NNN}.yaml with this shape:
---
probe_id: P-NNN
feature_id: F-NNN
test_class: <class>
verified_against_commit: <substrate manifest's last_scan_commit>
prompt_version: probe-runner/0.1.0
maintainer_curated: false # set to true once the maintainer fills in the body
stack_profile: odd-minimal # adjust if the test_class requires a different profile
expected_outcome: |
TODO: in one paragraph, describe the user-observable behaviour this probe
pins. Frame as a user-facing promise that the system either keeps (PASS-PASSING)
or visibly breaks (PASS-PINNING-BUG). When the fix lands, the assertion flips.
Refer to the related_refactoring_scopes / related_test_gaps / related_doc_gaps
in cross_references below for what the probe is targeting.
---
# P-NNN — <descriptive title>
arrange:
- kind: docker-compose-up
# TODO: seed any data_source / data_entity / namespace rows the probe needs.
# Use a unique numeric ID (1000 + probe-number is the slice-2/3 convention).
- kind: sql
query: |
INSERT INTO data_source (id, oddrn, name)
VALUES (NNNN, '//probe-source/db', 'probe-source-P-NNN')
ON CONFLICT (id) DO NOTHING
# TODO: add more seed steps as needed (data_entity, namespace, term, etc.)
# Pre-load any initial state the probe will observe.
act:
# TODO: the action whose effect the probe measures. Examples:
# - kind: rest
# method: GET | POST | PUT | DELETE
# path: /api/...
# body: {...} # for POST/PUT; supports ${capture_name} substitution
# count: N # for repeated calls
# expect_any_status: true # for probes intentionally testing 4xx/5xx paths
# - kind: browser
# path: /dataentities/...
# wait_until: load
# post_settle_ms: 6000
# xhr_filter_regex: '^http://localhost:18080/api/...$'
observe:
# TODO: capture measurable state. Examples:
# - kind: sql
# query: "SELECT view_count FROM data_entity WHERE id = NNNN"
# capture_as: final_view_count
# - kind: response_contains
# from: <act-step capture_as>
# item_field: id
# target_value: NNNN
# capture_as: entity_appeared
# - kind: latency_distribution
# over: "all act steps"
# capture_percentiles: [p50, p95, p99]
# capture_as: latency_curve
assert:
# TODO: pin the user-observable behaviour. Format each assertion as a Python
# expression using captured variable names. Safe-eval forbids function calls,
# so use comparisons / membership / arithmetic only.
- "TODO: replace with a real assertion"
cleanup:
- kind: docker-compose-down
destroy_volumes: true
cross_references:
refactoring_scopes: [<auto-populated from feature-flows.yaml#F-NNN.related_refactoring_scopes>]
test_gaps: [<auto-populated>]
doc_gaps: [<auto-populated>]
related_concepts: [<auto-populated>]
retrospectives: [<auto-populated>]
realism_caveats: |
TODO: what this probe approximates rather than fully verifies. Examples
from slice-2/3 probes: stack_profile DISABLED-only (LOGIN_FORM/OAUTH2/LDAP
variants deferred); image tag mismatched with substrate commit; small seed
vs production-shape; etc.3. Report
Concise output:
- Count of skeletons created
- Path of each created file
- Next steps: "Edit each skeleton's TODOs; run
/probe-run --validate <P-NNN>to confirm well-formedness; then/probe-run <P-NNN>to execute against the local mirror."
Rules
- Never overwrite an existing probe file unless
--forceis explicit (slice-4 doesn't expose--forceyet — refuses to clobber). - Pre-populate cross-references from feature-flows.yaml's
related_*arrays. The probe-runner reads these on PASS/FAIL to merge sidecar confidence annotations. - Use the next-available numeric ID by default (
max(existing P-NNN) + 1). - Respect the stack_profile the feature implies (currently always
odd-minimal; slice-5 introduces variants). - Mark
maintainer_curated: falsein the skeleton; the maintainer flips totrueonce the TODO blocks are filled in. - Local-only execution per APPROACH.md section 5 rule 12: every step in the generated skeleton must be runnable on the local workstation. No remote URLs, no managed services.
Cross-references
- ADR anchor:
adrs/drafts/dynamic-verification-layer.mdslice 4 - Runner:
lineage/_extractor/probe-runtime/runner.py(executes the probes once filled in) - Existing probes for shape reference:
lineage/odd-platform/probes/P-001.yaml...P-009.yaml - Sidecar merge:
merge_probe_into_sidecars()inrunner.py— fires after PASS/FAIL and consults the probe's cross_references to update per-node sidecars
Implementation note
This skill orchestrates a small Python helper that handles the feature-flows.yaml parse + the next-ID derivation. Inline implementation lives in the skill body when invoked:
# Pseudocode:
import yaml, sys, re
from pathlib import Path
workspace = Path(".")
ff_path = workspace / "lineage" / "odd-platform" / "feature-flows.yaml"
docs = list(yaml.safe_load_all(ff_path.read_text()))
body = docs[1]
features = body["features"]
# Find next probe ID
existing_probes = sorted(p.name for p in (workspace / "lineage" / "odd-platform" / "probes").glob("P-*.yaml"))
max_n = max(int(re.match(r"P-(\d+)", n).group(1)) for n in existing_probes) if existing_probes else 0
next_n = max_n + 1
# For each feature × test_class combination requested, generate skeleton
# at lineage/odd-platform/probes/P-{next_n:03d}.yaml with the template above.The skill body is intentionally light — it's a wrapper around the YAML write. No new subagent needed; the maintainer can fill in TODOs interactively after generation.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

