live-verify

Retired. The harness and its test accounts were torn down 2026-10-03, so there is nothing to wake. Live verification against Snowflake, Unity Catalog, Azure or AWS waits on a cloud test instance (#2224); until then, say a change is unverified live rather than run this. The steps below are kept for when an instance exists.

Purpose

Some claims cannot be settled by a unit test, because the thing that decides them is on the other side of a driver. CLAUDE.md §13 states it as a rule:

For anything whose type or value crosses a driver boundary, only a live run is evidence.

The cost of ignoring it is recorded: Unity Catalog freshness had never worked once (#953) while three suites ran green, because the Databricks connector returns a TIMESTAMP's MAX as a str. Parquet freshness was broken on every Parquet file (#520) because Arrow-backed dtypes fail is_datetime64_any_dtype. Both survived a full CI suite. Both died in the first minute of a live run.

This skill is the repeatable procedure for getting that evidence.

Cost gate — read before waking anything

• The harness compute has been stopped by default since 2026-07-04 (#590). Awake it burns ~CAD 17/day (~0.70/hour); stopped it costs ~0.
• Azure runs as PayAsYouGo — awake resources bill real money. There IS a monthly account spend limit, and it fails in the worst way: on 2026-08-08 it killed the window agent mid-poll while the harness stayed Running unattended for ~6h (ops-log; extra burn ≈ CAD 4–5). Treat the limit as an agent-killer, not a cost ceiling.
• Therefore: never leave a window open. stop runs in every exit path, including when the verification fails. If you are interrupted, stop the harness first and report second. A window driven by an agent dies with the agent — the stop is not guaranteed (the 2026-08-08 incident): after any window, verify the stop against Azure (az containerapp list -g dataq-rg --query "[].{n:name,s:properties.runningStatus}" -o table), never against the script's own log alone.
• Before choosing a Snowflake-touching leg (--adf, --dbt, the Snowflake DAG): check the rotation register in docs/ops-log.md first. Short-lived Snowflake PATs (~15–25 days in practice) and ADF's stale inline ls_snowflake credential (2026-08-08 entry) can kill a leg before it starts — a dead credential then masquerades as the defect you came to verify.
• If the user has not clearly asked for a live run, ask before waking. A window is an outward-facing, billable action.

The harness script

~/Coding/Python/DataQ-harness/scripts/harness_window.sh — harness-side, not git-tracked (ADR 0021). Read its header before the first use in a session; the leg semantics below are summarised from it and the script is the authority.

harness_window.sh status                                    # read-only: app/trigger/job states
harness_window.sh start                                     # wake: redis → Airflow → workers → trigger, ADF triggers on
harness_window.sh run [--adf] [--dags] [--dbt] [--iceberg]  # kick the flows as MANUAL executions, re-suspend after
harness_window.sh stop                                      # sleep: reverse order, crons disarmed
harness_window.sh window [...]                              # start → run → stop, one shot

Pick the legs by what you need to exercise — each one costs time and money:

Full-cycle window takes ~11.5 min. run works standalone (the mockdata jobs do not depend on the Airflow apps).

Suspended-job semantics: an ACA job refuses manual starts, so run resumes each job, starts it, and re-suspends — nothing is left armed on a cron.

Procedure

1. Decide what evidence you need — before waking anything

Write down, explicitly, the reading you expect to observe and what would falsify it. Vague intent produces vague evidence, and a green suite is the thing that already fooled us.

Good: "UC freshness on retail.orders should return a float age in hours; a lower-case and an UPPER-case column name must return identical values." Bad: "check that UC works."

Choose the minimum legs that produce that reading.

2. Wake and run

~/Coding/Python/DataQ-harness/scripts/harness_window.sh status   # confirm the starting state
~/Coding/Python/DataQ-harness/scripts/harness_window.sh start
~/Coding/Python/DataQ-harness/scripts/harness_window.sh run [legs]

Run these in the background — a full window is ~11.5 min and macOS has no timeout.

3. Exercise the app against the live data

Pick what matches the change:

• Browser (deployed stack): E2E_LIVE_BASE_URL=https://<frontend-host> pnpm e2e in frontend/ — the opt-in frontend/e2e-live/ lane (#531). Never runs in CI.
• API (deployed stack):

  DATAQ_API=https://<frontend-host> DATAQ_BEARER=$(az account get-access-token \
      --resource api://<api-app-id> --query accessToken -o tsv) \
      python -m backend.scripts.e2e_smoke

  The frontend nginx proxies /api to the internal api. Note e2e_smoke.py proves the app layer, not the datasources — it is a precondition, not the evidence.
• MCP: the 4-query protocol smoke against live /mcp/ (trailing slash matters — docs/site/guides/mcp-setup.md).
• The actual reading: trigger the suite whose check you are verifying and read the result row — metric_value, observed_value, the run status. This is the evidence; everything above is scaffolding.

4. Judge the reading — the part that matters

For each expected reading:

• Is there a value at all? A check that reports "no reading" and stays green is the #953 failure mode. Absence is a red result, not a neutral one.
• Is the value plausible? #520's real signal was 531.8h — the harness genuinely had stopped producing. A number you cannot explain is not a pass.
• Do the case variants agree? Run the identifier lower-case and UPPER-case. Identical values prove the dialect quoting is right (#937); differing values prove it is not.
• Did it work for the right reason? If a run turned green after a fix, confirm the fix is why. A dead credential also produces a suspiciously quiet failure — and a datasource connection with a dead credential has no visible state until a run fails (#954), so check worker logs before concluding.

5. Sleep — always

~/Coding/Python/DataQ-harness/scripts/harness_window.sh stop
~/Coding/Python/DataQ-harness/scripts/harness_window.sh status   # confirm everything is down

stop also disarms the mockdata crons and the dbt-lineage nightly cron. Verify with status AND against Azure directly (az containerapp list …runningStatus…); do not assume — the script's own log is not evidence the stop happened.

6. Record the evidence

A live run whose result is not written down has to be paid for twice.

• docs/site/reference/feature-matrix.md — tick a cell only for a datasource you just observed working. A ✅ with no live run is a claim, not a fact.
• CLAUDE.md §13 — the reading and the date, with the actual numbers ("UC freshness 239.06h, lower/UPPER identical" beats "UC verified").
• A GitHub issue for anything the run found — working-agreement #3, never a silent fix.
• docs/ops-log.md — every harness start/stop gets an append-only entry with absolute UTC timestamps (CLAUDE.md §12; the settings.json hook will remind you, but write it as part of the window, not as an afterthought).
• If the run found a defect a test could not, say so explicitly and name the boundary. That sentence is what turns one incident into a rule.

Rules

• Never print or write a secret. Key Vault values go straight from command substitution into the consuming command — no masked echo, no scratch file. This is a standing constraint, not a preference.
• Never leave the window open. Stop before reporting.
• Never tick a docs cell you did not observe.
• The harness is demo-scoped. Credentials expire and self-signal via #419 alerting; recovery is re-mint + Key Vault update.

See also

• /agents driver-boundary-guard — finds the boundaries before the live run does, and tells you which claims a unit test genuinely cannot settle
• deploy/README.md — pre-deploy and post-deploy smoke checklists
• docs/site/operate/runbook-faq.md — the live-smoke runbook entry
• ADR 0021 — why the harness lives outside this repo