Healthcare Eval Harness

Use this skill to check a health care app before release. Patient safety comes first.

One failed CRITICAL test must stop the release. Never skip or lower a CRITICAL gate.

Credit: This skill was contributed by Dr. Keyur Patel of Health1 Super Speciality Hospitals.

Test tools: The examples use Jest. You may use Vitest, pytest, PHPUnit, or another test tool. Keep the same test groups and pass rules.

When to Use

Use this skill:

• Before each release of an EMR or EHR app
• After a change to drug, dose, alert, or score rules
• After a patient data schema change
• After an access or login change
• After a merge conflict in clinical code
• When you add or change a CI safety gate
• After a vendor, HL7, FHIR, lab, or API change
• After a bug that could affect care or patient data

Also run the tests after changes to test data, time zones, units, code sets, or clinical rule files.

Safety Rules

• Treat missing tests as a failure.
• Treat a crashed or timed-out test as a failure.
• Do not hide a failed command with || true unless the next step reads a valid test report and applies the right gate.
• Use fixed, fake patient data in tests.
• Never put real PHI in test files, logs, reports, or build files.
• Test the real clinical logic. Do not replace it with a mock.
• Pin key tools and rule data to known versions.
• Record the commit, test time, rule version, and result.
• Do not call a release safe based on test coverage alone.
• Get clinical review when rules or expected results change.

This harness helps find risk. It does not prove that an app is safe, approved, or legal.

Gate Rules

Run all five groups.

A HIGH gate below 95% must block the normal release. A named reviewer may accept the risk only if your release policy allows it. Save the reason, owner, and end date. Never use this rule to pass a CRITICAL gate.

Use test counts for the 95% rule:

pass rate = passed tests / all tests * 100

Skipped, pending, and todo tests do not count as passed. If the report cannot show them, fail the gate.

Test Groups

1. CDSS Accuracy

Level: CRITICAL

Test:

• Drug pairs in both orders
• Drug and allergy checks
• Dose limits
• Age, weight, kidney, and liver rules
• Unit changes and rounding
• Missing, zero, bad, and very large values
• Clinical scores against the source rule
• Rule edges, such as one value below, at, and above a limit
• No false negative for each known unsafe case
• Clear failure when a rule cannot run
• Rule and code-set versions

npx jest --testPathPattern='tests/cdss' --bail --ci --coverage

2. PHI Exposure

Level: CRITICAL

Test:

• API errors and error pages
• App and server logs
• URLs and query text
• Browser storage and cache
• Export files and file names
• Email, alert, and job text
• Access without login
• Access after logout or token end
• Role and site access
• Cross-patient and cross-site access
• Service keys and secret values
• Test reports and build files

npx jest --testPathPattern='tests/security/phi' --bail --ci

3. Data Integrity

Level: CRITICAL

Test:

• Locked and signed visits
• Audit records
• Delete and link rules
• No records without a valid parent
• Two users editing at once
• Retry and repeat requests
• Failed writes and rollbacks
• Date, time zone, and daylight saving changes
• Unit and code changes
• Backup restore and data move steps when in scope

npx jest --testPathPattern='tests/data-integrity' --bail --ci

4. Clinical Workflow

Level: HIGH

Test full user flows, such as:

• Start, update, sign, and reopen a visit
• Draw and save a template
• Build and change a medicine set
• Search for a drug or diagnosis
• Make a prescription PDF
• Show and clear a red flag
• Save a draft after a network error
• Use a keyboard and screen reader for key safety steps

npx jest --testPathPattern='tests/clinical' --ci

5. Integration Compliance

Level: HIGH

Test:

• HL7 v2 message reading
• FHIR resource checks
• Lab name, code, unit, and range maps
• Bad, cut-off, late, and repeat messages
• Missing fields
• Unknown codes
• Time zones and text encoding
• Retry, timeout, and out-of-order events
• Vendor version changes

npx jest --testPathPattern='tests/integration' --ci

Check a HIGH Gate

This script needs jq. It uses whole numbers, so it does not need bc. It fails if no tests run, the test report is bad, any test is skipped, or the pass rate is under 95%.

set -u

tmp_json=$(mktemp)
trap 'rm -f "$tmp_json"' EXIT

npx jest --testPathPattern='tests/clinical' \
  --ci \
  --json \
  --outputFile="$tmp_json" || test_exit=$?

if ! jq -e . "$tmp_json" >/dev/null 2>&1; then
  echo "No valid clinical test report" >&2
  exit 1
fi

total=$(jq -r '.numTotalTests // 0' "$tmp_json")
passed=$(jq -r '.numPassedTests // 0' "$tmp_json")
pending=$(jq -r '.numPendingTests // 0' "$tmp_json")

if [ "$total" -eq 0 ]; then
  echo "No clinical tests found" >&2
  exit 1
fi

rate_x100=$((passed * 10000 / total))
printf 'Clinical pass rate: %d.%02d%% (%d/%d)\n' \
  "$((rate_x100 / 100))" "$((rate_x100 % 100))" "$passed" "$total"

if [ "$pending" -ne 0 ]; then
  echo "Clinical gate failed: $pending tests were skipped" >&2
  exit 1
fi

if [ "$rate_x100" -lt 9500 ]; then
  echo "Clinical gate failed: pass rate is below 95%" >&2
  exit 1
fi

if [ "${test_exit:-0}" -ne 0 ] && [ "$passed" -ne "$total" ]; then
  echo "Clinical gate failed" >&2
  exit 1
fi

Change tests/clinical and the labels to use this script for the integration gate.

CI Example

name: Healthcare Safety Gate

on:
  push:
  pull_request:

jobs:
  safety-gate:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: npm

      - run: npm ci

      - name: CDSS Accuracy
        run: >
          npx jest
          --testPathPattern='tests/cdss'
          --bail
          --ci
          --coverage
          --coverageThreshold='{"global":{"branches":80,"functions":80,"lines":80}}'

      - name: PHI Exposure
        run: >
          npx jest
          --testPathPattern='tests/security/phi'
          --bail
          --ci

      - name: Data Integrity
        run: >
          npx jest
          --testPathPattern='tests/data-integrity'
          --bail
          --ci

      - name: Clinical Workflow
        run: ./scripts/check-high-gate.sh tests/clinical "Clinical Workflow"

      - name: Integration Compliance
        run: ./scripts/check-high-gate.sh tests/integration "Integration Compliance"

Put the HIGH gate script in scripts/check-high-gate.sh. Make the test path and group name its two inputs. Pin action versions to full commit IDs if your build rules call for it.

Concrete Usage Example

A dose rule changes from 10 mg/kg to 8 mg/kg for children with low kidney function.

1. Add tests for 7.99, 8.00, and 8.01 mg/kg.
2. Test missing weight, zero weight, and weight in pounds.
3. Test low kidney function at each rule edge.
4. Test that the alert names the right dose and unit.
5. Run all three CRITICAL groups.
6. Run both HIGH groups.
7. Ask a clinical reviewer to check the new expected results.
8. Save a report with the commit and rule source.
9. Release only when every CRITICAL gate passes and each HIGH gate is at least 95%.

Example command:

npx jest --testPathPattern='tests/cdss' --bail --ci --coverage &&
npx jest --testPathPattern='tests/security/phi' --bail --ci &&
npx jest --testPathPattern='tests/data-integrity' --bail --ci &&
./scripts/check-high-gate.sh tests/clinical "Clinical Workflow" &&
./scripts/check-high-gate.sh tests/integration "Integration Compliance"

Report Form

# Healthcare Safety Check

- Date: 2026-03-27
- Commit: abc1234
- Rule version: 2026.03
- Test data: fake patients only
- Clinical reviewer: Name or pending

| Group | Total | Passed | Skipped | Status |
|---|---:|---:|---:|---|
| CDSS Accuracy | 39 | 39 | 0 | PASS |
| PHI Exposure | 8 | 8 | 0 | PASS |
| Data Integrity | 12 | 12 | 0 | PASS |
| Clinical Workflow | 22 | 21 | 0 | PASS, 95.45% |
| Integration | 6 | 6 | 0 | PASS |

Coverage: 84%
Open risks: None
Result: Ready for release review

Do not write SAFE TO DEPLOY when a review, approval, or known risk is still open.

Do Not

• Skip tests because they passed before
• Set a CRITICAL pass rule below 100%
• turn off fast failure for CRITICAL tests
• Mock the main clinical engine in safety tests
• Release while a CRITICAL gate is red
• Ignore missing, skipped, timed-out, or crashed tests
• Use real patient data
• Change expected results only to make a test pass
• Trust coverage as proof of safe care
• Hide a HIGH gate failure without a named review