system-health-monitoring
Skill0 starsChanged 7 months agoMITView on GitHub
Heartbeat file + cron watchdog + push notification alerts for long-running services. Use when deploying any background service, when user says "monitoring", "heartbeat", "watchdog", "health check", or needs to know if a service is alive.
In this repository37 other files it gives its agents.
- ablation-study-frameworkSkill
- async-http-websocket-patternsSkill
- brain-dumpSkill
- chaos-controlled-behavior-systemSkill
- daily-digestSkill
- database-reset-and-seedSkill
- duckdb-analytics-layerSkill
- event-driven-backtest-engineSkill
- event-study-full-pipelineSkill
- event-study-stats-suiteSkill
- failure-learning-systemSkill
- fastapi-sse-live-dashboardSkill
- goal-task-decompositionSkill
- high-information-gain-interviewerSkill
- idea-extraction-batch-pipelineSkill
- idea-grouping-taxonomySkill
- interview-belief-extractorSkill
- journal-captureSkill
- journal-reviewSkill
- keyword-filter-frameworkSkill
- langgraph-agentic-pipelineSkill
- llm-as-judge-eval-frameworkSkill
- multi-tab-dashboard-rendererSkill
- multi-tier-memory-systemSkill
- parquet-storage-with-walSkill
- project-insightsSkill
- pydantic-typed-config-loaderSkill
- quality-trend-detectionSkill
- research-gap-finderSkill
- safe-commitSkill
- simtom-knowledge-isolationSkill
- statistical-signal-detectionSkill
- streaming-dataset-pipelineSkill
- structured-data-extraction-pipelineSkill
- structured-logging-setupSkill
- textual-tui-reactiveSkill
- vector-db-ingestionSkill
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 254
Crash triggers shutdown: Use add_done_callback on tasks so a crash in any task triggers clean shutdown of all tasks.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
System Health Monitoring
Implement heartbeat files, health status tracking, and external watchdog alerting for long-running background services.
When to Use
- Deploying a long-running service (collector, worker, daemon)
- Need to detect stalled or crashed processes
- Want push notifications when a service goes down
- Building a health endpoint or status page
Architecture
Service Process
|-- REST poll task
|-- WebSocket task
|-- Heartbeat task <-- writes JSON file every N seconds
|
v
data/.heartbeat <-- JSON file on disk
|
v
Cron watchdog <-- reads file, alerts if stale
|
v
Push notification <-- ntfy, Pushover, email, etc.Step-by-Step Implementation
Step 1: Write a rich heartbeat file from your service
The heartbeat is not just a timestamp. Include operational metrics so the watchdog (or a dashboard) can assess health at a glance.
import asyncio
import json
import time
from pathlib import Path
class HealthMonitor:
"""Writes periodic heartbeat files with operational metrics."""
def __init__(self, heartbeat_path: Path, interval_sec: int = 30) -> None:
self._path = heartbeat_path
self._interval = interval_sec
self._path.parent.mkdir(parents=True, exist_ok=True)
# Metrics — updated by other components
self.active_items: int = 0
self.pending_rows: int = 0
self.last_poll_ts: float = 0.0
self.last_event_ts: float = 0.0
self.error_count: int = 0
async def run(self) -> None:
"""Heartbeat loop — run as an asyncio task."""
while True:
self._write()
await asyncio.sleep(self._interval)
def _write(self) -> None:
self._path.write_text(json.dumps({
"ts": time.time(),
"active_items": self.active_items,
"pending_rows": self.pending_rows,
"last_poll": self.last_poll_ts,
"last_event": self.last_event_ts,
"error_count": self.error_count,
}))Step 2: Launch the heartbeat as a background task
import asyncio
import signal
class ServiceRunner:
def __init__(self, data_dir: Path) -> None:
self._health = HealthMonitor(data_dir / ".heartbeat")
async def run(self) -> None:
stop_event = asyncio.Event()
loop = asyncio.get_running_loop()
for sig in (signal.SIGINT, signal.SIGTERM):
loop.add_signal_handler(sig, stop_event.set)
tasks = [
asyncio.create_task(self._main_work_loop(), name="worker"),
asyncio.create_task(self._health.run(), name="heartbeat"),
]
# If any task crashes, trigger shutdown
def _on_crash(task: asyncio.Task) -> None:
if not task.cancelled() and task.exception():
stop_event.set()
for task in tasks:
task.add_done_callback(_on_crash)
try:
await stop_event.wait()
finally:
for task in tasks:
task.cancel()
await asyncio.gather(*tasks, return_exceptions=True)Step 3: Read heartbeat from a dashboard or health endpoint
import time
from pathlib import Path
def check_service_health(heartbeat_path: Path) -> dict:
"""Read heartbeat file and determine service status."""
if not heartbeat_path.exists():
return {"status": "unknown", "heartbeat_age_sec": None}
try:
data = json.loads(heartbeat_path.read_text())
except (json.JSONDecodeError, OSError):
return {"status": "error", "heartbeat_age_sec": None}
age = time.time() - data.get("ts", 0)
if age < 60:
status = "running"
elif age < 300:
status = "stale"
else:
status = "dead"
return {
"status": status,
"heartbeat_age_sec": round(age, 1),
"active_items": data.get("active_items", 0),
"pending_rows": data.get("pending_rows", 0),
"error_count": data.get("error_count", 0),
}Step 4: Create a cron watchdog script
This runs every 5 minutes via cron. If the heartbeat is stale, it sends an alert.
#!/usr/bin/env python3
"""Watchdog: check heartbeat file, alert if stale."""
import json
import sys
import time
from pathlib import Path
HEARTBEAT_PATH = Path("data/.heartbeat")
MAX_AGE_SEC = 120 # Alert if older than 2 minutes
NTFY_TOPIC = "my-service-alerts" # or use Pushover, email, etc.
def check_and_alert() -> None:
if not HEARTBEAT_PATH.exists():
send_alert("Heartbeat file missing — service may not be running")
sys.exit(1)
try:
data = json.loads(HEARTBEAT_PATH.read_text())
except Exception:
send_alert("Heartbeat file corrupted")
sys.exit(1)
age = time.time() - data.get("ts", 0)
if age > MAX_AGE_SEC:
send_alert(
f"Service heartbeat stale ({age:.0f}s old). "
f"Last poll: {data.get('last_poll', 'unknown')}, "
f"Errors: {data.get('error_count', '?')}"
)
sys.exit(1)
print(f"OK: heartbeat {age:.0f}s old")
def send_alert(message: str) -> None:
"""Send push notification via ntfy.sh (free, no signup)."""
import urllib.request
try:
req = urllib.request.Request(
f"https://ntfy.sh/{NTFY_TOPIC}",
data=message.encode(),
headers={"Title": "Service Alert", "Priority": "high"},
)
urllib.request.urlopen(req)
print(f"ALERT sent: {message}")
except Exception as e:
print(f"Failed to send alert: {e}", file=sys.stderr)
if __name__ == "__main__":
check_and_alert()Step 5: Set up the cron job
# Check every 5 minutes
*/5 * * * * cd /path/to/project && /path/to/python watchdog.py >> /var/log/watchdog.log 2>&1Step 6: Display health in a dashboard
def format_health_display(health: dict) -> str:
"""Format health data for display."""
status = health["status"]
age = health["heartbeat_age_sec"]
if status == "running":
status_line = f"RUNNING (heartbeat {age:.1f}s ago)"
elif status == "stale":
status_line = f"STALE (heartbeat {age:.1f}s ago — may be stuck)"
elif status == "dead":
status_line = f"DEAD (heartbeat {age:.1f}s ago)"
else:
status_line = "UNKNOWN (no heartbeat file)"
return (
f"Status: {status_line}\n"
f"Active items: {health.get('active_items', '?')}\n"
f"Pending rows: {health.get('pending_rows', '?')}\n"
f"Errors: {health.get('error_count', '?')}"
)Key Principles
- Rich heartbeats: Include operational metrics (active items, pending work, error counts), not just a timestamp. This lets you diagnose problems without SSH-ing in.
- Three-tier status:
running(<60s),stale(<300s),dead(>300s). Adjust thresholds based on your service's expected heartbeat interval. - File-based, not network-based: Heartbeat files are simpler and more reliable than HTTP health endpoints for background services. No port conflicts, no firewall issues.
- Error isolation: The heartbeat task is independent. If the main work loop crashes, the heartbeat stops updating, and the watchdog detects it.
- Crash triggers shutdown: Use
add_done_callbackon tasks so a crash in any task triggers clean shutdown of all tasks. This way systemd/supervisord can restart the whole service.
Dependencies
Minimal — only stdlib for the core pattern. Optional: ntfy.sh (free push notifications, no signup).
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

