Job Scheduler — Development with Claude

Project Context

A distributed, cron-capable job scheduler with priority queueing, worker pools, and at-least-once execution. The hard problems are the classic distributed-systems ones: exactly one scheduler must decide what to run (or jobs double-fire), a crashed worker mid-job must not lose the work (or jobs silently vanish), and a poison job must not retry forever (or it starves the pool). The design answers these with Redis leader election, a visibility-timeout queue, and bounded exponential-backoff retries with a dead-letter queue.

Learning goals: distributed coordination (leader election, distributed locks), at-least-once delivery with recovery, priority scheduling on a sorted set, and failure isolation with circuit breakers.

Architecture at a Glance (what actually runs)

Two datastores and several independently-scaled process roles — this matches docker-compose.yml and backend/package.json:

Supporting libraries that matter: cron-parser (computes next_run_time for recurring jobs), opossum (per-handler circuit breakers), prom-client (17+ metrics at GET /metrics), pino (structured logs). Built-in handlers: http.webhook, shell.command, system.cleanup, and test.echo/delay/log.

Key Design Decisions

1. Single active scheduler via Redis leader election

LeaderElection uses SET key instanceId EX 30 NX to claim leadership, refreshes it on a heartbeat at TTL/3 (~10s), and releases it with a Lua compare-and-delete so it can never delete another instance's lock. Only the leader scans for due jobs. Trade-off: this makes scheduling a single-writer operation (no duplicate enqueues), but leadership failover has a gap — if the leader dies, up to ~30s (the lock TTL) can pass before a standby claims the lock and scanning resumes. That's acceptable because jobs are time-windowed, not sub-second, and a brief scheduling pause is far cheaper than the double-firing a multi-leader design would cause.

2. At-least-once via a visibility-timeout queue

enqueue does ZADD with an inverted-priority score (Date.now() - priority*1_000_000) so higher priority dequeues first via ZPOPMIN; dequeue atomically pops and records the item in the :processing set with a future timeout score. A completed job is ZREM'd; a job whose worker crashes leaves its :processing entry, and the scheduler's recovery loop (ZRANGEBYSCORE … -inf now) re-enqueues anything past its timeout. Trade-off: this is at-least-once, not exactly-once — a worker that stalls past the visibility timeout but is still alive can have its job re-run, so handlers must be idempotent. We accept possible duplicates to guarantee no lost executions, which is the right bias for a scheduler.

3. Bounded exponential backoff + dead-letter queue

Retries follow min(initial_backoff_ms * 2^attempt, max_backoff_ms) up to max_retries; the scheduler's retry loop creates a fresh job_executions row (attempt+1) rather than mutating the old one, preserving full attempt history. After the retry budget is exhausted the item is LPUSH'd to the dead-letter list. Trade-off: a genuinely broken job is parked for human inspection instead of retried forever — we give up automatic self-healing for that job in exchange for not letting one poison payload burn worker capacity.

4. Per-handler circuit breakers (Opossum)

Each handler type gets its own breaker (opens at 50% error rate over 5 requests, tests recovery after 30s). If the webhook target is down, its breaker opens and those jobs are requeued for later rather than counted as failures — isolating one bad dependency from the shell/log handlers. Trade-off: added state and tuning per handler, justified because a single failing external endpoint would otherwise drain every worker slot.

5. Separate scheduler and worker processes

Scheduling (leader-only, coordination-heavy) and execution (embarrassingly parallel, stateless) scale on different axes, so they're different entry points (scheduler/index.ts, worker/index.ts) sharing the same codebase. Trade-off: more processes to run locally, bought back by being able to add workers without touching the scheduler.

Current State

Implemented and running end to end: job CRUD + pause/resume/trigger, cron and one-shot scheduling, the leader-elected scheduler with all four loops (scan, stalled-recovery, retry, metrics), three workers with bounded concurrency, the reliable priority queue with visibility timeout and DLQ, exponential-backoff retries, per-handler Opossum circuit breakers, a full prom-client metric set at GET /metrics, structured pino logging, and the React dashboard (jobs / executions / workers / DLQ views).

Note: monitoring is implemented, not pending — the earlier checklist-style CLAUDE.md listed "Add monitoring" as "Not started," which contradicted src/shared/metrics.ts and the Opossum breakers already emitting metrics.

Auth is session-based and enforced on every /api/v1 route. Users live in Redis (job_scheduler:user:*), not in the Postgres users table — that table is seeded but nothing reads it, which is a wart worth knowing about. ensureAdminUser() creates admin / admin123 on API boot (override with ADMIN_USERNAME / ADMIN_PASSWORD); that is the login the dashboard and the screenshot harness use.

Intentionally omitted: DAG/job-dependency workflows, multi-tenancy (jobs are single-tenant), request rate limiting, and a comprehensive automated test suite.

Iteration & Repair Log

• Metrics + circuit-breaker pass. src/shared/metrics.ts and src/shared/circuit-breaker.ts were added, wiring prom-client and Opossum through the scheduler/worker/API and instrumenting the queue, scheduler leadership, and per-handler breaker state. This is what makes the old "monitoring not started" note obsolete.
• DB layer split. The repository was broken into job-repository.ts, execution-repository.ts, schedule-repository.ts, and queries.ts under src/db/ for focus.
• Migrate-at-startup + dual scripts. migrate() is idempotent (CREATE … IF NOT EXISTS) and is called both from the scheduler on boot and via npm run migrate / npm run db:migrate (both aliases exist). Seeding is npm run seed / npm run db:seed.
• 2026-07-29 — the project did not actually run; six separate faults. It had zero screenshots, and each fix uncovered the next:
  1. Every backend service failed to boot. tsconfig sets "module": "commonjs", but 17 files used ESM-style .js import suffixes (from '../db/repository.js'), which CommonJS require cannot resolve — so api, scheduler, and all workers died on their first import with Cannot find module. migrate.ts was the one file using extensionless imports, which is why migrations appeared to work while nothing else did. 67 imports rewritten to match the module system.
  2. A stale compiled vite.config.js shadowed vite.config.ts. tsconfig.node.json is a composite project with no outDir, so npm run build emitted the compiled config next to the source, and Vite resolves .js before .ts. The dev server kept starting on the old port 3000 no matter what the TypeScript said. Fixed by pointing the emit at node_modules/.tmp.
  3. docker-compose up -d started the whole app, not just infrastructure, binding host 3001/3000 before the local dev servers could. The health check passed against the container, hiding it. App services now sit behind profiles: ["app"].
  4. The frontend had no login screen at all while every /api/v1 route requires a session, so the dashboard could never load data. Added routes/Login.tsx, an auth store, and a gate in Layout keyed on authChecked (not !user, which flashes the login form on every reload).
  5. The dashboard called /api/v1/metrics, which does not exist — the route is /api/v1/metrics/system, and bare /metrics is the Prometheus scrape endpoint. The 404 became a null metrics object, so every stat card rendered 0 while the job table beneath it was full.
  6. db-seed/seed.sql had an INSERT missing a value (weekly-report-generation omitted next_run_time), which aborted the transaction and left the database empty.
• npm run dev now starts the real topology (api + scheduler + 3 workers via concurrently), matching what docker-compose describes. Previously it ran a single all-in-one process, so the Workers page could only ever show one worker; the seeded execution history references worker-1/2/3. The old single-process entry point is still available as dev:all-in-one.
• CLAUDE.md rewrite (earlier pass). Replaced the generic Phase 1–4 checklist ("Phase 3: Not started", "will be updated throughout the development process") with the real architecture and decision rationale grounded in scheduler/index.ts, queue/reliable-queue.ts, and queue/leader-election.ts.

Open Questions

1. Job dependencies: to support DAG workflows, add a dependencies field and trigger downstream jobs on completion — but how to detect and reject cycles at definition time?
2. Failover gap: the ~30s leadership TTL bounds worst-case scheduling latency after a leader crash. Is a shorter TTL (faster failover, more Redis chatter) or a standby pre-warm worth it?
3. Exactly-once: could a per-execution distributed lock (DistributedLock already exists) plus a "completed" marker in Postgres upgrade at-least-once toward effectively-once for non-idempotent handlers?
4. Queue durability: the queue lives in Redis; on Redis loss, in-flight queue state is gone though job_executions in Postgres survives. Is a rebuild-from-Postgres recovery path worth adding?

Resources

• Designing a Distributed Job Scheduler
• Redis distributed locks — the SET NX EX + Lua pattern used for leader election
• cron-parser — next-run computation
• Opossum circuit breaker — per-handler failure isolation