agentleFS
nhattrung0911nhattrung0911/shipwright/skills/operating-production-services/SKILL.md

operating-production-services

Skill0 starsChanged 5 months agoMITView on GitHub

Use when a system will run live and must stay up — designing traffic controls (rate limiting, throttling, timeouts, retries, circuit breakers, idempotency, caching, graceful shutdown, autoscaling) and Day-2 operations/maintenance (monitoring, alerting, SLO, on-call, incident response, deploys/rollback, backups, DB upkeep, dependency patching, log/data retention, cost). Triggers before launch and for any "how do we keep it running" concern. Stack-agnostic.

4 other projects use a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 1 is distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Do NOT call a live service production-ready until:

    line 57

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Operating Production Services

Overview

Building it is half the job; keeping it alive under real traffic and over time is the other half. This skill covers two areas the build checklist only name-drops: runtime reliability controls and Day-2 operations/maintenance.

Core principle: Assume everything fails — slow networks, traffic spikes, bad deploys, abusive clients, aging dependencies. Design controls so failure degrades gracefully and recovery is routine, not heroic.

Announce when relevant: "Using operating-production-services for runtime controls and maintenance."

A. Runtime Reliability Controls

ControlWhat / howBar to clear
Rate limitingCap requests per client key (user/IP/API-key). Algorithm: token bucket or sliding window. Return 429 + Retry-After header. Apply at edge/gateway AND sensitive endpoints (login, signup, write APIs)Abusive client throttled, normal user unaffected; limits documented
Throttling / quotasPer-tenant quotas; backpressure when overloaded; load shedding (drop low-priority work) before total collapseSystem degrades, doesn't crash, under overload
TimeoutsEvery outbound call (DB, API, cache) has a timeout. No unbounded waitsNo request hangs forever
RetriesRetry only idempotent ops; exponential backoff + jitter; cap attempts; never retry 4xx except 408/429 (honor Retry-After)No retry storms; transient errors recover
Circuit breakerTrip open after repeated downstream failures; fail fast; half-open probe to recoverOne sick dependency doesn't cascade
IdempotencyIdempotency keys on writes/payments so retries don't double-charge/double-createDuplicate request = single effect
CachingClient/CDN/server/DB layers; explicit invalidation + TTL; stampede protectionCache never serves stale silently; no thundering herd
Graceful shutdownOn deploy/restart: stop taking new work, drain in-flight, close connections cleanlyZero dropped requests on redeploy
Health/readinessLiveness (am I up) + readiness (can I serve) + startup probe for slow-init apps; load balancer respects themUnready instance gets no traffic; slow boot not killed
AutoscalingScale on real signal (CPU/queue depth/latency); min/max bounds; scale-in cooldown/stabilization to prevent flappingSpikes absorbed; no thrash; no runaway cost
Edge protectionWAF, DDoS mitigation, bot filtering at the perimeterCommon attacks blocked before app

B. Day-2 Operations & Maintenance

AreaWhat / cadenceBar to clear
MonitoringMetrics (latency, error rate, throughput, saturation — the "golden signals") + dashboardsCan answer "is it healthy right now" in 10s
AlertingAlert on symptoms (SLO burn-rate breach), not every blip; every alert maps to a runbook; route to a person; no alert fatigueReal problems page someone; noise suppressed
TracingDistributed tracing with propagated request IDs across services, not just metrics/logsCan localize cross-service latency/errors
SLO / error budgetDefine target (e.g. 99.9% / p95 latency); track budget; freeze risky changes when budget spentReliability is measured, not vibes
On-call & runbookWho responds; written runbook per common failure (symptom → diagnosis → fix)New engineer can resolve a page from the runbook
Incident responseDetect → mitigate → communicate (status page) → blameless postmortem with action itemsEvery incident produces a prevention task
DeploysProgressive: canary or blue-green; automated rollback on error-rate spike; never big-bang prodBad deploy auto-reverts; no manual ftp
Schema migrationsBackward-compatible expand→migrate→contract; no locking DDL on hot tables; decoupled from app deploy; reversibleMigration can't break running old code; no long table lock
Disaster recoveryDefine RTO/RPO; test region/AZ failover; backups ≠ DR; documented recovery procedureFailover drill meets RTO/RPO
Backups & restoreAutomated backups, encrypted at rest + access-controlled; periodically test restore (a backup never restored = no backup); offsite copyRestore drill passes on a schedule
Cert & secret rotationAutomate TLS cert renewal; alert on cert/secret expiry N days out; rotate secrets on scheduleNo expired-cert outage; rotation tested
Queue / async healthDead-letter queue for poison messages past retry cap; alert on DLQ depth; monitor lagStuck messages isolated, not silently lost
Database upkeepIndex health, vacuum/analyze, slow-query review, connection-pool limits, archival of old rowsQueries stay fast as data grows
Dependency patchingRegular CVE scan + update cadence; pin + lockfile; test before bumping; track EOL/deprecatedNo unpatched critical CVE lingering
Log & data retentionLog rotation + retention policy; scheduled cleanup jobs for expired/temp data; PII retention limitsDisks don't fill; data isn't hoarded forever
Feature flagsDecouple deploy from release; kill-switch for risky features; clean up stale flagsCan disable a feature without redeploy
Cost monitoringOngoing spend dashboards + budget alarms; catch runaway resources/queriesNo surprise bill; cost tracked like latency
Capacity & deprecationPeriodic capacity review vs growth; planned sunset path for old APIs/versions with noticeScale ahead of the wall; clean retirement

Pre-Launch Ops Gate

Do NOT call a live service production-ready until:

  • Rate limiting + timeouts + retries (with backoff) on critical paths
  • Graceful shutdown + health/readiness probes wired to the load balancer
  • Monitoring dashboard + alerting to a real person live BEFORE launch
  • Runbook for top failures; on-call owner named
  • Progressive deploy + automated rollback tested
  • Backups automated, encrypted, AND a restore drill passed
  • DR: RTO/RPO defined; failover tested
  • TLS cert auto-renew + expiry alerting; secret rotation set
  • Distributed tracing wired; DLQ for async work
  • Dependency CVE scan clean; patch cadence set
  • Log retention + scheduled data-cleanup jobs configured
  • Cost budget alarms set

Common Mistakes

MistakeFix
Rate limit only on login, not write APIsLimit every abusable endpoint; return 429 + Retry-After
Retry without backoff/jitterRetry storm DOSes your own backend — add backoff + jitter, cap attempts
Retrying non-idempotent writesAdd idempotency keys first, or don't retry
"We have backups" (never restored)A backup never test-restored is not a backup — drill it
Alert on everythingAlert fatigue hides real fires — alert on SLO symptoms
Big-bang prod deployCanary/blue-green + auto-rollback
Caching with no invalidation planDefine TTL + invalidation before adding cache

Companion skills

  • Overall gate → disciplined-delivery (its "observable" + DoD reference this)
  • Web build → shipping-production-websites (pillars 9/10/13 = this skill's depth)
  • Security perimeter overlaps → securing-applications (rate-limit also a security control; WAF, A09 logging)

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.