go-resilience
Skill7 starsChanged 4 days agoApache-2.0View on GitHub
Use when handling Go dependency failures or overload with retries, backoff/jitter, budgets, idempotency, duplicate delivery, circuit breakers, bulkheads, rate limits, backpressure, or degradation. Unknown root causes belong to go-troubleshooting.
In this repository25 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- go-code-refactorSkill
- go-code-reviewSkill
- go-codeSkill
- go-concurrencySkill
- go-contextSkill
- go-databaseSkill
- go-data-structuresSkill
- go-defensiveSkill
- go-documentationSkill
- go-error-handlingSkill
- go-functionsSkill
- go-genericsSkill
- go-httpSkill
- go-interfacesSkill
- go-lintingSkill
- go-loggingSkill
- go-namingSkill
- go-packagesSkill
- go-performanceSkill
- go-securitySkill
- go-style-coreSkill
- go-testingSkill
- go-troubleshootingSkill
The file
We split this skill into the rules it gives an agent
6 rules. About 0 are common, 3 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 43
Follow go-packages before adding a library.
- Distinctive: line 57
Use capped exponential backoff with jitter and a cancellation-aware wait.
- Distinctive: line 61
Use a fresh replayable request body for each attempt: the Transport may still be reading the previous one (go-http).
- Very specific: line 117
Use durable atomic claims and explicit in-progress/completed/unknown states for duplicate delivery.
- Very specific: line 161
Test the selected policy, not just eventual success.
- Distinctive: line 177
Use go-testing for test mechanics and go-linting for the actual repository gate.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Go Resilience
Compatibility: Baseline Go 1.27 (see COMPATIBILITY.md). The policies are version-independent;
testing/synctestrequires Go 1.25+.
Preserve operation semantics while bounding time, work, and failure impact. A timeout leaves the remote outcome uncertain; another attempt consumes real capacity and may repeat a side effect.
Resource Routing
references/RETRIES-AND-IDEMPOTENCY.md- Read for HTTP/RPC retries, Retry-After, request replay, ambiguous outcomes, duplicate events, or durable deduplication.references/LOAD-AND-DEGRADATION.md- Read for concurrency/rate admission, queue bounds, circuit breakers, recovery, fallback, or multi-tenant/fleet limits.
Start With the Contract
Identify the operation's side effects, replay guarantee, caller deadline, dependency capacity/quota, and required behavior when it cannot complete. Inspect existing SDK, transport, proxy, and queue policies before adding a wrapper. Defaults in different layers can multiply attempts.
Implement or review the protection the task needs; a routine client does not need every pattern below. Preserve explicit delivery, consistency, freshness, and security requirements.
Choose the Protection
| Problem | First mechanism | Boundary |
| A dependency call runs too long | End-to-end budget with bounded attempts | Cancellation does not undo remote effects |
| A transient failure on a replay-safe operation | Bounded retry with backoff/jitter | Permanent errors and ambiguous unsafe writes stop |
| Extra attempts sustain an outage | Retry budget and one coordinated owner | Per-call attempt count alone cannot cap fleet load |
| Too much simultaneous work | Bulkhead / bounded concurrency | Bound waiting work too; a semaphore is not a rate limit |
| Requests exceed a quota | Rate admission and bounded backpressure | State whether quota is per process, tenant, or fleet |
| A failing dependency needs recovery time | Circuit breaker if rejection adds value | It neither limits all concurrency nor makes replay safe |
| An optional dependency is unavailable | Contract-approved degraded result | No invented success, weaker authorization, or unmarked stale data |
Follow go-packages before adding a library.
golang.org/x/time/rate is an external Go module for token buckets; the standard
library has no general circuit breaker.
Retry Invariants
- Retry only when both the failure classification and operation contract permit replay. A 5xx or network timeout does not prove nothing happened; documented 429 handling is not forbidden merely because it is a 4xx.
- Count total attempts including the first. Bound attempts, elapsed time, and aggregate extra traffic; inventory unavoidable lower-layer attempts.
- Queueing, rate admission, backoff, attempts, and response processing share the caller's total budget. Each attempt derives from it and can have a shorter timeout. Do not reset the total budget or detach retries after cancellation.
- Use capped exponential backoff with jitter and a cancellation-aware wait. A valid server retry delay is a lower bound under this policy: do not shrink it to a local cap. If it cannot fit, stop or use an already-authorized durable rescheduling path; parsing overflow is not permission to retry earlier.
- Use a fresh replayable request body for each attempt: the Transport may still
be reading the previous one (go-http).
Finish/close a discarded response and release attempt resources before
waiting. Preserve the attempt context until the accepted response body is
consumed or explicitly handed off with cleanup ownership; returning from
Dodoes not finish the body. - Scope retry ownership where operation semantics are known. Application, SDK, mesh, and broker retries must compose to a bounded policy rather than retry independently. Every real attempt passes dependency admission and breaker gates.
The whole policy in one function — the caller classifies the failure, the loop owns the budget, the wait, and cancellation:
// retry makes at most attempts calls, with capped, jittered backoff between them.
// ctx's deadline is the only elapsed-time bound: without one, a server delay of
// any length is slept out, so callers give ctx a deadline. base is at most an
// hour: base<<20 must fit a Duration.
func retry(ctx context.Context, attempts int, base, maxWait time.Duration,
attempt func(context.Context) (retryAfter time.Duration, retryable bool, err error)) error {
for i := range attempts {
retryAfter, retryable, err := attempt(ctx)
if err == nil || !retryable || i == attempts-1 {
return err
}
wait := min(base<<min(i, 20), maxWait) // an uncapped shift overflows to a negative wait
wait = wait/2 + rand.N(wait/2+1) //nolint:gosec // G404: retry jitter, not a secret; within [wait/2, wait]
wait = max(wait, retryAfter) // Retry-After is a lower bound
if d, ok := ctx.Deadline(); ok && time.Until(d) <= wait {
return err // the wait outlasts the budget: stop now, do not sleep it out
}
if ctx.Err() != nil { // checked first: with a zero wait the select below picks at random
return errors.Join(ctx.Err(), err)
}
select {
case <-ctx.Done():
return errors.Join(ctx.Err(), err)
case <-time.After(wait):
}
}
return errors.New("retry: attempts < 1, nothing was tried") // not a success
}attempt closes a discarded response before returning (closing is enough:
go-http), and the permit for the dependency is
released before the wait, not held through it.
Idempotency and Delivery
A key is meaningful only when the receiver enforces it. Keep one key per logical operation, stable payload, and documented account/tenant scope and retention. A new attempt is not a new operation. For an already-sent request, recover the key actually sent; a newly chosen stable key cannot deduplicate an earlier one. An in-memory map is insufficient for cross-replica or restart-safe deduplication.
Use durable atomic claims and explicit in-progress/completed/unknown states for duplicate delivery. A lease expiring does not prove the previous worker stopped or the remote operation failed. SQL can atomically bind local effects to a receipt; a local transaction/outbox does not atomically commit an unrelated remote effect. If its outcome cannot be reconciled and replay is unsafe, expose that limit instead of promising exactly-once completion.
Deduplication is not ordering: at-least-once delivery also delivers late, and
an older event applied second reverts the row. An event that sets state
carries a version or source timestamp, and the write keeps the newest: the
UPDATE sets the new version and matches only WHERE id = … AND version <
the event's version, so an older event changes no row (the statement is in
RETRIES-AND-IDEMPOTENCY.md).
Overload and Recovery
Bound admission before spawning unbounded goroutines. Choose reject, wait, or durable backpressure from the caller's contract; do not silently drop work that must be retained. Both admission and waiting need cancellation/lifetime bounds. Isolate expensive tenants or dependencies without unbounded per-key state.
Release a dependency's per-attempt concurrency permit during backoff; retain a separate bound on pending logical work. Rate tokens govern attempts over time, not in-flight calls. Multiple local limiters do not enforce one global quota.
A breaker samples relevant dependency outcomes, limits half-open probes, and
ignores stale completions from earlier state generations. Only the
dependency's own failures count: a caller's context.Canceled, a local
rejection, and a validation error leave the breaker unchanged. A
DeadlineExceeded counts when the per-attempt timeout fired and not when the
caller's context had already ended (ctx.Err() != nil on the caller's ctx).
Coordinate recovery admission so retries cannot bypass an open breaker or flood
the recovering dependency. Choose windows/thresholds from traffic and failure
behavior, not universal magic numbers; with no traffic data they are
constructor parameters, and the report names the chosen values as assumptions.
Fallback is an alternate result with a contract: identity scope, freshness, consistency, and caller-visible degradation. Preserve security decisions and required side effects. Returning an empty list or cached allow on failure is not a neutral substitute. Route authorization rules to go-security.
Verify Failure Behavior
Test the selected policy, not just eventual success. Use controlled outcomes,
recorded attempt times/counts, injected randomness/clock or testing/synctest
(Go 1.25+) where suitable; real network I/O still needs appropriate test seams.
- Cancellation before admission, during backoff, and during the active call; exhausted deadlines start no further intended attempts.
- Nonretryable errors stop; safe transient errors recover within budget; server delay is honored; request bodies replay and response cleanup occurs.
- Concurrent duplicates, payload mismatch, and crash windows do not produce unsupported guarantees; test durable constraints with the actual database.
- Saturation bounds in-flight and queued work, preserves required delivery, and contains a noisy tenant. Recovery probes remain bounded under concurrency.
- Fallback preserves identity/freshness/error semantics; old breaker outcomes cannot change a newer state. Observe logical requests separately from attempts, retries exhausted/suppressed, queue wait/rejection, and degraded responses.
Use go-testing for test mechanics and go-linting for the actual repository gate. Race tests can expose in-process synchronization bugs; they do not establish distributed idempotency. Report executed results, unrun checks, and remaining contract limits honestly; follow go-style-core for report length. A proposed policy is not evidence of outage recovery.
Related Skills
- go-http: clients, transports, response handling, server shutdown.
- go-context and go-concurrency: cancellation propagation, goroutine lifetime, synchronization.
- go-database: transactions, constraints, pool and query mechanics.
- go-error-handling and go-logging: error matching and structured logs; no high-cardinality or secret metric labels.
- go-troubleshooting: incident evidence and diagnosis before speculative retry or breaker changes.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

