agentleFS
h0rn3th0rn3t/golang-skills/skills/go-resilience/SKILL.md

go-resilience

Skill7 starsChanged 4 days agoApache-2.0View on GitHub

Use when handling Go dependency failures or overload with retries, backoff/jitter, budgets, idempotency, duplicate delivery, circuit breakers, bulkheads, rate limits, backpressure, or degradation. Unknown root causes belong to go-troubleshooting.

The file

We split this skill into the rules it gives an agent

6 rules. About 0 are common, 3 are distinctive, and 3 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Follow go-packages before adding a library.

    line 43
  2. Distinctive:

    Use capped exponential backoff with jitter and a cancellation-aware wait.

    line 57
  3. Distinctive:

    Use a fresh replayable request body for each attempt: the Transport may still be reading the previous one (go-http).

    line 61
  4. Very specific:

    Use durable atomic claims and explicit in-progress/completed/unknown states for duplicate delivery.

    line 117
  5. Very specific:

    Test the selected policy, not just eventual success.

    line 161
  6. Distinctive:

    Use go-testing for test mechanics and go-linting for the actual repository gate.

    line 177

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Go Resilience

Compatibility: Baseline Go 1.27 (see COMPATIBILITY.md). The policies are version-independent; testing/synctest requires Go 1.25+.

Preserve operation semantics while bounding time, work, and failure impact. A timeout leaves the remote outcome uncertain; another attempt consumes real capacity and may repeat a side effect.

Resource Routing

  • references/RETRIES-AND-IDEMPOTENCY.md - Read for HTTP/RPC retries, Retry-After, request replay, ambiguous outcomes, duplicate events, or durable deduplication.
  • references/LOAD-AND-DEGRADATION.md - Read for concurrency/rate admission, queue bounds, circuit breakers, recovery, fallback, or multi-tenant/fleet limits.

Start With the Contract

Identify the operation's side effects, replay guarantee, caller deadline, dependency capacity/quota, and required behavior when it cannot complete. Inspect existing SDK, transport, proxy, and queue policies before adding a wrapper. Defaults in different layers can multiply attempts.

Implement or review the protection the task needs; a routine client does not need every pattern below. Preserve explicit delivery, consistency, freshness, and security requirements.

Choose the Protection

ProblemFirst mechanismBoundary
A dependency call runs too longEnd-to-end budget with bounded attemptsCancellation does not undo remote effects
A transient failure on a replay-safe operationBounded retry with backoff/jitterPermanent errors and ambiguous unsafe writes stop
Extra attempts sustain an outageRetry budget and one coordinated ownerPer-call attempt count alone cannot cap fleet load
Too much simultaneous workBulkhead / bounded concurrencyBound waiting work too; a semaphore is not a rate limit
Requests exceed a quotaRate admission and bounded backpressureState whether quota is per process, tenant, or fleet
A failing dependency needs recovery timeCircuit breaker if rejection adds valueIt neither limits all concurrency nor makes replay safe
An optional dependency is unavailableContract-approved degraded resultNo invented success, weaker authorization, or unmarked stale data

Follow go-packages before adding a library. golang.org/x/time/rate is an external Go module for token buckets; the standard library has no general circuit breaker.

Retry Invariants

  • Retry only when both the failure classification and operation contract permit replay. A 5xx or network timeout does not prove nothing happened; documented 429 handling is not forbidden merely because it is a 4xx.
  • Count total attempts including the first. Bound attempts, elapsed time, and aggregate extra traffic; inventory unavoidable lower-layer attempts.
  • Queueing, rate admission, backoff, attempts, and response processing share the caller's total budget. Each attempt derives from it and can have a shorter timeout. Do not reset the total budget or detach retries after cancellation.
  • Use capped exponential backoff with jitter and a cancellation-aware wait. A valid server retry delay is a lower bound under this policy: do not shrink it to a local cap. If it cannot fit, stop or use an already-authorized durable rescheduling path; parsing overflow is not permission to retry earlier.
  • Use a fresh replayable request body for each attempt: the Transport may still be reading the previous one (go-http). Finish/close a discarded response and release attempt resources before waiting. Preserve the attempt context until the accepted response body is consumed or explicitly handed off with cleanup ownership; returning from Do does not finish the body.
  • Scope retry ownership where operation semantics are known. Application, SDK, mesh, and broker retries must compose to a bounded policy rather than retry independently. Every real attempt passes dependency admission and breaker gates.

The whole policy in one function — the caller classifies the failure, the loop owns the budget, the wait, and cancellation:

// retry makes at most attempts calls, with capped, jittered backoff between them.
// ctx's deadline is the only elapsed-time bound: without one, a server delay of
// any length is slept out, so callers give ctx a deadline. base is at most an
// hour: base<<20 must fit a Duration.
func retry(ctx context.Context, attempts int, base, maxWait time.Duration,
    attempt func(context.Context) (retryAfter time.Duration, retryable bool, err error)) error {
    for i := range attempts {
        retryAfter, retryable, err := attempt(ctx)
        if err == nil || !retryable || i == attempts-1 {
            return err
        }
        wait := min(base<<min(i, 20), maxWait) // an uncapped shift overflows to a negative wait
        wait = wait/2 + rand.N(wait/2+1) //nolint:gosec // G404: retry jitter, not a secret; within [wait/2, wait]
        wait = max(wait, retryAfter)     // Retry-After is a lower bound
        if d, ok := ctx.Deadline(); ok && time.Until(d) <= wait {
            return err // the wait outlasts the budget: stop now, do not sleep it out
        }
        if ctx.Err() != nil { // checked first: with a zero wait the select below picks at random
            return errors.Join(ctx.Err(), err)
        }
        select {
        case <-ctx.Done():
            return errors.Join(ctx.Err(), err)
        case <-time.After(wait):
        }
    }
    return errors.New("retry: attempts < 1, nothing was tried") // not a success
}

attempt closes a discarded response before returning (closing is enough: go-http), and the permit for the dependency is released before the wait, not held through it.

Idempotency and Delivery

A key is meaningful only when the receiver enforces it. Keep one key per logical operation, stable payload, and documented account/tenant scope and retention. A new attempt is not a new operation. For an already-sent request, recover the key actually sent; a newly chosen stable key cannot deduplicate an earlier one. An in-memory map is insufficient for cross-replica or restart-safe deduplication.

Use durable atomic claims and explicit in-progress/completed/unknown states for duplicate delivery. A lease expiring does not prove the previous worker stopped or the remote operation failed. SQL can atomically bind local effects to a receipt; a local transaction/outbox does not atomically commit an unrelated remote effect. If its outcome cannot be reconciled and replay is unsafe, expose that limit instead of promising exactly-once completion.

Deduplication is not ordering: at-least-once delivery also delivers late, and an older event applied second reverts the row. An event that sets state carries a version or source timestamp, and the write keeps the newest: the UPDATE sets the new version and matches only WHERE id = … AND version < the event's version, so an older event changes no row (the statement is in RETRIES-AND-IDEMPOTENCY.md).

Overload and Recovery

Bound admission before spawning unbounded goroutines. Choose reject, wait, or durable backpressure from the caller's contract; do not silently drop work that must be retained. Both admission and waiting need cancellation/lifetime bounds. Isolate expensive tenants or dependencies without unbounded per-key state.

Release a dependency's per-attempt concurrency permit during backoff; retain a separate bound on pending logical work. Rate tokens govern attempts over time, not in-flight calls. Multiple local limiters do not enforce one global quota.

A breaker samples relevant dependency outcomes, limits half-open probes, and ignores stale completions from earlier state generations. Only the dependency's own failures count: a caller's context.Canceled, a local rejection, and a validation error leave the breaker unchanged. A DeadlineExceeded counts when the per-attempt timeout fired and not when the caller's context had already ended (ctx.Err() != nil on the caller's ctx). Coordinate recovery admission so retries cannot bypass an open breaker or flood the recovering dependency. Choose windows/thresholds from traffic and failure behavior, not universal magic numbers; with no traffic data they are constructor parameters, and the report names the chosen values as assumptions.

Fallback is an alternate result with a contract: identity scope, freshness, consistency, and caller-visible degradation. Preserve security decisions and required side effects. Returning an empty list or cached allow on failure is not a neutral substitute. Route authorization rules to go-security.

Verify Failure Behavior

Test the selected policy, not just eventual success. Use controlled outcomes, recorded attempt times/counts, injected randomness/clock or testing/synctest (Go 1.25+) where suitable; real network I/O still needs appropriate test seams.

  • Cancellation before admission, during backoff, and during the active call; exhausted deadlines start no further intended attempts.
  • Nonretryable errors stop; safe transient errors recover within budget; server delay is honored; request bodies replay and response cleanup occurs.
  • Concurrent duplicates, payload mismatch, and crash windows do not produce unsupported guarantees; test durable constraints with the actual database.
  • Saturation bounds in-flight and queued work, preserves required delivery, and contains a noisy tenant. Recovery probes remain bounded under concurrency.
  • Fallback preserves identity/freshness/error semantics; old breaker outcomes cannot change a newer state. Observe logical requests separately from attempts, retries exhausted/suppressed, queue wait/rejection, and degraded responses.

Use go-testing for test mechanics and go-linting for the actual repository gate. Race tests can expose in-process synchronization bugs; they do not establish distributed idempotency. Report executed results, unrun checks, and remaining contract limits honestly; follow go-style-core for report length. A proposed policy is not evidence of outage recovery.

Related Skills

  • go-http: clients, transports, response handling, server shutdown.
  • go-context and go-concurrency: cancellation propagation, goroutine lifetime, synchronization.
  • go-database: transactions, constraints, pool and query mechanics.
  • go-error-handling and go-logging: error matching and structured logs; no high-cardinality or secret metric labels.
  • go-troubleshooting: incident evidence and diagnosis before speculative retry or breaker changes.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.