golang-observability
Skill1 starsChanged 5 months agoMITView on GitHub
Golang everyday observability — the always-on signals in production. Covers structured logging with slog, Prometheus metrics, OpenTelemetry distributed tracing, continuous profiling with pprof/Pyroscope, server-side RUM event tracking, alerting, and Grafana dashboards. Apply when instrumenting Go services for production monitoring, setting up metrics or alerting, adding OpenTelemetry tracing, correlating logs with traces, migrating legacy loggers (zap/logrus/zerolog) to slog, adding observability to new features, or implementing GDPR/CCPA-compliant tracking with Customer Data Platforms (CDP). Not for temporary deep-dive performance investigation (→ See `samber/cc-skills-golang@golang-benchmark` and `samber/cc-skills-golang@golang-performance` skills).
- Installs packages
13 other projects use a version of this, 7 of them edited.
Adopted by
Most recently taken up by
- aiproxer/aiproxertheir own version, first seen Nov 8, 2026
- TeddyAlbina/skillsword for word, first seen Nov 1, 2026
- samber/cc-skills-golangedited, first seen Nov 1, 2026
- poizoncoded/skillsedited, first seen Nov 1, 2026
- klawisha012/agentsyncedited, first seen Oct 25, 2026
- jniltinho/llama-modelword for word, first seen Oct 15, 2026
Used with
Projects with it also use golang-gopls, golang-refactoring and 1 more.
Other files people use alongside this one.
Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
jkeddariSkillgolang-gopls
Golang semantic code intelligence with the official gopls language server: definitions, references, symbols, call and implementation hierarchies, diagnostics, formatting, code actions, safe rename, extract, inline, and package API discovery. Use when navigating or refactoring a local Go workspace or when semantic analysis is safer than text search.
Also in 5 projects that use this
jkeddariSkillgolang-refactoring
Golang refactoring workflow for safe structural change: blast-radius discovery, coverage-adaptive safety nets, semantic rename and extract operations, package moves, import-cycle removal, staged commits, and behavior-preserving verification. Use when restructuring existing Go code without intentionally changing its public behavior.
Also in 5 projects that use this
rockcookiesSkillgolang-pkg-go-dev
Golang package and module documentation and exploration via `godig`, a pkg.go.dev API client (CLI + MCP server) — package docs, API references, symbols, code examples, available versions, importers (who imports a package), licenses, and known vulnerabilities. Read-only, no auth. Use for looking up any Go/Golang library's documentation, API signatures, usage examples, which versions exist, whether a dependency has CVEs, or who imports a package — prefer this over Context7 for any Go package or module. Triggers on: how to use a Go library, Go API docs, import usage, code examples, pkg.go.dev. Not for upgrading dependencies (→ See `golang-dependency-management` skill) or choosing a library (→ See `golang-popular-libraries` skill). Not for local symbols, or for navigating an already-used dependency's resolved source, call sites, or generic instantiations — → See `golang-gopls` skill for those.
Also in 4 projects that use this
Other versions13 projects use a version of this.
Word for word · 3
- alexastrum/skl★ 11100% same
- TeddyAlbina/skills★ 2100% same
- jniltinho/llama-model★ 0100% same
Edited · 7
- samber/cc-skills-golang★ 3.9k96% same
- klawisha012/agentsync★ 1098% same
- tstapler/dotfiles★ 897% same
- poizoncoded/skills★ 295% same
- rockcookies/skills★ 195% same
- guynhsichngeodiec/cc-skills-golang★ 095% same
- rzisso/golang-skills★ 096% same
Wrote their own version · 3
- aiproxer/aiproxer★ 24same name
- jkeddari/go-skills★ 1same name
- reagin/agent-skills★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository42 other files it gives its agents.
- golang-benchmarkSkill
- golang-cliSkill
- golang-code-styleSkill
- golang-concurrencySkill
- golang-contextSkill
- golang-continuous-integrationSkill
- golang-databaseSkill
- golang-data-structuresSkill
- golang-dependency-injectionSkill
- golang-dependency-managementSkill
- golang-design-patternsSkill
- golang-documentationSkill
- golang-error-handlingSkill
- golang-google-wireSkill
- golang-graphqlSkill
- golang-grpcSkill
- golang-how-toSkill
- golang-lintSkill
- golang-modernizeSkill
- golang-namingSkill
- golang-performanceSkill
- golang-popular-librariesSkill
- golang-project-layoutSkill
- golang-safetySkill
- golang-samber-doSkill
- golang-samber-hotSkill
- golang-samber-loSkill
- golang-samber-moSkill
- golang-samber-oopsSkill
- golang-samber-roSkill
- golang-samber-slogSkill
- golang-securitySkill
- golang-spf13-cobraSkill
- golang-spf13-viperSkill
- golang-stay-updatedSkill
- golang-stretchr-testifySkill
- golang-structs-interfacesSkill
- golang-swaggerSkill
- golang-testingSkill
- golang-troubleshootingSkill
- golang-uber-digSkill
- golang-uber-fxSkill
The file
We split this skill into the rules it gives an agent
4 rules. 1 is common, 1 is distinctive, and 2 are very specific to this project.
1
Common
Most of the 10 edited versions we read keep these.
1
Distinctive
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Common: line 34
When using observability libraries (Prometheus client, OpenTelemetry SDK, vendor integrations), refer to the library's official …
- Distinctive: line 67
Use third-party slog handler libraries only when the stdlib handler composition is insufficient.
- Very specific: line 130
If the project currently uses zap, logrus, or zerolog, migrate to log/slog.
- Very specific: line 134
Add slog as the new logger with slog.SetDefault()
“Common” is measured: kept by at least 6 in 10 of the 10 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Persona: You are a Go observability engineer. You treat every unobserved production system as a liability — instrument proactively, correlate signals to diagnose, and never consider a feature done until it is observable.
Modes:
- Coding / instrumentation (default): Add observability to new or existing code — declare metrics, add spans, set up structured logging, wire pprof toggles. Follow the sequential instrumentation guide.
- Review mode — reviewing a PR's instrumentation changes. Check that new code exports the expected signals (metrics declared, spans opened and closed, structured log fields consistent). Sequential.
- Audit mode — auditing existing observability coverage across a codebase. Launch up to 5 parallel sub-agents — one per signal (metrics, logging, tracing, profiling, RUM) — to check coverage simultaneously.
Community default. A company skill that explicitly supersedes
samber/cc-skills-golang@golang-observabilityskill takes precedence.
Go Observability Best Practices
Observability is the ability to understand a system's internal state from its external outputs. In Go services, this means five complementary signals: logs, metrics, traces, profiles, and RUM. Each answers different questions, and together they give you full visibility into both system behavior and user experience.
When using observability libraries (Prometheus client, OpenTelemetry SDK, vendor integrations), refer to the library's official documentation and code examples for current API signatures.
Best Practices Summary
- Use structured logging with
log/slog— production services MUST emit structured logs (JSON), not freeform strings - Choose the right log level — Debug for development, Info for normal operations, Warn for degraded states, Error for failures requiring attention
- Log with context — use
slog.InfoContext(ctx, ...)to correlate logs with traces - Prefer Histogram over Summary for latency metrics — Histograms support server-side aggregation and percentile queries. Every HTTP endpoint MUST have latency and error rate metrics.
- Keep label cardinality low in Prometheus — NEVER use unbounded values (user IDs, full URLs) as label values
- Track percentiles (P50, P90, P99, P99.9) using Histograms +
histogram_quantile()in PromQL - Set up OpenTelemetry tracing on new projects — configure the TracerProvider early, then add spans everywhere
- Add spans to every meaningful operation — service methods, DB queries, external API calls, message queue operations
- Propagate context everywhere — context is the vehicle that carries trace_id, span_id, and deadlines across service boundaries
- Enable profiling via environment variables — toggle pprof and continuous profiling on/off without redeploying
- Correlate signals — inject trace_id into logs, use exemplars to link metrics to traces
- A feature is not done until it is observable — declare metrics, add proper logging, create spans
- awesome-prometheus-alerts provides ~500 ready-to-use alerting rules organized by technology for infrastructure and dependency monitoring
Cross-References
See samber/cc-skills-golang@golang-error-handling skill for the single handling rule. See samber/cc-skills-golang@golang-troubleshooting skill for using observability signals to diagnose production issues. See samber/cc-skills-golang@golang-security skill for protecting pprof endpoints and avoiding PII in logs. See samber/cc-skills-golang@golang-context skill for propagating trace context across service boundaries. See samber/cc-skills@promql-cli skill for querying and exploring PromQL expressions against Prometheus from the CLI.
Go 1.26+: slog multi-handler
For simple fan-out to multiple slog handlers, prefer stdlib slog.NewMultiHandler before adding third-party handler-composition dependencies.
logger := slog.New(slog.NewMultiHandler(
slog.NewJSONHandler(os.Stdout, nil),
auditHandler,
))Use third-party slog handler libraries only when the stdlib handler composition is insufficient.
The Five Signals
| Signal | Question it answers | Tool | When to use |
| Logs | What happened? | log/slog | Discrete events, errors, audit trails |
| Metrics | How much / how fast? | Prometheus client | Aggregated measurements, alerting, SLOs |
| Traces | Where did time go? | OpenTelemetry | Request flow across services, latency breakdown |
| Profiles | Why is it slow / using memory? | pprof, Pyroscope | CPU hotspots, memory leaks, lock contention |
| RUM | How do users experience it? | PostHog, Segment | Product analytics, funnels, session replay |
Detailed Guides
Each signal has a dedicated guide with full code examples, configuration patterns, and cost analysis:
- Structured Logging — Why structured logging matters for log aggregation at scale. Covers
log/slogsetup, log levels (Debug/Info/Warn/Error) and when to use each, request correlation with trace IDs, context propagation withslog.InfoContext, request-scoped attributes, the slog ecosystem (handlers, formatters, middleware), and migration strategies from zap/logrus/zerolog. - Metrics Collection — Prometheus client setup and the four metric types (Counter for rate-of-change, Gauge for snapshots, Histogram for latency aggregation). Deep dive: why Histograms beat Summaries (server-side aggregation, supports
histogram_quantilePromQL), naming conventions, the PromQL-as-comments convention (write queries above metric declarations for discoverability), production-grade PromQL examples, multi-window SLO burn rate alerting, and the high-cardinality label problem (why unbounded values like user IDs destroy performance). - Distributed Tracing — When and how to use OpenTelemetry SDK to trace request flows across services. Covers spans (creating, attributes, status recording),
otelhttpmiddleware for HTTP instrumentation, error recording withspan.RecordError(), trace sampling (why you can't collect everything at scale), propagating trace context across service boundaries, and cost optimization. - Profiling — On-demand profiling with pprof (CPU, heap, goroutine, mutex, block profiles) — how to enable it in production, secure it with auth, and toggle via environment variables without redeploying. Continuous profiling with Pyroscope for always-on performance visibility. Cost implications of each profiling type and mitigation strategies.
- Real User Monitoring — Understanding how users actually experience your service. Covers product analytics (event tracking, funnels), Customer Data Platform integration, and critical compliance: GDPR/CCPA consent checks, data subject rights (user deletion endpoints), and privacy checklist for tracking. Server-side event tracking (PostHog, Segment) and identity key best practices.
- Alerting — Proactive problem detection. Covers the four golden signals (latency, traffic, errors, saturation), awesome-prometheus-alerts provides ~500 ready-to-use rules by technology, Go runtime alerts (goroutine leaks, GC pressure, OOM risk), severity levels, and common mistakes that break alerting (using
irateinstead ofrate, missingfor:duration to avoid flapping). - Grafana Dashboards — Prebuilt dashboards for Go runtime monitoring (heap allocation, GC pause frequency, goroutine count, CPU). Explains the standard dashboards to install, how to customize them for your service, and when each dashboard answers a different operational question.
Correlating Signals
Signals are most powerful when connected. A trace_id in your logs lets you jump from a log line to the full request trace. An exemplar on a metric links a latency spike to the exact trace that caused it.
Logs + Traces: otelslog bridge
import "go.opentelemetry.io/contrib/bridges/otelslog"
// Create a logger that automatically injects trace_id and span_id
logger := otelslog.NewHandler("my-service")
slog.SetDefault(slog.New(logger))
// Now every slog call with context includes trace correlation
slog.InfoContext(ctx, "order created", "order_id", orderID)
// Output includes: {"trace_id":"abc123", "span_id":"def456", "msg":"order created", ...}Metrics + Traces: Exemplars
// When recording a histogram observation, attach the trace_id as an exemplar
// so you can jump from a P99 spike directly to the offending trace
obs := histogram.WithLabelValues("POST", "/orders")
if eo, ok := obs.(prometheus.ExemplarObserver); ok {
eo.ObserveWithExemplar(duration, prometheus.Labels{"trace_id": traceID})
} else {
obs.Observe(duration)
}Migrating Legacy Loggers
If the project currently uses zap, logrus, or zerolog, migrate to log/slog. It is the standard library logger since Go 1.21, has a stable API, and the ecosystem has consolidated around it. Continuing with third-party loggers means maintaining an extra dependency for no benefit.
Migration strategy:
- Add
slogas the new logger withslog.SetDefault() - Bridge handlers during migration route slog output through the existing logger: samber/slog-zap, samber/slog-logrus, samber/slog-zerolog
- Gradually replace all
zap.L().Info(...)/logrus.Info(...)/log.Info().Msg(...)calls withslog.Info(...) - Once fully migrated, remove the bridge handler and the old logger dependency
Definition of Done for Observability
A feature is not production-ready until it is observable. Before marking a feature as done, verify:
- Metrics declared — counters for operations/errors, histograms for latencies, gauges for saturation. Each metric var has PromQL queries and alert rules as comments above its declaration.
- Logging is proper — structured key-value pairs with
slog, context variants used (slog.InfoContext), no PII in logs, errors MUST be either logged OR returned (NEVER both). - Spans created — every service method, DB query, and external API call has a span with relevant attributes, errors recorded with
span.RecordError(). - Dashboards and alerts exist — the PromQL from your metric comments is wired into Grafana dashboards and Prometheus alerting rules. Ready-to-use alert rules for common infrastructure dependencies are available at awesome-prometheus-alerts.
- RUM events tracked — key business events tracked server-side (PostHog/Segment), identity key is
user_id(not email), consent checked before tracking.
Common Mistakes
// ✗ Bad — log AND return (error gets logged multiple times up the chain)
if err != nil {
slog.Error("query failed", "error", err)
return fmt.Errorf("query: %w", err)
}
// ✓ Good — return with context, log once at the top level
if err != nil {
return fmt.Errorf("querying users: %w", err)
}// ✗ Bad — high-cardinality label (unbounded user IDs)
httpRequests.WithLabelValues(r.Method, r.URL.Path, userID).Inc()
// ✓ Good — bounded label values only
httpRequests.WithLabelValues(r.Method, routePattern).Inc()// ✗ Bad — not passing context (breaks trace propagation)
result, err := db.Query("SELECT ...")
// ✓ Good — context flows through, trace continues
result, err := db.QueryContext(ctx, "SELECT ...")// ✗ Bad — using Summary for latency (can't aggregate across instances)
prometheus.NewSummary(prometheus.SummaryOpts{
Name: "http_request_duration_seconds",
Objectives: map[float64]float64{0.99: 0.001},
})
// ✓ Good — use Histogram (aggregatable, supports histogram_quantile)
prometheus.NewHistogram(prometheus.HistogramOpts{
Name: "http_request_duration_seconds",
Buckets: prometheus.DefBuckets,
})Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

