Performance Profiling

Never optimize what you have not measured. Profile, fix the largest bottleneck, measure again under the same conditions. A change without a before/after number is a guess.

When to use

• A request, job, function or command is slower than it should be.
• Tail latency (p95/p99) is high or spiky, even if the average looks fine.
• CPU, memory, GC time, or cloud cost is too high.
• Throughput tops out before hardware does.
• The user asks to "make it faster", profile, or benchmark something.

Not for: browser page-load and rendering (web-performance), a single slow SQL query (sql-optimization), or capacity planning without a concrete bottleneck.

Workflow

1. State the problem as a number. Metric, current value, target, conditions. Example: "p99 of POST /orders is 1.8 s at 300 req/s on staging; target under 400 ms." If the user gives no target, propose one and confirm it.
2. Pick the right metric (table below). Use percentiles, never the mean, for latency.
3. Reproduce in a production-like setup. Release/optimized build, realistic data volume and shape, same config and dependency versions, no debugger attached, JIT warmed up. Record a baseline over several runs and note the spread.
4. Find what bounds you before opening a profiler: CPU, memory/GC, I/O, locks, or a downstream dependency. Use traces or APM first if they exist: is the time even spent in this process?
5. Profile with the right tool for the bound and the stack. See references/profilers.md. Prefer low-overhead sampling profilers.
6. Form one hypothesis and make one change. Fix the widest bar in the profile first. Leave code that takes 1% of time alone.
7. Re-measure under identical conditions. Compare distributions (several runs), not single numbers. Confirm the profile changed the way you expected.
8. Check correctness. Tests pass; output is identical; no new race conditions or stale-cache bugs.
9. Add a regression guard (benchmark in CI, load test, or alert) so the win stays.
10. Report a before/after table, what changed, why it helped, and any trade-offs (memory for speed, staleness for speed, complexity).

Choose the metric

Find the bound

Reading a flame graph

• Width is the share of samples (time). The x-axis is not time order.
• The y-axis is stack depth; the top edge is where the CPU actually was.
• Look for wide plateaus near the top: that is self time worth optimizing.
• A CPU profile only shows on-CPU time. If the process waits (I/O, locks), use a wall-clock or off-CPU mode, or traces.
• Compare a before and after flame graph of the same workload to confirm the fix.

Common wins (check in this order)

Benchmarking properly

• Use a real harness, not a stopwatch loop:

  Stack	Harness
  Java/Kotlin	JMH
  C#/.NET	BenchmarkDotNet (add [MemoryDiagnoser] for allocations)
  Python	pyperf, pytest-benchmark, or python -m timeit for one-liners
  Go	go test -bench=. -benchmem -count=10, compare runs with benchstat
  Rust	criterion
  JS/TS	tinybench or mitata
  Any CLI	hyperfine --warmup 3 'old-cmd' 'new-cmd'

• Warm up before measuring (JIT, caches, connection pools).
• Run many iterations and several processes; report median and spread. Treat differences inside the noise as no difference.
• Benchmark optimized builds only (--release, -c Release, production NODE_ENV).
• Guard against dead-code elimination and constant folding: consume results (JMH Blackhole, return values).
• Use realistic inputs. Sorted, tiny, or always-cached data gives misleading results.
• Keep the machine quiet: no other heavy processes, power plugged in, same hardware for before and after.
• A microbenchmark proves a local speedup; only an end-to-end measurement proves user impact.

Load testing

• Use a load generator such as k6, wrk2, vegeta, hey or Locust against a staging environment.
• Prefer open-model (fixed arrival rate) load, e.g. k6's constant-arrival-rate executor or wrk2's -R. Closed-loop tools that wait for each response under-report latency when the system slows down (coordinated omission).
• Ramp up gradually and record where latency or errors break the SLO; that is the real capacity.

Safety

• Do not run load tests against production or shared environments without explicit confirmation from the user.
• Ask before attaching a profiler to a production process. Prefer low-overhead samplers. Heap dumps can pause the process and contain secrets and personal data: treat dump files as sensitive and never commit or share them.
• Never trade correctness for speed silently (skipping validation, weakening consistency, unbounded caches). Call out every such trade-off.

Anti-patterns

• Optimizing by intuition — the real bottleneck is rarely where you guess. Profile first.
• Reporting the mean — averages hide the tail users feel.
• Benchmarking debug builds or cold JITs — numbers will not transfer.
• Changing several things at once — you cannot tell which change helped or hurt.
• Micro-optimizing a function that is 1% of the profile — no visible effect, more complexity.
• Caching without bounds or invalidation — trades a latency problem for a memory or correctness problem.
• Tiny test data — hides N+1s, missing indexes and O(n²) code.
• Declaring victory on one run — noise can exceed the improvement.

Done when

• Before and after numbers exist for the target metric under the same conditions, over several runs.
• The target is met, or the report explains what limits further gains.
• A new profile shows the bottleneck moved or shrank.
• Tests pass and behaviour is unchanged.
• A regression guard (benchmark, load test, or alert) is in place.
• Trade-offs are written down in the PR or a code comment.

References

• references/profilers.md — read when choosing or running a profiler, heap analyzer, tracer or load generator for a specific language or OS.