Observability and Instrumentation

Overview

You cannot fix what you cannot see. This skill makes production behavior observable: crashes and ANRs reported with enough context to fix, startup and jank measured against Play Vitals thresholds, and logs structured so they help without leaking user data. Instrumentation is part of the feature, not an afterthought bolted on when the first bad review arrives.

When to Use

• Adding a feature whose failures would be invisible without instrumentation (payments, sync, background work)
• Setting up or auditing crash reporting and performance monitoring
• Before any release that will be watched during staged rollout (see shipping-and-launch)
• Investigating field-only issues where local reproduction failed (see debugging-and-error-recovery)

Skip when: Prototypes or internal builds that will never reach users — but wire observability in before the first external release, not after.

Core Process

Step 1: Crash and ANR Reporting

1. Crashlytics (or Sentry) with context, not just stack traces:

// Attach the state that turns a stack trace into a diagnosis
FirebaseCrashlytics.getInstance().apply {
    setCustomKey("screen", "task_detail")
    setCustomKey("sync_state", syncState.name)
    setUserId(pseudonymousId)          // NEVER an email or real identifier
    recordException(NonFatalSyncError(cause))  // non-fatals for handled-but-wrong paths
}

2. Reporting rules:
   • Record non-fatal exceptions for caught-but-abnormal paths — a swallowed exception is an invisible bug
   • Custom keys over log spam: state at crash time beats a breadcrumb trail
   • Upload the R8 mapping file automatically in the release pipeline (Crashlytics Gradle plugin does this) — an obfuscated stack trace is noise
   • ANRs are surfaced by Play Vitals, not your crash SDK — watch both

Step 2: Play Vitals Thresholds

3. Know the numbers Google judges you by (Play Console → Android Vitals):

Exceeding a threshold suppresses your Play Store visibility. Vitals is the scoreboard; your in-app instrumentation exists to explain why a number moved.

Step 3: Structured Logging

4. Timber with a release tree — logs are for debug builds, telemetry is for release:

class App : Application() {
    override fun onCreate() {
        super.onCreate()
        Timber.plant(
            if (BuildConfig.DEBUG) Timber.DebugTree()
            else CrashReportingTree()   // routes WARN/ERROR to Crashlytics, drops the rest
        )
    }
}

// GOOD: structured, no PII
Timber.w("sync_failed attempt=%d reason=%s", attempt, reason.name)

// BAD: PII in a log line — logcat is world-readable on rooted devices
Timber.d("sync failed for user %s token %s", email, token)

5. Logging rules:
   • No PII, tokens, or request bodies at any level
   • Log.d/Log.v stripped in release via R8 (-assumenosideeffects, see ../../references/security-checklist.md)
   • One event, one line, stable key=value shape — greppable beats prose

When several entry points write to one log, name the entry point. The same SyncWorker enqueued periodically, by an FCM push, and by pull-to-refresh — plus a foreground service and the UI reporting sync errors to the same tag — produces interchangeable sync_failed lines and non-fatals. Attributing one then falls back to elimination (WorkManager history, device state, release timing), which holds only while those records still exist. Stamp the entry point where the run starts and carry it across the boundary, never re-derive it downstream:

// Enqueue site names the path; the worker reads it, it doesn't guess
fun enqueueSync(entry: String) = WorkManager.getInstance(context).enqueue(
    OneTimeWorkRequestBuilder<SyncWorker>()
        .setInputData(workDataOf("entry" to entry))   // "fcm_push", "pull_to_refresh"; the periodic request sets "periodic"
        .build()
)

// In SyncWorker.doWork()
val entry = inputData.getString("entry") ?: "unknown"
Timber.w("sync_failed entry=%s attempt=%d reason=%s", entry, runAttemptCount, reason.name)
FirebaseCrashlytics.getInstance().recordException(
    NonFatalSyncError(cause),
    CustomKeysAndValues.Builder().putString("entry", entry).build()  // per-event, not global
)

Prefer per-event keys over setCustomKey("entry", ...): global custom keys are last-write-wins across the process, so concurrent workers overwrite each other's entry point. A field that merely correlates with an entry point (thread name, screen) is a hint, not an attribution.

Step 4: Performance Instrumentation

6. Measure startup honestly with reportFullyDrawn:

// The system's TTID stops at first frame; report when content is actually usable
class TaskListActivity : ComponentActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContent {
            val uiState by viewModel.uiState.collectAsStateWithLifecycle()
            if (uiState is TaskListUiState.Success) {
                LaunchedEffect(Unit) { reportFullyDrawn() }
            }
            TaskListContent(uiState)
        }
    }
}

7. Custom traces for the flows that matter (Firebase Performance or androidx.tracing):

val trace = Firebase.performance.newTrace("checkout_flow")
trace.start()
// ...
trace.putMetric("items", cart.size.toLong())
trace.stop()

8. Gate regressions in CI: run Macrobenchmark on the release candidate and fail on startup/jank regressions against the previous baseline (see performance-optimization and ci-cd-and-automation). A regression caught in CI costs a re-run; caught in Vitals it costs users.

Step 5: Release Health During Rollout

9. Staged rollout is only as good as what you watch (see shipping-and-launch):
   • Define the abort criteria before rolling: e.g. "halt at crash rate > 0.5% or ANR > 0.3% on the new version"
   • Compare version-over-version, not absolute: a new crash cluster at 5% rollout predicts the 100% disaster
   • Watch: Crashlytics velocity alerts, Vitals per-version, key business events (did sign-ins drop?)

Step 6: Runbooks for Every Alert

10. Every alert links to a runbook — Crashlytics velocity alerts, ANR spikes, backend error rate on mobile endpoints. A runbook answers three questions without making the on-call reader think: what it means, what to check first, who to escalate to. Store them in docs/runbooks/, named after the alert, and put the link in the alert itself.

# Runbook: Crashlytics velocity alert on new version
**Means:** A crash cluster is growing fast — likely a regression in the version under staged rollout.
**First check:** Is the issue in the newest versionCode only? If yes, halt the staged rollout
  (Play Console → Release → Production → Halt rollout), then check Remote Config flags changed
  in the last 24h — a flag flip can crash old versions too.
**Escalate to:** Release owner for this version; #android-oncall if a kill switch is needed.

11. Expand beyond three lines only when the first check can't decide. Five steps covering the three most common causes beat twenty steps covering every edge case — the long one gets skimmed at 3 a.m.
12. Update the runbook when closing every incident it was used in. A wrong or missing step is fixed before the incident is marked resolved; a stale runbook builds false confidence.

Common Rationalizations

Red Flags

• catch (e: Exception) { } with no recordException — swallowed failures are invisible
• Log lines containing emails, tokens, or request bodies
• Release builds still planting Timber.DebugTree()
• No mapping file upload in the release pipeline
• Staged rollout with no written abort criteria
• One log tag or non-fatal type fed by several Workers, a service, and the UI, with no field naming which one produced it
• Entry point set via global setCustomKey from concurrent paths, or re-derived inside the Worker instead of passed in
• Alerts with no runbook link, or runbooks untouched after the incidents they were used in
• Startup "measured" only by TTID with no reportFullyDrawn
• Performance claims in PRs with no Macrobenchmark or trace evidence

Verification

• Crash reporting captures custom keys and non-fatals for the changed flows
• R8 mapping file uploaded automatically on release builds
• No PII/tokens in any log statement (grep the diff for log calls)
• Release log tree drops DEBUG/VERBOSE; R8 strips Log.d/Log.v
• reportFullyDrawn called when primary content is usable
• Macrobenchmark (or trace) evidence attached for performance-sensitive changes
• Rollout abort criteria written down with owner and thresholds
• Every log stream or non-fatal reached from more than one entry point carries an entry field, set where the run starts and passed across Worker/service boundaries
• Every alert links to a runbook in docs/runbooks/ stating what it means, the first check, and who to escalate to