Observability
Study Guide

Metrics: numeric time series, cheap aggregation, good for dashboards/alerts. Logs: discrete events, high cardinality text, great for debugging detail. Traces: request path across services, latency breakdown. Use all three (three pillars) - correlate with exemplars and trace IDs in logs.

A time series is a metric name + sorted label set to timestamped samples. Labels identify dimensions (instance, job, path). High-cardinality labels (user IDs) explode storage - design cardinality carefully.

Kubernetes SD discovers pods/services; relabel configs map K8s metadata to target labels. Sidecar vs centralized scraping patterns exist; kube-prometheus-stack is common. RBAC needed for API access.

Counter only increases (reset on process restart) - use rate(). Gauge goes up/down - memory, queue depth. Histogram buckets latency/size - compute quantiles with recording rules or Grafana; beware histogram bucket cardinality.

rate() needs range ≥2 scrape intervals; too-short windows are noisy. Counter resets (pod restart) handled by Prometheus. irate is more volatile for dashboards; prefer rate for alerting stability. Alignment of scrape and eval windows matters.

On-call: RED/USE golden signals, clear SLO burn, links to runbooks, minimal clutter. Wallboards can be high-level KPIs. Tag dashboards by service and env; avoid duplicating the same chart 20 ways.

Rate (requests/sec), Errors (failed requests), Duration (latency distribution). Service-oriented complement to USE (utilization, saturation, errors) for resources.

For every resource: Utilization (% time busy), Saturation (queued work), Errors (count). Great for nodes, disks, NICs - complements RED for services.

Precompute expensive PromQL (aggregates, histogram quantiles) on scrape path. Faster dashboards, consistent alert queries, lower query load. Trade-off: more metrics to store and manage.

Alert on symptoms (SLO burn, user-visible errors) not every cause. Require sustained breaches, dedupe/aggregate by service, on-call rotations, severity levels, regular alert audits, auto-resolve noise, runbooks mandatory.

Key-value fields (level, trace_id, user) instead of free text. JSON (or logfmt) enables query filters in Loki/ELK without fragile regex. Consistent schema beats clever prose.

ELK/Opensearch: index-heavy, powerful search, higher cost. Loki: index labels only, cheap at scale, pairs with Grafana. Commercial SIEM adds security analytics. Choose by query patterns, retention, and budget.

Volume: bytes/sec ingested. Cardinality: unique label combinations - high cardinality (per-request IDs as labels) breaks Loki-like systems and expensive in Elasticsearch. Sample debug logs or use traces instead.

Sidecar (Fluent Bit per pod): isolation, per-tenant config, more resources. Daemonset: one agent per node reads container logs - simpler, shared resources. Both ship to central sink.

Redact at source, tokenize fields, restrict access with RBAC, short retention for raw logs, separate security logging to SIEM, encryption at rest/transit, data classification policies, avoid logging bodies of requests by default.

Trace: end-to-end request DAG. Span: one timed operation (HTTP call, DB query). Parent/child links show call tree. Context propagation (trace_id, span_id) passes across services (W3C Trace Context headers).

Head-based: decide sample at trace start - simple, may drop interesting rare errors. Tail-based: buffer spans, decide after seeing full trace (e.g. errors/slow) - better signal, higher memory cost (Tempo/Grafana Agent patterns).

OTel SDK instruments apps; Collector receives, processes, exports to backends (Jaeger, Tempo, X-Ray); semantic conventions standardize attribute names. Vendor-neutral instrumentation reduces lock-in.

Exemplars link histogram buckets to trace IDs. Logs include trace_id for drill-down. Unified query UI (Grafana) or jump links from alert to trace. Reduces mean time to resolution.

CPU/memory for span creation, network export, and tail sampling buffers. Mitigate: aggressive sampling for health checks, async export, batching, eBPF-based auto-instrumentation where appropriate, avoid huge span attributes.

SLI: measured metric (availability, latency). SLO: target over window (99.9% success monthly). SLA: contractual consequence with customers - usually stricter external promise than internal SLO.

Burn compares consumed errors to allowed budget. Multi-window (e.g. 1h + 6h) reduces flapping: short window catches sudden incidents, long window catches slow leaks. Google SRE workbook pattern.

Black-box: synthetic checks from outside (Pingdom, canaries) - user-visible. White-box: internal metrics (queue depth, GC pauses). Need both: black-box catches DNS/LB issues white-box misses.

Impact, first steps, dashboards, logs queries, recent deploys, escalation path, rollback/feature flag toggles, known false positives. Goal: any on-call engineer can start mitigation without tribal knowledge.

Measure client-side outcomes (timeouts, error codes), cache/fallback strategies, track vendor status pages, define internal SLO on your handling of dependency failure, contractual SLAs where possible.

Kernel-level programs for low-overhead metrics, security auditing, network tracing without full packet capture to userspace. Tools like Pixie, Cilium Hubble, bpftrace. Requires recent kernels and careful safety review.

RUM: actual browser/mobile sessions - Core Web Vitals, geo/device variance. Synthetic: scripted probes from fixed locations - steady baseline. Combine: synthetic for uptime, RUM for real UX.

Too many unique label combinations (per-user, per-IP) makes TSDB storage and query cost explode. Fix: aggregate at ingest, drop labels, use logs/traces for high-cardinality debugging, recording rules to roll up.

Tier retention (hot/warm/cold), sampling, drop debug logs in prod, index only useful fields, use object storage for archives, chargeback by team, review top cardinality metrics monthly, negotiate commercial tool ingest.

Samples CPU/heap periodically (Parca, Pyroscope, Datadog profiler) to find hot code paths. Worth it for latency/cost optimization of steady-state services; overhead low with sampling; less critical for tiny services.