Unit 6.1 · about 3 h
274 / 324Metrics, logs, traces, correlation and cardinality
The Programming stage already instrumented a pipeline with OpenTelemetry, bounded labels and secret exclusion in its logs-metrics-traces unit, taken here as assumed background. This unit turns that telemetry into evidence: SLOs, saturation readings and incident diagnosis all read the same signals, so the signals must be cheap, bounded and joinable before any decision uses them.
- explain
You can now
- Explain what each of metrics, structured logs, traces and events records, and which questions each answers.
- Connect the signals of one request through a propagated request identifier, and state where correlation stops proving causation.
- Predict the stored series count from a label set, and replace an unbounded label with a bounded design plus sampling.
Covers
- Metrics, structured logs, traces and events
- Request correlation, trace context and causal limits
- Metric cardinality, labels, aggregation and sampling
Study, in this order
7 steps
- Monitoring definitions and black-box versus white-boxGoogle Site Reliability Engineering - selected core chapters
Definitions (white-box versus black-box monitoring); Black-Box Versus White-Box; Symptoms Versus Causes
- The four golden signals and tail instrumentationGoogle Site Reliability Engineering - selected core chapters
The Four Golden Signals (latency, traffic, errors, saturation); Worrying About Your Tail: histogram buckets 0 to 10 ms, 10 to 30 ms, 30 to 100 ms and up by factors near 3; Choosing an Appropriate Resolution for Measurements
Depth · Read the bucket example as the instrumentation contract. Skip the Bigtable and Gmail case studies; they return in the incidents unit.
- Monitoring data sources and purposeful metricsGoogle SRE Workbook - SLO, load, incident and canary exercises
Sources of Monitoring Data (Move information from logs to metrics); Metrics with Purpose (Dependencies; Saturation; Status of Served Traffic; Implementing Purposeful Metrics)
- Labels, vectors and series costGoogle Site Reliability Engineering - selected core chapters
Storage in the Time-Series Arena (about 24 bytes per data point; 1 million series for 12 hours at 1-minute intervals in under 17 GB RAM); Labels and Vectors (labelset, variable expression, vectors)
- Signals: traces, metrics, logs and baggageOpenTelemetry Python - metrics, logs and traces
Signals: Traces (path of a request); Metrics (measurement at runtime); Logs (recording of an event); Baggage (context passed between signals)
- Python manual instrumentation: tracer, spans, attributes and metricsOpenTelemetry Python - metrics, logs and traces
Traces (Acquire Tracer; Creating spans; Add attributes to a span); Metrics (Creating and using synchronous instruments)
- Sampling: head versus tailOpenTelemetry Python - metrics, logs and traces
Terminology (sampled versus not sampled); Head Sampling (decision from trace ID before the outcome); Tail Sampling (error and latency criteria; stateful operating cost)
Focus on
- White-box metrics expose internals and imminent failure; black-box checks prove user-visible symptoms. Page on symptoms, debug with causes.
- One label-value combination is one stored series. At \(3 \text{ services} \times 5 \text{ endpoints} \times 3 \text{ codes} \times 4 \text{ instances}\) the metric holds 180 series; adding a 200,000-value user label takes it to 9,000,000.
- Head sampling decides from the trace ID before the outcome is known, so it cannot keep every error. Tail sampling waits for completed spans and pays stateful cost.
- Keep what the Programming unit enforced: bounded label values, no secrets in attributes, and raw records retained behind every aggregate.
Reproduce closed-book
user_id with 200,000 values: 9,000,000. State which label breaks the budget and give the bounded replacement: drop user_id from the metric and keep it on sampled traces only.Traps
- Treating a shared request ID as proof of causation. Sampling, retries and shared queues break the causal reading; the ID joins evidence, it does not assign blame.
- Putting request IDs, user IDs or tokens in metric labels. Each new value mints new series, and secrets in labels leak into every backend that stores them.
- Averaging per-service p99 values and reading the result as the fleet p99. Merge histogram bucket counts or pool raw samples, then take the percentile.
Practice
The section mini-project covers this: the instrumented request path (request ID, queue wait, handler time, storage time, status, bounded labels) with local export, raw record retention and an enforced cardinality budget.
Self-test
-
When should a page come from black-box rather than white-box monitoring?
When the condition is already user-visible and urgent. White-box predicts or explains; black-box proves the symptom. Every page stays urgent, actionable and needing intelligence. -
A metric has labels with 3, 5, 3 and 4 values. How many series? What happens when a 200,000-value label is added?
180 series; then 9,000,000. Collection, storage and query cost scale with series count, so the unbounded label must move to sampled traces or logs. -
Why can head sampling miss every error in a release?
Its decision uses the trace ID before the outcome exists. Keeping all errors needs tail sampling, which waits for completed spans and pays stateful cost. -
Two services report p99 values. Can you average them into a combined p99?
No. The average of two quantiles is not the quantile of the pooled data. Merge their histograms or retained raw samples, with sample sizes, and compute it again.
Completion criterion
Stop when
You compute both series counts without notes, join one request across a metric, a log line and two spans by ID, and can name the label you would refuse.
Mark passed, then open 6.2 Only mark this when you can do it without notes.Leads to
- 6.2 Latency distributions, throughput and saturation Read tails and saturation from the histograms this unit instruments.
- 6.3 SLIs, SLOs, error budgets and recovery objectives Turn these signals into SLI metrics with windows and budgets.
Comes after
- 7.3 Tenant isolation, input validation and injectionKeep tenant identifiers low-cardinality and secret-free in metrics and traces.
Resources for this unit
- Google Site Reliability Engineering - selected core chapters Read Chapters 3, 4, 6, 10, 12, 17, 19-22: risk, SLOs, monitoring, alerting, troubleshooting, reliability tests, load balancing, overload and cascading failure. Apply every concept to the same bounded RPC and store rather than inventing separate examples.
- Google SRE Workbook - SLO, load, incident and canary exercises Use Chapters 2, 4, 9-11, 12, 14, 16 and 17. Define one SLO, estimate capacity, load through saturation, canary a changed version, roll it back, and write one evidence-based incident timeline.
- OpenTelemetry Python - metrics, logs and traces Instrument request ID, queue wait, handler time, storage time, status and bounded labels. Export locally, retain raw load records and verify that instrumentation does not hide the bottleneck or create unbounded cardinality.
Also in this section
- AlgoMaster System Design - quantitative and failure drills Use Solve Little's Law, Calculate System Availability, Plan Reliability Recovery Targets, Calculate Amdahl's Law Speedup, Analyze Capacity Bottlenecks, Simulate Partition Quorums and Audit Failure Domains only after producing your own estimates. Compare the platform answer with measured service behavior.
- Python Experimental Systems Kit - measurement slice Use its artifact, bounded-pipeline, failure-injection, profiling and reproducibility requirements as the evidence checklist. Reuse the current service instead of building the full project again.
- Gil Tene - How NOT to Measure Latency (talk) Use for coordinated-omission vocabulary only: closed-loop testers that wait for each reply, the missing samples under queueing, and the correction concept of recording intended send time. Do not adopt
wrk,heyorab; write the small Python generator in the saturation unit instead. - M/M/1 queue - model definition and stationary analysis Read Model definition (Poisson arrivals, exponential service, single server, infinite buffer, FIFO) and Stationary analysis (stability \(\lambda < \mu\), \(\rho = \lambda/\mu\), \(L = \rho/(1 - \rho)\), response time \(1/(\mu - \lambda)\) via Little's Law). Use the closed forms as approximations with assumptions stated, checked by your own simulation.