Unit 6.6 · about 5 h
40 / 324Structured logs, metrics and traces
A profile explains where code spends resources; telemetry explains which requests and failures are affected. Correlation should make a fault diagnosable without turning every request ID into a metric series.
- implement
You can now
- Add structured event logs, low-cardinality counters and latency histograms to a bounded service.
- Preserve trace context through async workers and link request logs to the relevant span.
- Locate a seeded dependency failure using telemetry while excluding credentials and request payloads.
Covers
- Structured logs, correlation IDs and secret redaction
- Low-cardinality metrics, latency distributions and queue depth
- Trace context, spans and cross-boundary latency
Study, in this order
3 steps
- Instrument one request, not a whole platformOpenTelemetry Python instrumentation
Setup; Traces: Creating spans, Creating nested spans, Get the current span, Set span status, Record exceptions in spans, Change the default propagation format; Metrics: Creating and using synchronous instruments; Logs; Semantic convention stability
Depth · Use local or in-memory exporters and pin the SDK version. The Logs API/SDK is under development; structured Python logging plus explicit trace/span fields is sufficient. Do not build a Collector deployment.
- Metric attribute sets have a budgetOpenTelemetry Python instrumentation
Cardinality limits: Configuration, Overflow attribute, Synchronous instrument cardinality limits and Asynchronous instrument cardinality limits
Depth · Understand limits and overflow, not SDK internals. Keep the application's label space bounded even when an SDK limit exists.
- Minimize sensitive telemetry at the sourceOpenTelemetry Python instrumentation
Your responsibility; Data minimization; Protecting sensitive data; Risk and limitations of hashing for anonymization
Focus on
- Log event name, request ID, trace ID, span ID, outcome class and elapsed time as fields. Allowlist fields before serialization; exception messages and URLs can contain secrets too.
- Use counters for completed, failed, rejected and retried work; a histogram for latency; and a gauge for current queue depth. Route templates and bounded error classes are labels; raw paths, user IDs and request IDs are not.
- Create child spans while the intended parent context is active. New asyncio tasks copy current context at creation; a long-lived queue worker needs the producer's context carried with each item rather than assumed ambient context.
- Trace IDs join per-request evidence but do not describe aggregate health. Sampling may omit a request's trace; keep useful error logs and metrics even when no exported trace is available.
Reproduce closed-book
ok and error. Pass a dummy authorization secret and assert it appears nowhere in exported logs, spans or metrics. Mark the dependency span ERROR and locate that request from the signals, without new print statements.Traps
- Putting
trace_idorrequest_idon every metric. Correlation then creates unbounded attribute sets instead of a useful aggregate. - Assuming the context of a task created at startup follows every item it later dequeues. Propagate the item's context explicitly.
- Treating hashed user IDs as safe anonymization or recording full exception payloads before redaction. Predictable hashes can be reversed by enumeration.
Practice
The section mini-project covers this: structured request logs, bounded-label metrics, per-item trace propagation, secret exclusion and diagnosis of a seeded pipeline fault.
Self-test
-
Why can two bounded labels still create many metric series?
Distinct combinations multiply. Budget the full attribute-set space, not each label in isolation. -
Does
record_exceptionautomatically express your handled failure policy?No. It records an exception event. Set span status deliberately when a handled failure makes the operation unsuccessful. -
Which signal reveals queue pressure without looking up a particular request?
A queue-depth gauge together with admission rejection, retry and latency metrics. Logs and traces explain individual cases. -
How should a queue worker recover a producer's trace context?
Carry the context with the item and attach it while processing, detaching infinally. Task-creation context alone is not per-item propagation.
Completion criterion
Stop when
You can find the seeded failure by request and trace identity, verify parentage and the bounded metric attribute sets, and prove the dummy secret is absent from every exported signal.
Mark passed, then open 6.7 Only mark this when you can do it without notes.Leads to
- 6.7 Health checks, containers, supervision and runbooks Turn the signals into health decisions and recovery instructions.
Comes after
- 5.8 Dependency locks, run manifests and quality gatesEmit the logs, metrics and traces the manifest points at.
- 6.4 Backpressure, load shedding and graceful shutdownExpose queue growth, rejection and recovery instead of hiding overload.
- 6.5 Profiling CPU and memory, and benchmarking honestlyCombine profiles with request-level evidence to separate CPU work from dependency waits.
Resources for this unit
- OpenTelemetry Python instrumentation Manually instrument one request so its structured logs, one low-cardinality metric and one trace share a correlation identity. Use the telemetry to locate one seeded fault.
Also in this section
- AlgoMaster Concurrency Study Concurrency 101; the Python units on threads, the GIL, multiprocessing, asyncio and concurrent.futures; synchronization primitives; concurrency challenges; locking strategies; and concurrency patterns. Then solve Bounded Buffer, Thread-Safe Blocking Queue, Thread-Safe Rate Limiter and Task Scheduler with Dependencies.
- Python concurrency documentation Use asyncio task groups, cancellation, queues and timeouts; concurrent.futures; multiprocessing; threading; and signal documentation to verify lifecycle and failure behavior.
- High Performance Python, 3rd edition Read Chapter 2 Profiling to Find Bottlenecks, Chapter 9 Asynchronous I/O, Chapter 10 The multiprocessing Module and Chapter 12 Using Less RAM, and use the companion code (https://github.com/mynameisfiber/high_performance_python_3e) as experiment starting points. Change the workloads and measure the regimes where concurrency helps, loses or only moves the bottleneck.
- Operating Systems: Three Easy Pieces - concurrency Read Chapters 26-33, from threads through event-based concurrency, for locks, condition variables, semaphores, concurrency bugs and event loops. Do selected homework only for a weak mechanism.
- Docker guides Use the Python container guide and build best practices to run as non-root, pin dependencies, handle signals and test CPU and memory limits.
- Algorithmica - Getting Accurate Results Use Measuring the Right Thing and Reducing Noise for benchmark design: input distribution, latency versus throughput, warm-up, frequency scaling and pinning. The examples are C++; the checks transfer to Python.
- Timeouts, retries, and backoff with jitter Read Timeouts, Retries and backoff, and Jitter for deadline scope, idempotency, one retry owner and bounded retry load. Interpret the five-layer example as three total attempts per layer, not three retries plus the initial attempt.
- Site Reliability Engineering - overload and troubleshooting Read Chapter 21, The Pitfalls of Queries per Second, Client-Side Throttling, Utilization Signals, Handling Overload Errors and Conclusions. Read Chapter 12, Triage, Examine, Diagnose, Test and Treat, and Making Troubleshooting Easier for a service runbook. Skip datacenter load-balancing implementation details.
- Liveness, Readiness, and Startup Probes Read Liveness probe, Readiness probe and Startup probe for the different health decisions and failure actions. Apply the contracts to a local service; do not install Kubernetes or configure a cluster.