Unit 8.2 · about 4 h
222 / 324Streams, asynchronous copies and honest timing
Every speedup claim in this section stands or falls on timing discipline. Async execution makes naive timers lie, and a hidden sync can pose as a slow kernel.
- explain
- measure
You can now
- Explain why GPU launches return before the work finishes and where PyTorch syncs for you.
- Order
wait_streamandrecord_streamcorrectly when a side stream consumes a default-stream tensor. - Time a GPU operation with events and show why a bare wall clock understates it.
Covers
- Device streams, asynchronous copies and synchronization
- Accidental synchronization and measurement error
Study, in this order
3 steps
- Asynchronous execution and CUDA streamsPyTorch - checkpointing, numerical accuracy and CUDA semantics
Asynchronous execution: enqueue semantics,
CUDA_LAUNCH_BLOCKING, and theEventtiming example; CUDA streams: default-stream auto-sync, thewait_streamplusrecord_streamfix, andStreamsemantics of backward passesDepth · Skip the TF32, GEMM precision, allocator-tuning and CUDA-graph sections here. They belong to later units.
- Pinned memory and asynchronous copiesPyTorch - checkpointing, numerical accuracy and CUDA semantics
Best practices, Use pinned memory buffers:
pin_memory,non_blockingcopies, and overlapping transfer with compute - Operations that force synchronizationPyTorch - reliability and performance recipes
GPU specific optimizations, Avoid unnecessary CPU-GPU synchronization: the
print,item,copy,nonzeroand data-dependent-control-flow list
Focus on
- Launches enqueue; they do not execute. On-device order is preserved per stream, and PyTorch syncs around CPU-GPU copies on the default stream only.
- Side streams get no automatic safety. The consumer stream waits on the producer with
wait_stream, and the tensor records its new stream withrecord_streamso its memory is not reused early. non_blocking=Trueis a request, not a command. From pageable host memory the copy still syncs; pin the source first.- Backward follows forward stream placement. Grads produced under a side-stream backward must be consumed in the same stream context or after an explicit wait.
Reproduce closed-book
fp32 matmul both ways: CUDA events around the op with a synchronize before reading, versus a bare perf_counter around the launch. The matmul is \(2 \times 2048^3\), about 17.2 GFLOP, so the event time is a few milliseconds at most (4.1 ms on an RTX 3050 6 GB), while the bare clock reads microseconds (0.007 ms there) because it measures enqueue only. Exact numbers depend on hardware; the size of the gap is the assertion.Traps
- Timing with
time.perf_counterand no sync, then reporting the result as kernel time. You timed the launch, not the work. - Calling
.item(), printing a CUDA tensor, or branching on one inside the timed region. Each forces a sync and the stall gets billed to the wrong line. - Assuming default-stream safety extends to side streams. In the docs example,
sumon a side stream may start beforenormal_finishes on the default stream. The side stream mustwait_streamon the default stream, andrecord_streamprotects the tensor if it is freed while the side stream still uses it.
Practice
The section mini-project covers this: synchronized phase timing with explicit stream boundaries.
Self-test
-
An
Event-timed matmul reports 4.1 ms while a bare wall clock reports 0.007 ms. Which is the kernel time and why?TheEventtime. Launches are asynchronous, so the bare clock measures enqueue only. Events plus asynchronizemeasure the queued work. -
Data comes from pageable host memory with
.to(device, non_blocking=True). Is the copy asynchronous?Effectively no. Asynchronous host-to-device copies need pinned source memory, so setpin_memory=Truein the loader first. -
loss.backward()runs undertorch.cuda.stream(s). Where may the.gradtensors be consumed?Inside the same stream context, or after the consuming stream waits ons. Using them on the default stream without a wait is unsafe in current PyTorch. -
Which of
print(t),t.item()andt.to('cpu')force a host sync for a CUDA tensort?All three. Each needs the values or the transfer finished on the host before Python continues.
Completion criterion
Stop when
Your two-way timing shows the enqueue gap on your GPU, and you can point at the sync in any snippet that prints, copies, or branches on a CUDA tensor.
Mark passed, then open 8.3 Only mark this when you can do it without notes.Leads to
- 8.9 Profiling training phases and verifying optimizations Apply event discipline inside annotated profiled phases.
Resources for this unit
- PyTorch - reliability and performance recipes Use the Profiler, Automatic Mixed Precision, model loading and Benchmark recipes, then the Performance Tuning Guide. Apply each to one compact model from an earlier section and keep a CPU fallback when CUDA is unavailable.
- PyTorch - checkpointing, numerical accuracy and CUDA semantics Use the official user-guide entries for torch.utils.checkpoint, numerical accuracy, CUDA semantics, reproducibility and serialization. Predict memory categories before measuring and define resume equivalence before running.
Also in this section
- PyTorch - compiler troubleshooting Read graph breaks, guards, recompilations and logging. Compile one stable fragment, intentionally trigger a graph break or recompilation, inspect it, and compare eager and compiled outputs and gradients.
- Deep Learning Training Debugger Use the failure-injection, measurement and evidence rules as the deeper standard. The full Tier 2 project is a later route and is not added to this stage milestone.
- TensorTonic - mixed-precision memory and transformer FLOPs Solve only Mixed-Precision Training Memory Accountant and Transformer Training FLOP Estimator. Reconcile both with the compact model's profiler and document every included and excluded term.
- Learn PyTorch - profiler walkthrough Use notebook 10 only if the official profiler route remains unclear. Transfer the relevant annotations into the compact model instead of keeping the tutorial code.
- TrenTorch - precision, checkpointing and resume Solve Loss Scaling, FP16/BF16 Representable Range, Gradient Checkpointing and Resume-from-checkpoint in NumPy, then reproduce each behavior with the PyTorch API in the compact model.
- Python os - `os.replace` Read
os.replacefor the atomic-rename step of checkpoint writes: save to a temp file in the same directory,flushandfsync, then replace the final name. - Hands-On Machine Learning with Scikit-Learn and PyTorch Read Appendix B, Reduced Precision Models and Mixed-Precision Training, for why
float16training underflows and whatfp32primary weights and loss scaling fix.