Tensor layout: shape, stride, views and copies

A tensor is storage plus an interpretation. Backward passes that assume contiguous input misread transposed data, and view-versus-copy mistakes turn a reshape into silent corruption.

explain
implement

You can now

Compute an element address from shape, stride and index, and predict contiguity after transpose or slicing.
Classify each reshape, transpose, slice and index operation as view or copy, and state the reshape and indexing rules you may not rely on.
Handle noncontiguous inputs in your own engine: read them correctly through strides or copy to contiguous first, and justify the choice.

Covers

Shape, axes, dtype, layout, stride and storage
View, copy, reshape, transpose and noncontiguous inputs

Study, in this order

3 steps

Tensor creation, shape and memory reuse
D2L - tensors, autograd, MLP and backpropagation

Getting Started: arange, zeros, ones, shape, size and reshape with -1 inference; Indexing and Slicing; Saving Memory: Y = Y + X reallocates (id changes) while Z[:] = X + Y and X += Y reuse storage

Depth · The page never defines stride, view or storage: those come from the two PyTorch notes below. Use this page for construction, shape inference and the realloc-versus-reuse demos.

Done
Which ops share storage
PyTorch Tensor Views

The view/storage sharing example with data_ptr and mutation through the view; the transpose example turning a contiguous tensor non-contiguous with no data movement; the full view-ops list; reshape may-copy and contiguous conditional-copy notes; basic indexing returns views while advanced indexing copies

Done
Strides as element steps
`torch.Tensor.stride` - stride and storage offset

Stride definition: per-dimension element steps that, with storage_offset, locate every element; combine with the Tensor Views transpose and expand layouts

Done

Focus on

A tensor is storage plus shape, stride, offset and dtype. Element [2, 1] of a (3, 4) C-contiguous float32 tensor sits at offset \(2 \times 4 + 1 \times 1 = 9\), so the strides are (4, 1) in elements. NumPy reports the same layout in bytes: (16, 4) for float32.
Transpose swaps shape and strides without moving data: (3, 4) with strides (4, 1) becomes (4, 3) with strides (1, 4), non-contiguous but still a view, so [1, 2] of the transpose aliases offset 9 of the base.
Reshape, reshape_as and flatten may return a view or a copy: never rely on which. view() demands a compatible layout instead. Basic indexing gives views; advanced indexing (integer or boolean arrays) copies.
Expand is a stride-0 view: no storage advances along the stretched axis, which is why perturbing one expanded element moves all its aliases and breaks point-wise gradient checks.

Reproduce closed-book

Closed-book, for a C-contiguous (3, 4) float32 tensor: give strides (4, 1) and offset 9 for [2, 1]; after transpose give shape (4, 3), strides (1, 4), non-contiguous, with [1, 2] aliasing offset 9. Then classify view, reshape, transpose, x[0] and x[[0, 2]] as copy or view.

Traps

Assuming transpose or reshape copies. Both can be views: writing through them mutates the base, and the docs mutate t through b to prove it.
Assuming view() always works. It needs a compatible layout; a transposed tensor needs reshape (which may copy) or .contiguous() first.
Mixing byte and element strides. Torch strides count elements; NumPy .strides counts bytes. Convert with itemsize before comparing.

Practice

The section mini-project covers this: noncontiguous and transposed inputs to your engine, view-versus-copy storage checks with shared-storage assertions, and offset accounting verified against the framework on adversarial shapes.

Self-test

A (3, 4) float32 tensor is C-contiguous. What are its strides, and what is the offset of [2, 1]?
Transpose it. What are the new shape, strides and contiguity, and which base offset does [1, 2] read?
Which of these copy: x[0], x[[0, 2]], x.reshape(-1), x.transpose(0, 1)?
Y = Y + X versus Y[:] = X + Y: which reallocates, and how do you tell?

Completion criterion

Stop when

You compute strides, offsets and aliasing for transpose and expand cases from memory, and your engine reads a transposed input correctly either through strides or through an explicit contiguous copy.

Mark passed, then open 1.7 Only mark this when you can do it without notes.

Leads to

1.7 Aliasing, mutation and module ownership See what views plus mutation do to gradients.
2.1 PyTorch tensors: creation, indexing, storage and strides Meet storage, strides and views in the framework.

Comes after

2.1 PyTorch tensors: creation, indexing, storage and stridesRebuild stride and layout reasoning inside your own autodiff engine.

Resources for this unit

D2L - tensors, autograd, MLP and backpropagation
D2L authorsrequiredimplementation
Use chapter_preliminaries ndarray and autograd, then MLP, backpropagation, and numerical stability and initialization. Recreate scalar operations, affine maps, activation, reduction, stable cross-entropy, topological backward and unbroadcast in a blank file.
PyTorch Tensor Views
PyTorchreferencereference
Read the view and storage sharing example (data_ptr, mutating the view changes the base), the transpose non-contiguous example, the full view-ops list, and the special notes: reshape may copy, contiguous copies only when needed, basic indexing returns views while advanced indexing copies.
`torch.Tensor.stride` - stride and storage offset
PyTorchreferencereference
Read the stride definition: strides are per-dimension element steps, and with storage_offset they locate every element. Combine with the Tensor Views page for transposed and expanded layouts.

Also in this section

Understanding Deep Learning - Chapters 3 to 7
Simon J. D. Princeprimarytheory
Read Chapters 3-7, emphasizing Chapter 5 for likelihood-derived losses and Chapter 7 for backpropagation and initialization. Use notebooks 3.1, 4.1, 5.1, 6.1 and 7.1 only as equation and plot companions.
TensorTonic - Build Micrograd from Scratch
TensorTonicrequiredpractice
Use the 25-problem Micrograd study plan in order for derivatives, gradient checks, value graphs, reverse mode, accumulation, operators, modules, losses and training. Preserve local tests as the evidence instead of a platform score.
Deep-ML candidate - autograd and loss problems
Deep-MLoptionalpractice
After the five-sample platform check passes, use at most one of 26 Basic Autograd Operations, 997 Manual Cross-Entropy Backprop or 1227 Stable Softmax as a changed-prompt drill.
CS231n Linear Classification - Softmax classifier and numeric stability
Stanford CS231nreferencereference
Read the Softmax classifier section for the cross-entropy form \(L_i = -f_{y_i} + \log \sum_j e^{f_j}\), then the Practical issues: Numeric stability part for the max-subtraction code f -= np.max(f).
CS231n Neural Networks 3 - Gradient checks
Stanford CS231nreferencereference
Read the Gradient Checks section only: centered formula with \(O(h^2)\) error, relative error with thresholds, double precision, step size \(h\) near 1e-5, kinks, few datapoints, burn-in, regularization and dropout discipline, and checking a few dimensions per parameter.
PyTorch Broadcasting semantics
PyTorchreferencereference
Read General semantics for the trailing-dimension rules, prepend-1 alignment and max-size result, with the (5, 1, 4, 1) plus (3, 1, 1) examples; read In-place semantics for why an in-place tensor may not change shape.
`torch.autograd.gradcheck` API and tolerances
PyTorchreferencereference
Read the gradcheck signature defaults (eps 1e-6, atol 1e-5, rtol 1e-3) and the note that inputs with overlapping memory, such as expand outputs, will likely fail the check.
`torch.nn.functional.cross_entropy` - `reduction` and `ignore_index`
PyTorchreferencereference
Read the ignore_index and reduction parameters: ignored targets contribute no gradient, and mean reduction averages over the non-ignored targets only.
PyTorch Autograd mechanics - in-place operations
PyTorchreferencereference
Read In-place operations with autograd (why in-place rewrites the graph and is discouraged) and In-place correctness checks (the version counter and the error on accessing a modified saved tensor).
`torch.nn.Module` - parameters, buffers and state
PyTorchreferencereference
Read the parameters and buffers documentation: modules collect parameters recursively for the optimizer, while register_buffer holds persistent non-parameter tensors saved alongside them.