PyTorch tensors: creation, indexing, storage and strides

Every later failure in this stage, from silent dtype promotion to aliasing bugs, starts as a wrong picture of what a tensor owns. Storage, stride and device are that picture.

explain
implement

You can now

Create tensors with explicit dtype and device, and predict shape, dtype and device after indexing, concatenation and device moves.
Compute a storage offset from shape, stride and index, and classify a tensor as contiguous, transposed or sliced from its stride.
Predict when a torch tensor and a NumPy array share memory, and when an op copies instead.

Covers

Tensor creation, indexing, devices and dtypes
Storage, stride, offset, contiguity and layout

Study, in this order

2 steps

Tensors: initialization, attributes, indexing and NumPy sharing
PyTorch - Learn the Basics

Initializing a Tensor; Attributes of a Tensor; Operations on Tensors through Single-element tensors; Bridge with NumPy

Done
Tensor storage, strides and transpose sharing
Deep Learning with PyTorch, 2nd edition - framework foundations

Book Sections 3.8 and 3.9 (printed pp. 55-64): storage, offset, size and stride, transposing without copying, contiguous tensors. Notebook cells: Cells using points.storage(), storage_offset(), stride(), the transpose storage-identity check, is_contiguous() and contiguous()

Depth · Use only the storage, stride, transpose and contiguity cells. Skip the named-tensor notebook; named dimensions are out of scope here. The book calls points.storage(); on PyTorch 2.14 that warns that TypedStorage is deprecated, so use untyped_storage() or data_ptr() for the identity checks.

Done

Focus on

dtype is inferred at creation and the device is CPU unless you choose one. Tensor.to() returns a tensor (the same one when nothing changes), so assign the result. Module.to() is the exception: it converts the module in place.
Element offset is storage_offset plus the sum over dims of index times stride. Transpose and slicing change stride and offset, not storage.
Basic indexing gives views. .numpy() and torch.from_numpy() share one CPU allocation, so in-place edits leak across the boundary.
contiguous() returns the same tensor when already contiguous and copies otherwise. Never branch on reshape returning a view.

Reproduce closed-book

Closed-book, take x = torch.zeros(3, 4). Write the shape, stride, storage_offset and is_contiguous of x, xt = x.t() and y = x[::2]. Show that xt[2][1] and x[1][2] name the same element, and assert the shared storage with data_ptr.

Traps

Reading transpose or permute as a copy. They return views, so writing through them changes the base.
Trusting reshape to return a view. It may copy; check data_ptr instead of assuming.
Calling x.to(device) on a tensor and then using x. The call returns the moved tensor; x stays where it was. (model.to(device) does move the module in place, which is why the habit forms.)

Practice

Learn PyTorch 00: tensor creation, matmul, seeds and squeeze (tasks 2 to 10)

Exercises, tasks 2 to 10: (7, 7) and (2, 3) creation, matmul with transpose, manual_seed replays, GPU seed lookup, argmax and argmin, squeeze of (1, 1, 1, 10)

Write the tasks in your own script, not in the solutions notebook. Task 1 is background reading; tasks 2 to 10 are the checked work. Run the GPU tasks on whatever device you have and say which one you used.

Self-test

x = torch.zeros(3, 4). What are the strides of x and of x.t(), and is either contiguous?
Which element does xt[2][1] alias, and at what storage offset?
t = torch.ones(5); n = t.numpy(); t.add_(1). What is n now, and why?
When does c = t.contiguous() copy, and when does it return t itself?

Completion criterion

Stop when

Your written shape, stride and offset table for x, x.t() and x[::2] matches data_ptr and is_contiguous checks, and tasks 2 to 10 run without the solutions notebook.

Mark passed, then open 2.2 Only mark this when you can do it without notes.

Leads to

2.2 Autograd state: views, in-place ops, leaves and accumulation Views share the storage described here; test aliasing against data_ptr.
1.6 Tensor layout: shape, stride, views and copies Rebuild stride and layout reasoning inside your own autodiff engine.

Comes after

1.6 Tensor layout: shape, stride, views and copiesMeet storage, strides and views in the framework.

Resources for this unit

PyTorch - Learn the Basics
PyTorchprimarycourse
Complete Tensors, Datasets and DataLoaders, Transforms, Build Model, Autograd, Optimization, and Save and Load Model. Rebuild each essential example in a small src package instead of retaining a tutorial notebook.
Deep Learning with PyTorch, 2nd edition - framework foundations
Howard Huang, Luca Antiga, Eli Stevens and Thomas Viehmann (Manning)supportimplementation
Read Sections 3.8-3.9 and 5.5 when tensor storage, strides or autograd need a second explanation, with the Chapter 3 and 5 companion notebooks for runnable cells. Inspect the Chapter 4 and 7 code only for a diagnosed data-representation or loading gap.

Also in this section

PyTorch - Autograd mechanics, serialization and reproducibility
PyTorchrequiredreference
Read Autograd mechanics, Serialization semantics and Reproducibility. Test leaf behavior, saved tensors, grad modes, state dictionaries and the exact tolerance promised by one interrupted and resumed run.
Learn PyTorch - selected framework exercises
Daniel Bourkerequiredpractice
Complete exercise 00, tasks 1-5 in 01, tasks 1-7 in 02, tasks 5-13 in 03, tasks 1-3 in 05, and exercises 1-2 in 07. Recreate solutions independently and verify behavior against current official docs.
NeetCode ML - PyTorch closing drills
NeetCodeoptionalpractice
After the module gate, solve Pytorch Basics, Layer Normalization, Batch Normalization and RMS Normalization cold. Explain shapes, state and train-evaluation behavior aloud. These do not replace the package and resume test.
PyTorch Automatic Mixed Precision recipe - Saving/Resuming
PyTorchreferencereference
Read the Saving/Resuming section: save scaler.state_dict() with the model and optimizer state, at the start of an iteration or after scaler.update(), and how to resume with or without AMP.
Hands-On Machine Learning with Scikit-Learn and PyTorch
Aurelien Geron (O'Reilly, 2025)supporttheory
Read Chapter 10, Implementing Mini-Batch Gradient Descent Using DataLoaders and Model Evaluation, for the train and evaluate loops and how batch metrics must be aggregated.