Unit 1.3 · about 5 h
53 / 324The four subspaces, rank and solution sets
Rank tells you which outputs are reachable and which parameter changes leave outputs unchanged. These are different failure modes, even before fitting data.
- derive
You can now
- Find bases for all four fundamental subspaces and state their ambient spaces and dimensions.
- Prove rank-nullity and the orthogonal-complement relations for a real matrix.
- Describe every solution of \(Ax=b\) and distinguish existence from identifiability of \(x\).
Covers
- Column space, row space, null space and orthogonal complements
- Rank, invertibility, solution sets and identifiability
Study, in this order
2 steps
- Null spaces, solution sets and the four subspacesMIT 18.06 - Linear Algebra
Video Lectures 6 to 8, Column space and nullspace through Solving \(Ax=b\); Lecture 10, The four fundamental subspaces; Lecture 14, Orthogonal vectors and subspaces
Depth · In Lecture 14, focus on row/null and column/left-null orthogonality. The norm and angle calculations belong to the next unit.
- Rank-nullity and orthogonal complementsMathematics for Machine Learning
Chapter 2, section 2.7.3 (pp. 58 to 60), Theorem 2.24 on p. 60; Chapter 3, section 3.6 (pp. 79 to 80)
Focus on
- For \(A\) in \(\mathbb{R}^{m \times n}\) of rank \(r\), column and left-null spaces live in \(\mathbb{R}^m\) with dimensions \(r\) and \(m-r\). Row and null spaces live in \(\mathbb{R}^n\) with dimensions \(r\) and \(n-r\).
- Prove rank-nullity by extending a basis of \(\ker(A)\) to a domain basis; show that the images of the added vectors form a basis of \(\operatorname{im}(A)\).
- Each row dotted with \(x\) is a component of \(Ax\). This gives \(\ker(A)=\operatorname{row}(A)^\perp\); apply the same argument to \(A^\top\) for the other pair.
- \(Ax=b\) is solvable iff \(b\) lies in \(\operatorname{col}(A)\). When solvable, its full solution set is \(x_p+\ker(A)\), so unique coefficients require full column rank.
Reproduce closed-book
dtype=np.float64 and no random draws: matrix_rank(A) gives 2; the vector 2-norms of A@(-2,1,0), A.T@(-1,-1,1), and A@(3,0,4)-b each give 0.0, below 1e-12. The left-null vector dotted with \((3,4,8)\) gives approximately 1, certifying the hand inconsistency argument.Traps
- Pivot columns must be taken from the original \(A\) to form a column-space basis. Row reduction preserves the row space and kernel, not the column space as a subset of \(\mathbb{R}^m\).
- A tall matrix can have a unique solution for each attainable \(b\) without being surjective. Full column rank and full row rank answer different questions.
- Numerical rank depends on a singular-value threshold. Do not confuse that decision with exact dependence in a symbolic matrix.
Practice
18.06 Problem set 3: incompatible spaces and solution-set claims
Problem set 3, Problem 2 (Strang 4th ed., section 3.2, #24), then Problem 10 (section 3.4, #13)
The official solutions reproduce the statements. Attempt both before reading the solutions. Use dimensions for the first problem and compute a 2 by 2 counterexample for the minimum-length claim in the second.
Self-test
-
Why are the images of the added basis vectors independent in the rank-nullity proof?
A zero combination of their images puts the corresponding domain combination in \(\ker(A)\). Express it in the chosen kernel basis. Independence of the extended domain basis then forces all added-vector coefficients to be zero. Finite dimension permits the basis extension; linearity moves the combination through \(A\). -
When does a real \(m\) by \(n\) matrix give exactly one solution for every \(b\) in \(\mathbb{R}^m\)?
Existence for every \(b\) requires rank \(m\); uniqueness requires rank \(n\). Both imply \(m=n=\operatorname{rank}(A)\), so \(A\) is square and invertible. -
Why is the full solution set \(x_p+\ker(A)\), not \(\operatorname{span}\{x_p,\ker(A)\}\)?
Linearity gives \(A(x_p+z)=b\) when \(Az=0\). Conversely, any solution \(x\) satisfies \(A(x-x_p)=0\). The coefficient of \(x_p\) is fixed at one unless \(b=0\); scaling \(x_p\) generally scales \(b\). -
How can a left-null vector disprove solvability?
If \(A^\top l=0\), then \(l^\top Ax=0\) for every \(x\). Thus \(l^\top b\) must vanish for a solution to exist. For the worked matrix \(l=(-1,-1,1)\) and \(b=(3,4,8)\), it is 1, a contradiction.
Completion criterion
Stop when
You derive all four spaces and the full solution family for the worked matrix, prove rank-nullity, and reject the incompatible-space request in Problem set 3 using dimensions rather than trial and error.
Mark passed, then open 1.4 Only mark this when you can do it without notes.Leads to
- 1.5 Projections, residuals and least squares Project an unattainable right-hand side onto the column space.
- 2.1 Least squares: residuals, normal equations and stable solvers Diagnose non-identifiable coefficients in an applied design matrix.
Comes after
- 1.2 Linear maps, span, independence and change of basisDescribe what the map reaches and what it loses.
Resources for this unit
- MIT 18.06 - Linear Algebra Follow lectures 1-10, 14-17, 21-22, 25, 27, 29-31 and 33 with matching problems. This is the ordered first pass.
- Mathematics for Machine Learning Use Chapters 2-4 and §10.2 to check notation or re-derive a result. Do not treat it as a second course.
Also in this section
- MIT 6.042J - Proof strand Study textbook Chapters 1 and 3, §§4.1-4.4 and §§5.1-5.4 with matching Unit 1 problems. Leave the rest of the course.
- MIT 18.065 - Applied matrix bridge Use lectures 3-10 after 18.06 for SVD, PCA, low-rank approximation, matrix norms and solver difficulty.
- Introduction to Applied Linear Algebra Sample Chapters 1-13 and the extra exercises for computational practice in fitting, least squares and pseudoinverses.
- Essence of Linear Algebra Open only when a geometric picture is missing; it does not replace derivation or problem solving.
- NumPy broadcasting and matmul shape rules Read General broadcasting rules and Broadcastable arrays, then
matmulNotes for stacked matrices and one-dimensional promotion. These close the pool's array-semantics gap. - Fundamentals of Numerical Computation: conditioning and operation counts Python edition, section 2.8 through Residual and backward error for the fixed-matrix perturbation bound; section 2.5.3 for triangular-solve and LU flop counts. Use Exercise 2.8.7 for a condition-number proof, not the unseeded Hilbert experiment.