Linear Systems, Span & Least Squares

Understand how linear equations describe constraints, when a system can be solved exactly and how least squares handles noisy data.

By the end of this lesson

You will be able to

Translate two linear equations into matrix form.
Explain span and linear independence at an intuitive level.
Recognize when a square system has a unique solution.
Explain least squares as minimizing the sum of squared residuals.

A system asks for one set of values that satisfies several equations

For example, 2x + y = 5 and x - y = 1 can be written as Aθ = b. The rows of A encode equations, θ contains unknown values and b contains their right-hand sides. A unique solution requires a square system with independent constraints. Real datasets are often rectangular and noisy, so an exact solution may not exist.

A = np.array([[2.0, 1.0],
              [1.0, -1.0]])
b = np.array([5.0, 1.0])

solution = np.linalg.solve(A, b)
print(solution)  # [2. 1.]

Span and least squares

The span of a set of columns is the collection of all vectors that can be made from their linear combinations. If the target vector is not exactly in that span, least squares finds the combination whose residual is smallest in squared Euclidean length. It minimizes ||Aθ - b||₂². This is the bridge from solving exact equations to fitting a line through imperfect observations.

A = np.array([[1.0, 1.0],
              [1.0, 2.0],
              [1.0, 3.0]])
b = np.array([2.0, 2.9, 4.2])

solution, residuals, rank, singular_values = np.linalg.lstsq(A, b, rcond=None)
print(solution)  # intercept and slope

Apply what you learned

Exercise

Compare exact and best-fit solutions

Solve one exact system and then fit a line to three noisy observations. Notice why the second problem needs a best-fit answer.

Solve 2x + y = 5 and x - y = 1 using np.linalg.solve.
Verify both equations by substituting the returned values.
Create a design matrix with a column of ones and x-values [1, 2, 3].
Fit noisy targets [2.0, 2.9, 4.2] using np.linalg.lstsq.
Calculate residuals and explain why they are not all zero.

Done when

The first system satisfies both equations. The fitted line returns finite coefficients and a nonzero residual pattern that you can explain as the effect of noisy observations.

Free references

Study resources

3 links

01
Linear combinations, span, and basis vectors | Chapter 2 Video
Watch the single chapter for linear combinations, span and independent directions.

Done
02
numpy.linalg.lstsq Article
Read the description and first example; note the solution, residuals and rank that are returned.

Done
03
numpy.linalg.solve Article
Read the square-system example and the limitation that the matrix must be square and full-rank.

Done

Check your understanding

Self-test

2 questions

Question 1

What does least squares minimize?
Question 2

Why might a real dataset have no exact solution to Aθ = b?

Completion check

Move on when

You can solve a small square system, fit an overdetermined system and explain the difference between exact solutions and least-squares solutions.

Mark passed and continue Back to module