Linear Regression: Put the Math Together

Connect a feature matrix, a parameter vector, squared error and gradient descent in one small linear-regression implementation.

By the end of this lesson

You will be able to

Represent linear predictions as Xw + b.
Explain the relationship between least squares and mean squared error.
Implement gradient descent for slope and intercept.
Evaluate on a holdout set and compare against a simple baseline.

A linear model is a weighted sum plus an intercept

For one input, the prediction is ŷ = wx + b. For a matrix of examples, it is ŷ = Xw + b. The slope and intercept are parameters. The residual is ŷ - y, and mean squared error averages the squared residuals. Least squares gives a direct best-fit solution for the squared-error objective; gradient descent reaches a solution through repeated updates.

def predict(x, w, b):
    return w * x + b

def mse(y, y_hat):
    return np.mean((y_hat - y) ** 2)

Make evaluation honest

Split observations before fitting or scaling. For this exercise, sort by datetime and reserve the final 20% as a chronological holdout. Calculate the feature mean and standard deviation on the training portion only, then apply those same values to the holdout. Compare the model with a baseline that always predicts the training target mean. A relationship between temperature and demand is not proof that temperature caused the observed demand.

mean_x = x_train.mean()
std_x = x_train.std()
if std_x == 0:
    raise ValueError("Training feature has no variation")

x_train_scaled = (x_train - mean_x) / std_x
x_test_scaled = (x_test - mean_x) / std_x

Apply what you learned

Exercise

Build a one-feature bike-demand regression

Use the hourly Bike Sharing dataset from Module 01. Predict cnt from the normalized temp feature with a single linear model. Do not use scikit-learn; the goal is to make every mathematical operation visible.

Load hour.csv, sort by dteday and record the downloaded filename and actual row count.
Select temp as the single input and cnt as the target. Check missing values and confirm both columns are numeric.
Reserve the final 20% of rows as a chronological test set. Compute the mean and standard deviation of temp on the training set only, then scale both splits with those training statistics.
Initialize w and b. Implement batch gradient descent for MSE using dw = (2/n) Σ((ŷ - y)x) and db = (2/n) Σ(ŷ - y).
Record training loss over iterations and stop when it has stabilized or reached a clear plateau. Plot the loss curve and a scatter plot with the fitted line.
Evaluate the holdout MAE and RMSE. Compare them with a constant-mean baseline calculated from the training targets.
Use np.linalg.lstsq as an independent least-squares check after adding an intercept column. Explain any small differences and list at least two limitations of the one-feature model.

Done when

A reproducible script trains without scikit-learn, keeps all scaling statistics inside the training split, shows a decreasing loss curve, reports holdout MAE and RMSE beside a training-mean baseline, and includes a short interpretation that avoids causal claims.

Free references

Study resources

2 links

01
The Essence of Linear Regression Video
Watch the single walkthrough sections on fitting a line, residuals and least squares. Then implement the model yourself.

Done
02
numpy.linalg.lstsq Article
Read the first example to use least squares as an independent numerical cross-check.

Done

Check your understanding

Self-test

2 questions

Question 1

What does Xw + b represent in a one-feature linear model?
Question 2

Why are scaling statistics calculated only on the training split?

Completion check

Move on when

You can implement a one-feature linear regression model from scratch, evaluate it on a chronological holdout and explain its assumptions and limitations.

Back to module