Loss Functions & Gradient Descent

Define an objective that measures prediction error and follow its gradient to update model parameters iteratively.

By the end of this lesson

You will be able to

Distinguish a residual from a loss function.
Explain why mean squared error penalizes larger errors more.
Derive the gradient for a simple line-fitting objective.
Explain the effect of learning rate on convergence.

A loss function turns errors into an objective

A residual is the difference between an observed value and its prediction. Mean squared error averages squared residuals: MSE = (1/n) Σᵢ (ŷᵢ - yᵢ)². Squaring makes large errors count more heavily and prevents positive and negative residuals cancelling. The loss is a scalar function of model parameters, so derivatives tell us how it changes when a parameter changes.

y = np.array([3.0, 5.0, 8.0])
y_hat = np.array([2.5, 5.5, 7.0])
mse = np.mean((y_hat - y) ** 2)
print(mse)

Gradient descent takes repeated small steps

For a parameter vector θ, gradient descent updates θ ← θ - α∇L(θ). The learning rate α controls the step size. A value that is too small can make progress slow; a value that is too large can overshoot or diverge. Check the loss as training proceeds. A falling loss is evidence that the update is moving in a useful direction for the current objective.

w = 0.0
learning_rate = 0.05
for step in range(100):
    y_hat = w * x
    dw = (2 / len(x)) * np.sum((y_hat - y) * x)
    w -= learning_rate * dw

Apply what you learned

Exercise

Fit a line with gradient descent

Implement batch gradient descent for a one-dimensional model ŷ = wx. Keep the dataset tiny so each part of the gradient remains visible.

Create x = [1, 2, 3, 4] and y = [2, 4, 6, 8].
Define MSE as a function of w.
Calculate the derivative dw = (2/n) Σ ((wxᵢ - yᵢ)xᵢ).
Start with w = 0 and run 100 updates with a small learning rate.
Record the loss every few steps and compare its first and final values.

Done when

The loss decreases over training and w moves toward 2. If the loss rises or becomes unstable, explain how you would adjust the learning rate.

Free references

Study resources

2 links

01
The Main Ideas of Fitting a Line to Data Video
Watch the single least-squares explanation to understand the error objective before gradient descent.

Done
02
Gradient Descent, Step-by-Step Video
Watch the standalone gradient-descent walkthrough and focus on the update direction and learning rate.

Done

Check your understanding

Self-test

2 questions

Question 1

What is the gradient descent update for parameters θ?
Question 2

What can happen when the learning rate is too large?

Completion check

Move on when

You can calculate MSE, implement a gradient update and explain how the learning rate changes the optimization behavior.

Mark passed and continue Back to module