Limits, partial and directional derivatives, and gradients

1. Univariate derivatives, partials and gradients

   Mathematics for Machine Learning - Ch. 5

   Section 5.1, Differentiation of Univariate Functions (pp. 141-145): difference quotient (5.3), derivative as a limit (5.4), polynomial derivation (5.5)-(5.6), product, quotient, sum and chain rules (5.29)-(5.32). Section 5.2, Partial Differentiation and Gradients (pp. 146-149): partial derivatives, gradient as a row vector, the chain rule as matrix multiplication (5.48)-(5.53)

   Done
2. Derivatives as linear operators

   MIT 18.S096 - Matrix Calculus for ML

   Lecture 1, Part 2 notes (course notes Chapter 2, Sections 2.1-2.3): \(df = f'(x)[dx]\) as a linear operator, scalar-valued case \(df = (\operatorname{grad} f) \cdot dx\) with the gradient as the transpose of the derivative

   Done
3. Gradient geometry and directional derivatives

   MIT 18.02SC - Multivariable Calculus Unit 2

   Sessions 35 to 38 (Part B): gradient definition and perpendicularity to level curves, the proof, a worked example, directional derivatives

   Depth · Refresher only. Open only if partial derivatives or gradients are weak; it does not replace the MML derivations.

   Done

Focus on

• The derivative is the limit of the difference quotient; each partial derivative holds all other variables constant.
• For unit \(u\), \(D_u f = \operatorname{grad} f \cdot u\), and the maximum over \(u\) is \(\|\operatorname{grad} f\|\) at \(u = \operatorname{grad} f / \|\operatorname{grad} f\|\) by Cauchy-Schwarz.
• MML writes gradients as row vectors and 18.S096 as column vectors; the two differ by a transpose and the chain rule changes shape with the choice.
• Existing partials do not imply differentiability; continuous partials do.

Reproduce closed-book

Traps

• Reading \(D_u f\) as a partial derivative. A partial fixes an axis; a directional derivative moves along \(u\) and needs the full gradient dot product.
• Assuming equal partials imply a smooth function. With \(f(x, y) = x^2 y/(x^4 + y^2)\) off the origin and \(f(0, 0) = 0\), both partials at the origin are 0, yet \(f\) equals 1/2 along \(y = x^2\), so \(f\) is discontinuous and not differentiable there.
• Mixing gradient conventions across sources and silently dropping a transpose, which flips every later matrix shape.

Practice

Self-test

1. State the difference quotient and the assumption that turns it into the derivative.

   Show answer Got it Missed

2. For \(f(x_1, x_2) = x_1^2 x_2 + x_1 x_2^3\), give \(\operatorname{grad} f\) at (1, 2) and \(D_u f\) for \(u = (1, 1)/\sqrt{2}\).

   Show answer Got it Missed

3. Give a function with both partials zero at the origin that is not differentiable there, and the curve that exposes it.

   Show answer Got it Missed

4. MML gives a \(1 \times 2\) gradient where 18.S096 gives \(2 \times 1\) for the same \(f: \mathbb{R}^2 \to \mathbb{R}\). Which is right?

   Show answer Got it Missed

Leads to

• 2.2 Jacobians, Hessians and Taylor approximations Gradients stack into Jacobian rows and Hessian entries.
• 1.3 Computation graphs and reverse accumulation Reverse accumulation computes these partials on a graph; check it against centered differences.

Resources for this unit

• MIT 18.S096 - Matrix Calculus for ML
  MIT OpenCourseWareprimarycourse
  Use lectures 1-5, the Hessian portion of lecture 7 and computational-graph portion of lecture 8; do the supplied problems and notebooks.
• Mathematics for Machine Learning - Ch. 5
  Deisenroth, Faisal & Ongreferencetheory
  Use Chapter 5 as the compact derivative, Jacobian, chain-rule and backpropagation reference.
• MIT 18.02SC - Multivariable Calculus Unit 2
  MIT OpenCourseWareoptionalrefresher
  Use sessions 24-38 only if partial derivatives, gradients, Hessians or the multivariable chain rule are weak.