Handson-ml: Calculus Notebook

[ajkn]

Can you please inform whether there are any plans to develop Calculus Ipython Notebook?

[ageron]

Hi @ajkn ,

I've been wanting to write this notebook for so long, but I never found the time, so let me just quickly summarize the main ideas here. You don't actually need to know calculus in order to practice Machine Learning, since TensorFlow (or other Deep Learning libraries) take care of that for you, as we will see below.

All you really need is a good intuition of what a derivative is. In short, you can think of the derivative of a function as the slope of that function at each point. For example, if you consider the function f(x) = 2x, then the slope is constant, it is written f'(x), and it is always equal to 2: f'(x)=2.

Now let's consider a more complicated function, g(x)=x^2-3x+5, the slope keeps changing! In fact, the derivative of g(x) is g'(x)=2x-3. How can you figure this out yourself? Well, let's start by figuring out the slope between any two points x1 and x2 on a function: it is of course equal to the difference on the vertical axis (g(x2) - g(x1)) divided by the difference on the horizontal axis (x2 - x1):
slope_of_g(x1, x2) = (g(x2)-g(x1))/(x2-x1)

If x2 = x1 + e, you can rewrite this equation as:
slope_of_g(x1, x1+e) = (g(x1+e)-g(x1))/e

As x2 approaches x1 (i.e., as e gets smaller and smaller), you get a more and more precise approximation of the slope at the point x1. That's actually the definition of the derivative of g(x) at the point x: it is the limit of the slope between the point at x and the point at x+e as e approaches zero ("limit when e approaches 0" is noted lim e->0):
g'(x) = lim e->0 (g(x + e) - g(x)) / e

So, let's look at the example above:
g'(x) = lim e->0 (g(x + e) - g(x)) / e
Let's replace g(x) with its definition:
g'(x) = lim e->0 [(x+e)^2-3(x+e)+5 - (x^2 - 3x + 5)]/e
Then let's gradually simplify this equation:
g'(x) = lim e->0 [x^2 + 2xe + e^2 - 3x - 3e + 5 - x^2 +3x - 5]/e
g'(x) = lim e->0 [2xe + e^2 -3e]/e
g'(x) = lim e->0 [2x - 3 + e]
When e approaches zero, of course 2x - 3 + e approaches 2x - 3, so:
g'(x) = 2x - 3

Now in practice, you don't have to go through this process at all, as TensorFlow uses Reverse Mode Autodiff (see appendix D) to compute this for you. For example:

import tensorflow as tf
x = tf.placeholder(dtype=tf.float32)
g = tf.square(x) - 3.0*x + 5.0
[gp] = tf.gradients(g, x)
init = tf.global_variables_initializer()
with tf.Session():
    init.run()
    print("g(6) = 6 * 6 - 3 * 6 + 5  =", g.eval(feed_dict={x: 6.}))
    print("g'(7) = 2 * 7 - 3 =", gp.eval(feed_dict={x: 7.}))

This code outputs the correct result:

g(6) = 6 * 6 - 3 * 6 + 5  = 23.0
g'(7) = 2 * 7 - 3 = 11.0

The point at which the derivative of a function is zero is a point where the slope is horizontal. A function's minimum or maximum is always located at a point where the derivative is zero. If the derivative of a function is zero at a given point, it can be a minimum or maximum (it could be a local minimum/maximum, not necessarily the global minimum), or it could be a plateau, and so on. So when people search for a function's minimum (analytically, not using gradient descent), they usually compute the function's derivative first, and then try to find the points at which it is zero.

If the function has several inputs instead of one, for example f(x,y)=xy + 3x - 5y + 2, then you must compute the so-called "partial" derivative of the function with regards to x, which is written ∂f/∂x, just like a regular derivative, but keeping all other inputs constant (in this case y stays constant while you try to find the slope (f(x+e,y) - f(x,y))/e as e approaches zero, then you do the same thing for y. If you put all the partial derivatives in a vector, it is called a "gradient vector", noted ∇u f (u is the vector (x,y) and it should be a subscript).

Gradient descent is quite simple: first, estimate the slope of the cost function with regards to the model parameters (i.e., find all the partial derivatives of the cost function with regards to each model parameter). This gives you a vector that points "uphill": if you move the parameters in this direction, the cost will go up. So instead, you subtract the gradient vector from the model parameter vector in order to reduce the model's error (as measured by the cost function).

Hope this helps.

[ageron]

The calculus notebook is now available: math_differential_calculus.ipynb