Autograd in MinPy

This tutorial is also available in step-by-step notebook version on github. Please try it out!

Writing backprop is often the most tedious and error prone part of a deep net implementation. In fact, the feature of autograd has wide applications and goes beyond the domain of deep learning. MinPy’s autograd applies to any NumPy code that is imperatively programmed. Moreover, it is seemlessly integrated with MXNet’s symbolic program (see for example). By using MXNet’s execution engine, all operations can be executed in GPU if available.

A Close Look at Autograd System

MinPy’s implementation of autograd is insprired from the Autograd project. It computes a gradient function for any single-output function. For example, we define a simple function foo:

In [1]:
def foo(x):
    return x**2


Now we want to get its derivative. To do so, simply import grad from minpy.core.

In [2]:
import minpy.numpy as np  # currently need import this at the same time
from minpy.core import grad

d_foo = grad(foo)
In [3]:

You can also differentiate as many times as you want:

In [4]:
d_2_foo = grad(d_foo)
d_3_foo = grad(d_foo)

Now import matplotlib to visualize the derivatives.

In [5]:
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

x = np.linspace(-10, 10, 200)
# plt.plot only takes ndarray as input. Explicitly convert MinPy Array into ndarray.
plt.plot(x.asnumpy(), foo(x).asnumpy(),
         x.asnumpy(), d_foo(x).asnumpy(),
         x.asnumpy(), d_2_foo(x).asnumpy(),
         x.asnumpy(), d_3_foo(x).asnumpy())

Just as you expected.

Autograd also differentiates vector inputs. For example:

In [6]:
x = np.array([1, 2, 3, 4])
[ 2.  4.  6.  8.]

Gradient of Multivariate Functions

As for multivariate functions, you also need to specify arguments for derivative calculation. Only the specified argument will be calcualted. Just pass the position of the target argument (of a list of arguments) in grad. For example:

In [7]:
def bar(a, b, c):
    return 3*a + b**2 - c

We get their gradients by specifying their argument position.

In [8]:
gradient = grad(bar, [0, 1, 2])
grad_array = gradient(2, 3, 4)
print grad_array
[3.0, 6.0, -1.0]

grad_array[0], grad_array[1], and grad_array[2] are gradients of argument a, b, and c.

The following section will introduce a more comprehensive example on matrix calculus.

Autograd for Loss Function

Since in world of machine learning we optimize a scalar loss, Autograd is particular useful to obtain the gradient of input parameters for next updates. For example, we define an affine layer, relu layer, and a softmax loss. Before dive into this section, please see Logistic regression tutorial first for a simpler application of Autograd.

In [9]:
def affine(x, w, b):
    Computes the forward pass for an affine (fully-connected) layer.
    The input x has shape (N, d_1, ..., d_k) and contains a minibatch of N
    examples, where each example x[i] has shape (d_1, ..., d_k). We will
    reshape each input into a vector of dimension D = d_1 * ... * d_k, and
    then transform it to an output vector of dimension M.
    - x: A numpy array containing input data, of shape (N, d_1, ..., d_k)
    - w: A numpy array of weights, of shape (D, M)
    - b: A numpy array of biases, of shape (M,)
    Returns a tuple of:
    - out: output, of shape (N, M)
    out =, w) + b
    return out

def relu(x):
    Computes the forward pass for a layer of rectified linear units (ReLUs).
    - x: Inputs, of any shape
    Returns a tuple of:
    - out: Output, of the same shape as x
    out = np.maximum(0, x)
    return out

def softmax_loss(x, y):
    Computes the loss for softmax classification.
    - x: Input data, of shape (N, C) where x[i, j] is the score for the jth class
    for the ith input.
    - y: Vector of labels, of shape (N,) where y[i] is the label for x[i] and
    0 <= y[i] < C
    Returns a tuple of:
    - loss: Scalar giving the loss
    N = x.shape[0]
    probs = np.exp(x - np.max(x, axis=1, keepdims=True))
    probs = probs / np.sum(probs, axis=1, keepdims=True)
    loss = -np.sum(np.log(probs) * y) / N
    return loss

Then we use these layers to define a single layer fully-connected network, with a softmax output.

In [10]:
class SimpleNet(object):
    def __init__(self, input_size=100, num_class=3):
        # Define model parameters.
        self.params = {}
        self.params['w'] = np.random.randn(input_size, num_class) * 0.01
        self.params['b'] = np.zeros((1, 1))  # don't use int(1) (int cannot track gradient info)

    def forward(self, X):
        # First affine layer (fully-connected layer).
        y1 = affine(X, self.params['w'], self.params['b'])
        # ReLU activation.
        y2 = relu(y1)
        return y2

    def loss(self, X, y):
        # Compute softmax loss between the output and the label.
        return softmax_loss(self.forward(X), y)

We define some hyperparameters.

In [11]:
batch_size = 100
input_size = 50
num_class = 3

Here is the net and data.

In [12]:
net = SimpleNet(input_size, num_class)
x = np.random.randn(batch_size, hidden_size)
idx = np.random.randint(0, 3, size=batch_size)
y = np.zeros((batch_size, num_class))
y[np.arange(batch_size), idx] = 1

Now get gradients.

In [13]:
gradient = grad(net.loss)

Then we can get gradient by simply call gradient(X, y).

In [14]:
d_x = gradient(x, y)

Ok, Ok, I know you are not interested in x‘s gradient. I will show you how to get the gradient of the parameters. First, you need to define a function with the parameters as the arguments for Autograd to process. Autograd can only track the gradients in the parameter list.

In [15]:
def loss_func(w, b, X, y):
    net.params['w'] = w
    net.params['b'] = b
    return net.loss(X, y)

Yes, you just need to provide an entry in the new function’s parameter list for w and b and that’s it! Now let’s try to derive its gradient.

In [16]:
# 0, 1 are the positions of w, b in the paramter list.
gradient = grad(loss_func, [0, 1])

Note that you need to specify a list for the parameters that you want their gradients.

Now we have

In [17]:
d_w, d_b = gradient(net.params['w'], net.params['b'], x, y)

With d_w and d_b in hand, training net is just a piece of cake.

Less Calculation: Get Forward Pass and Backward Pass Simultaneously

Since gradient calculation in MinPy needs forward pass information, if you need the forward result and the gradient calculation at the same time, please use grad_and_loss to get them simultaneously. In fact, grad is just a wrapper of grad_and_loss. For example, we can get

In [18]:
from minpy.core import grad_and_loss
forward_backward = grad_and_loss(bar, [0, 1, 2])
grad_array, result = forward_backward(2, 3, 4)

grad_array and result are result of gradient and forward pass respectively.