The deeplearning module is useful to create and manipulate different neural network architectures. The core of this module is the NeuralNet class, which stores the definition of each layer of a neural network and a dictionary of learning parameters.
A NeuralNet object can be obtained from deeplearning.create(). The returned network can be provided to graphlab.neuralnet_classifier.create() to train a NeuralNetClassifier.
The general workflow involves three steps:
This function selects a default network architecture depending on the input dataset using simple rules: it creates a 1-layer Perceptron Network for dense numeric input, and a 1-layer Convolution Network for image data input.
Due to the high complexity of netualnet models, the default network does not always work out of the box, and you will often need to tweak the architectures a bit to make it work for your problem. Please see the reference section for a practical guide of tuning deeplearning architectures.
Training a convolutional neural network for digits recognition.
>>> import graphlab
Load the MNIST data. We have downloaded the dataset from the MNIST database, and save into SFrames on S3.
>>> data = graphlab.SFrame('http://s3.amazonaws.com/GraphLab-Datasets/mnist/sframe/train')
>>> test_data = graphlab.SFrame('http://s3.amazonaws.com/GraphLab-Datasets/mnist/sframe/test')
>>> training_data, validation_data = data.random_split(0.8)
We need to make sure all of the images are the same size, since neural nets have fixed input size.
>>> training_data['image'] = graphlab.image_analysis.resize(training_data['image'], 28, 28, 1)
>>> validation_data['image'] = graphlab.image_analysis.resize(validation_data['image'], 28, 28, 1)
>>> test_data['image'] = graphlab.image_analysis.resize(test_data['image'], 28, 28, 1)
Use the builtin NeuralNet architecture for MNIST ( a one layer convolutional neuralnet work)
>>> net = graphlab.deeplearning.create(training_data, target='label')
Layers of the neural network
>>> net.layers
layer[0]: ConvolutionLayer
stride = 2
num_channels = 10
kernel_size = 3
layer[1]: RectifiedLinearLayer
layer[2]: MaxPoolingLayer
stride = 2
kernel_size = 3
layer[3]: FlattenLayer
layer[4]: FullConnectionLayer
num_hidden_units = 100
layer[5]: RectifiedLinearLayer
layer[6]: DropoutLayer
threshold = 0.5
layer[7]: FullConnectionLayer
num_hidden_units = 10
layer[8]: SoftmaxLayer
Parameters of the neural network
>>> net.params
{'learning_rate': 0.001, 'momentum': 0.9}
Tweak some hyperparameters
>>> net.layers[4].num_hidden_units = 10
Train a NeuralNetClassifier using the specified network.
>>> m = graphlab.neuralnet_classifier.create(training_data, target='label',
... network = net,
... validation_set=validation_data,
... metric=['accuracy', 'recall@2'],
... max_iterations=3)
Classify the test data, and output the most likely class label. The score corresponds to the learned probability of the testing instance belonging to the predicited class.
>>> pred = m.classify(test_data)
>>> pred
+--------+-------+----------------+
| row_id | class | score |
+--------+-------+----------------+
| 0 | 0 | 0.998417854309 |
| 1 | 0 | 0.999230742455 |
| 2 | 0 | 0.999326109886 |
| 3 | 0 | 0.997855246067 |
| 4 | 0 | 0.997171103954 |
| 5 | 0 | 0.996235311031 |
| 6 | 0 | 0.999143242836 |
| 7 | 0 | 0.999519705772 |
| 8 | 0 | 0.999182283878 |
| 9 | 0 | 0.999905228615 |
| ... | ... | ... |
+--------+-------+----------------+
[10000 rows x 3 columns]
Predict the top 2 most likely digits
>>> pred_top2 = m.predict_topk(test_data, k=2)
>>> pred_top2
+--------+-------+-------------------+
| row_id | class | score |
+--------+-------+-------------------+
| 0 | 0 | 0.998417854309 |
| 0 | 6 | 0.000686840794515 |
| 1 | 0 | 0.999230742455 |
| 1 | 2 | 0.000284609268419 |
| 2 | 0 | 0.999326109886 |
| 2 | 8 | 0.000261707202299 |
| 3 | 0 | 0.997855246067 |
| 3 | 8 | 0.00118813838344 |
| 4 | 0 | 0.997171103954 |
| 4 | 6 | 0.00115600414574 |
| ... | ... | ... |
+--------+-------+-------------------+
[20000 rows x 3 columns]
Evaluate the classifier on the test data. Default metrics are accuracy, and confusion matrix.
>>> eval_ = m.evaluate(test_data)
>>> eval_
{'accuracy': 0.979200005531311, 'confusion_matrix':
+--------------+-----------------+-------+
| target_label | predicted_label | count |
+--------------+-----------------+-------+
| 0 | 0 | 969 |
| 2 | 0 | 2 |
| 5 | 0 | 2 |
| 6 | 0 | 9 |
| 7 | 0 | 1 |
| 9 | 0 | 2 |
| 1 | 1 | 1126 |
| 2 | 1 | 2 |
| 6 | 1 | 2 |
| 7 | 1 | 3 |
| ... | ... | ... |
+--------------+-----------------+-------+
[64 rows x 3 columns]}
There are two builtin example architectures for reproducing the MNIST and ImageNet experiments. In References, see ‘Imagenet Classification with Deep Convolutional Neural Networks’ for ImageNet architecture explanation and ‘Gradient-based Learning Applied to Document Recognition’ for MNIST architecture explanation:
>>> mnist_net = graphlab.deeplearning.get_builtin_neuralnet('mnist')
>>> imagenet_net = graphlab.deeplearning.get_builtin_neuralnet('imagenet')
It is possible to create a network from a dataset. GraphLab intelligently chooses a network architecture based on input data shape. If the input is of image type, a convolutional neural network is chosen. Otherwise, a simple multi-layer perceptron network is chosen.
# Import graphlab and load the MNIST dataset.
>>> import graphlab as gl
>>> dataset = gl.SFrame('http://s3.amazonaws.com/GraphLab-Datasets/mnist/sframe/train6k')
# Create a default NeuralNet for MNSIT data.
>>> net = gl.deeplearning.create(dataset, target='label')
# The architecture is represented as a list of `Layer` objects.
>>> net.layers
layer[0]: ConvolutionLayer
stride = 2
num_channels = 10
kernel_size = 3
layer[1]: MaxPoolingLayer
stride = 2
kernel_size = 3
layer[2]: FlattenLayer
layer[3]: FullConnectionLayer
num_hidden_units = 100
layer[4]: RectifiedLinearLayer
layer[5]: DropoutLayer
threshold = 0.5
layer[6]: FullConnectionLayer
num_hidden_units = 10
layer[7]: SoftmaxLayer
# And the learning parameter is stored as a dictionary.
>>> net.params
{'learning_rate': 0.001, 'momentum': 0.9}
# The NeuralNet object has a human readable string representation.
>>> net
### network layers ###
layer[0]: ConvolutionLayer
stride = 2
num_channels = 10
kernel_size = 3
layer[1]: MaxPoolingLayer
stride = 2
kernel_size = 3
layer[2]: FlattenLayer
layer[3]: FullConnectionLayer
num_hidden_units = 100
layer[4]: RectifiedLinearLayer
layer[5]: DropoutLayer
threshold = 0.5
layer[6]: FullConnectionLayer
num_hidden_units = 10
layer[7]: SoftmaxLayer
### end network layers ###
<BLANKLINE>
### network parameters ###
learning_rate = 0.001
momentum = 0.9
### end network parameters ###
Most of the time, it is easy to start with a default NeuralNet, and tweak the layers and parameters to fit your problem. However, sometimes it is desirable to create a NeuralNet from scratch. We provide two templated class to make it easy: MultiLayerPerceptrons, and ConvolutionNet.
Create a MultiLayerPerceptrons network with 3 hidden layers, the last layer is used for output.
>>> percpt_net = gl.deeplearning.MultiLayerPerceptrons(num_hidden_layers=3,
num_hidden_units=[15,10,10])
Create a ConvolutionNet with 2 convolution layers and 1 output layer with 10 units.
>>> conv_net = gl.deeplearning.ConvolutionNet(num_convolution_layers=2,
kernel_size=3,
num_channels=10,
num_output_units=10)
Stack the 2-layer perceptron net on top of the convolution net.We re-create the ConvolutionNet so that output size is 0, so we can define an output layer of our own.
>>> conv_net = gl.deeplearning.ConvolutionNet(num_convolution_layers=2,
kernel_size=3,
num_channels=10,
num_output_units=0)
>>> conv_net.layers.extend(percpt_net.layers)
>>> conv_net.layers
layer[0]: ConvolutionLayer
stride = 1
num_channels = 10
kernel_size = 3
layer[1]: RectifiedLinearLayer
layer[2]: MaxPoolingLayer
stride = 1
kernel_size = 3
layer[3]: ConvolutionLayer
stride = 1
num_channels = 10
kernel_size = 3
layer[4]: RectifiedLinearLayer
layer[5]: MaxPoolingLayer
stride = 1
kernel_size = 3
layer[6]: FlattenLayer
layer[7]: FullConnectionLayer
num_hidden_units = 15
layer[8]: SigmoidLayer
layer[9]: FullConnectionLayer
num_hidden_units = 10
layer[10]: SigmoidLayer
layer[11]: FullConnectionLayer
num_hidden_units = 10
layer[12]: SoftmaxLayer
Verify that the layers form a valid network architecture.
>>> conv_net.verify()
True
The NeuralNet object is used to create a NeuralNetClassifier.
>>> model = gl.neuralnet_classifier.create(dataset, target='label', network=net)
To change the a specific layer, simply modify its attribute.
>>> net.layers[-2].num_hidden_units=2
To add or remove an layer, simply use the python list operation.
>>> del net.layers[-1]
>>> net.layers.append(gl.deeplearning.layers.SoftmaxLayer())
To add parameters to the Neuralnet, simply use the python dict operation.
>>> net.params['learning_rate'] = 0.1
NeuralNet object can be saved into a text file using save().
>>> net = gl.deeplearning.create(dataset, target='label')
>>> net.save('mynet')
It can be loaded back using load().
>>> net2 = gl.deeplearning.load('mynet')
Use loads() to load a network from string.
>>> net_string = file('mynet').read()
>>> net2 = gl.deeplearning.loads(net_string)
| deeplearning.create | Creates a NeuralNet given the input dataset. |
| deeplearning.load | Load a saved NeuralNet object from url. |
| deeplearning.loads | Load a saved NeuralNet object from string. |
| deeplearning.NeuralNet | NeuralNet defines the architecture and learning parameters of a neural network. |
| deeplearning.MultiLayerPerceptrons | A specific NeuralNet architecture for general classification problem. |
| deeplearning.ConvolutionNet | A specific NeuralNet architecture used in image classification problem. |
| deeplearning.layers | |
| deeplearning.layers.FullConnectionLayer | A connection layer, connecting every unit in the incoming layer to every unit defined in this layer. |
| deeplearning.layers.FlattenLayer | A layer which takes a 2-d input coming from a convolution layer or pooling layer and flattens it into something a form a fully connected layer can take. |
| deeplearning.layers.SigmoidLayer | This layer takes all inputs and transforms them by the function listed below. |
| deeplearning.layers.RectifiedLinearLayer | This layer takes all inputs and transforms them by the function listed below. |
| deeplearning.layers.TanhLayer | This layer takes all inputs and transforms them by the function listed below. |
| deeplearning.layers.SoftplusLayer | This layer takes all inputs and transforms them by the function listed below. |
| deeplearning.layers.ConvolutionLayer | A connection layer which convolves a learned linear filter over the input, creating a feature map. |
| deeplearning.layers.MaxPoolingLayer | Max-Pooling partitions the input rectangles denoted by the kernel size. |
| deeplearning.layers.SumPoolingLayer | Sum-Pooling partitions the input rectangles denoted by the kernel size. |
| deeplearning.layers.AveragePoolingLayer | Average-Pooling partitions the input rectangles denoted by the kernel size. |
| deeplearning.layers.DropoutLayer | A dropout layer takes all inputs, and with probability ‘threshold’, sets it to 0. |
| deeplearning.layers.LocalResponseNormalizationLayer | Local response normalization normalizes output values by the values present in neighboring maps in corresponding positions. |
| deeplearning.layers.SoftmaxLayer | Output layer for multiclass classification output. |