Lecture Note
References:

• CNN in Matlab
• DeepLearning.net tutorial walks through an implementation of a ConvNet in Theano
• cuda-convnet2 by Alex Krizhevsky is a ConvNet implementation that supports multiple GPUs
• ConvNetJS CIFAR-10 demo allows you to play with ConvNet architectures and see the results and computations in real time, in the browser.
• Caffe, one of the most popular ConvNet libraries.
• Example Torch 7 ConvNet that achieves 7% error on CIFAR-10 with a single model
• Ben Graham’s Sparse ConvNet package, which Ben Graham used to great success to achieve less than 4% error on CIFAR-10.

Read More

Lecture note

Read More

Lecture Note
Hinton Note on the same topic
Reference
Read

• SGD tips and tricks from Leon Bottou
• Efficient BackProp (pdf) from Yann LeCun
• Practical Recommendations for Gradient-Based Training of Deep Architectures from Yoshua Bengio

About Nesterov’s Accelerated Momentum (NAG)
Advances in optimizing Recurrent Networks
Ilya Sutskever’s thesis

L-BFGS VS SGD

• On Optimization Methods for Deep Learning from Le et al. is a paper from 2011 comparing SGD vs. L-BFGS. Some of its conclusions have since been challenged.
• Large Scale Distributed Deep Networks is a paper from the Google Brain team, comparing L-BFGS and SGD variants in large-scale distributed optimization.
• SFO algorithm strives to combine the advantages of SGD with advantages of L-BFGS.

Read More

Data Preprocessing, Weight Initialization, Regularization (L2/L1/Maxnorm/Dropout), Loss functions
Lecture Notes
Some references:
Should read:
Elastic net regularization
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Dropout Training as Adaptive Regularization
DropConnect

others:
Understanding the difficulty of training deep feedforward neural networks
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Hierarchical Softmax

Read More

Lecture Notes
Useful Refenrences

• Deep Learning book in press by Bengio
• Do Deep Nets Really Need to be Deep?
• FitNets: Hints for Thin Deep Nets
• The Loss Surfaces of Multilayer Networks
• deeplearning.net tutorial with Theano
• ConvNetJS demos for intuitions
• Michael Nielsen’s

Read More

KNN, SVM, Softmax

Read More

import numpy as np
import scipy as scp

• np.flatnonzero # return indices that are non-zero in the flattened version of a.
  This is equivalent to a.ravel().nonzero()[0].

  see also
  nonzero
  Return the indices of the non-zero elements of the input array.
  ravel
  Return a 1-D array containing the elements of the input array.

• **numpy.random.choice(a, size=None, replace=True, p=None) #Generates a random sample from a given 1-D array

  a 1-D array-like or int
  If an ndarray, a random sample is generated from its elements. If an int, the random sample is generated as if a was np.arange(n)

• ** numpy.argsort(a, axis=-1, kind=’quicksort’, order=None)[source]
  Returns the indices that would sort an array.

• np.random.permutation
• np.array_split
• np.vstack

• Start Ipython notebook
  Ipython notebook.

• Use markdown cell, code cell, raw text cell for editing

• short cuts:
  Shift-Enter: run cell
  Ctrl-Enter: run cell in-place
  Alt-Enter: run cell, insert below

  Read More

Lecture Note
This lecture is very inspiring, actually it clears a haunting questions about the backpropogation.
I think this lecture is nice and concise. If you forget something, go back directly to the orinal lecture. Here just adding some images to remember.

Read More

lecture note
Not too much in this lecture if you know the basic knowledge about optimization. From my perspective, the most important thing is about(Gradient check).

Read More