Lecture Note
slides

Data Augmentation

1. Flip horizontally
2. Random crops/scales
3. Random mix/combinations of: translation, rotation, stretching, shearing, lens distortions, … (go crazy)
4. Color jittering

• Simple: Change contrast small amounts, jitter the color distributions, etc
• Fancy PCA way: 1. Compute PCA on all [R,G,B] points values in the training data; 2. sample some color offset along the principal components at each forward pass; 3. add the offset to all pixels in a training image

Notice the more general theme:

• Introduce a form of randomness in forward pass(Dropout, functional pooling)
• Marginalize over the noise distribution during prediction

GPU

• CPU: disk bottleneck, Harddisk is slow to read from
• GPU: memory bottleneck,e .g. AlexNet: ~3GB needed with batch size 256(Tesla K40: 12GB <- currently the max
  Titan Black: 6GB)
• GPU - CPU communication is a bottleneck: CPU data prefetch thread running while GPU performs forward/backward pass

FFT

Unfortunately, FFT Conv is slower with smaller filter sizes :(