Essential Tips for Training Neural Networks from Scratch

Slide Note

Neural network training involves key considerations like optimization for finding optimal parameters and generalization for testing data. Initialization, learning rate selection, and gradient descent techniques play crucial roles in achieving efficient training. Understanding the nuances of stochastic gradient descent and batch gradient descent is essential for successful training of neural networks.

owynnta Follow

Uploaded on Nov 27, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript

Tips for Training Neural Network scratch the surface

Two Concerns There are two things you have to concern. Optimization Can I find the best parameter set * in limited of time? Generalization Is the best parameter set * good for testing data as well?

Initialization For gradient descent, we need to pick an initialization parameter 0. Do not set all the parameters 0equal Set the parameters in 0 randomly

Learning Rate ( ) = C 1 1 i i i Set the learning rate carefully Toy Example y z w = 1 w + x = * = 0 b y = z b 1 Training Data (20 examples) x = [0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 5.5, 6.0, 6.5, 7.0, 7.5, 8.0, 8.5, 9.0, 9.5] y = [0.1, 0.4, 0.9, 1.6, 2.2, 2.5, 2.8, 3.5, 3.9, 4.7, 5.1, 5.3, 6.3, 6.5, 6.7, 7.5, 8.1, 8.5, 8.9, 9.5]

Learning Rate ( ) = C 1 1 i i i Toy Example Error Surface: C(w,b) start target

= 1 . 0 Learning Rate Toy Example Different learning rate = . 0 01 = . 0 001 ~ k 3 updates ~ 0. 3 updates k

( ) x 1 r r = r r y C f R 1 2 Gradient descent = r C R Gradient descent ( ) ( ) ( ) 1 r = 1 1 i r i = C 1 1 i i i C C R Stochastic Gradient descent ( ) = C 1 1 i i r i Pick an example xr If all example xrhave equal probabilities to be picked ( ) ( ) 1 r = 1 1 r i r i E C C R

Gradient descent Stochastic Gradient descent Training Data: ( )( , ) ( ) ( ) 1 1 2 2 r r R R , , , , , , x y x y x y x y Starting at 0 pick x1 ( ) ( ) ( C = = 1 0 1 0 C C 2 1 2 1 pick x2 Seen all the examples once ) = 1 1 r r r r pick xr One epoch ( ( ) ) pick xR = R R 1 R R 1 C + = R 1 R 1 R C pick x1

Gradient descent See only one example See all examples Toy Example Update 20 times in an epoch Gradient descent Stochastic Gradient descent 1 epoch

Gradient descent Gradient descent = ( ) ( ) ( ) 1 r = 1 1 i r i C 1 1 i i i C C R Stochastic Gradient descent Pick an example xr Mini Batch Gradient Descent Pick B examples as a batch b (B is batch size) ( ) = C 1 1 i i r i ( ) 1 x r = 1 1 i i r i C B b Average the gradient of the examples in the batch b Shuffle your data

Gradient descent Real Example: Handwriting Digit Classification Batch size = 1 Gradient descent

Two Concerns There are two things you have to concern. Optimization Can I find the best parameter set * in limited of time? Generalization Is the best parameter set * good for testing data as well?

Generalization You pick a best parameter set * Training Data: ( ) ( ) ry , r x *= r r y : ; r f x However, ( ) ux * u u y ; f x Testing Data: Testing Data: Training Data: Training data and testing data have different distribution.

Panacea Have more training data if possible Create more training data (?) Handwriting recognition: Original Training Data: Created Training Data: Shift 15

Reference Chapter 3 of Neural network and Deep Learning http://neuralnetworksanddeeplearning.com/ch ap3.html

Appendix

Overfitting The function that performs well on the training data does not necessarily perform well on the testing data. ( x , Training Data: ( ) x ~ ) = r r y : r f ry r ( ) x ~ ux u u y f Testing Data: Overfitting in our daily life: Memorize the answers of the previous examples

Joke for overfiting http://xkcd.com/1122/

Initialization For gradient descent, we need to pick an initialization parameter 0. Do not set all the parameters 0equal Or your parameters will always be equal, no matter how many times you update the parameters Randomly pick 0 If the last layer has more neurons, the initialization values should be smaller. E.g. Last layer has Nl-1 ( 1 / 1 , 0 ~ l ij N N w ) / 1 l l ij ~ U 1 / 1 , w N N 1 l l

MNIST The MNIST data comes in two parts. The first part contains 60,000 images to be used as training data. These images are scanned handwriting samples from 250 people, half of whom were US Census Bureau employees, and half of whom were high school students. The images are greyscale and 28 by 28 pixels in size. The second part of the MNIST data set is 10,000 images to be used as test data. Again, these are 28 by 28 greyscale images. git clone https://github.com/mnielsen/neural-networks- and-deep-learning.git http://yann.lecun.com/exdb/mnist/ http://www.deeplearning.net/tutorial/gettingstarted.html

MNIST The current (2013) record is classifying 9,979 of 10,000 images correctly. This was done by Li Wan, Matthew Zeiler, Sixin Zhang, Yann LeCun, and Rob Fergus. At that level the performance is close to human- equivalent, and is arguably better, since quite a few of the MNIST images are difficult even for humans to recognize with confidence.