Deep Learning Concepts through Podolski's Slides

Delve into the world of deep learning with Podolski's presentation slides covering topics like motivation, neural networks, Andrew Ng's perspectives, neuron types, and the essence of feature representation in deep learning algorithms.

• Deep Learning Concepts
• Neural Networks
• Andrew Ng
• Feature Representation
• Podolskis Slides

mcazi Follow

Uploaded on Sep 18, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. 1 Podolski Deep Learning Part 2 of 3

2. 2 General Outline Motivation Deep Learning Definition(s) Neural Networks Types Deep Learning Biology One Learning Hypothesis Features

3. 3 Motivations Andrew Ng joined AI to help the field progress. He summarized his initial experience as entirely composed of curve-fitting instead of doing AI. Give-and-take perspective on machine learning and biology The problem of features

4. 4 General Outline Motivation Deep Learning Definition(s) Neural Networks Types Deep Learning Biology One Learning Hypothesis Features

5. 5 Deep Learning Definition(s) Andrew Ng: Enhance learning algorithm to make them better and easier; attempt to make revolutionary advances Geoff Hinton: NN with more than one hidden layer Wikipedia: rebranding of NN

6. 6 General Outline Motivation Deep Learning Definition(s) Neural Networks Types Deep Learning Biology One Learning Hypothesis Features

7. 7 The Neurons Binary Threshold Linear Rectified Linear ? = ? + ???? ? = ? + ???? ? = ? + ???? ? ? ? ? = 1 ?? ? > ? 0 ?? ?????? ? = ? ?? ? > ? 0 ?? ?????? ? = ? For purposes of Deep Learning, we will be most interested in the Rectified Linear neuron.

8. 8 The Neurons Sigmoid Stochastic Binary Stochastic Rectified ? = ? + ???? Output is rate of producing spikes ? = ? + ???? ? ? 1 ? ? = 1 = 1 1 + ? ? ? = 1 + ? ? Types of Neural Networks Feed Forward Recurrent Symmetric

9. 9 General Outline Motivation Deep Learning Definition(s) Neural Networks Types Deep Learning Biology One Learning Hypothesis Features

10. 10 Deep Learning Input -> Feature Representation -> Learning Algorithm See Payam s Slides

11. 11 General Outline Motivation Deep Learning Definition(s) Neural Networks Types Deep Learning Biology One Learning Hypothesis Features

12. 12 Biology [Roe et al., 1992] Ferrets had nerve connections cut

13. 13 One-Learning Theory Human echolocation Tongue -duino Tongue possesses a very high nerve desnity Various sensors will output voltages Voltages stimulate tongue Due to high nerve density, brain quickly adapts and learns to recognize patterns Strong repercussions against modular perspective of brain

14. 14 General Outline Motivation Deep Learning Definition(s) Neural Networks Types Deep Learning Biology One Learning Hypothesis Features

15. 15 Features Geoff Hinton: Hand-engineering features Optimized discrimination Learning feature one layer at a time Problems of Features Examples in CV Very complicated, we have to hand engineer features "I've read the SIFT paper about 5 times now and still have no idea what it is doing Andrew Ng Coming up with features is difficult and time-consuming Typically requires expert knowledge We spend a lot of time tuning the features

16. 16 Features cont. Human vision (V1 system) detects edges first V2 handles combinations of edges V3? Biology influencing algorithms Mimicking this system, we can learn features hierarchically

17. 17 Finding Features Sparse Coding vs K-Means (based on [Coates, Ng 2012] Fairly interchangeable Differ in optimization objectives but both produce code vectors (basis vectors) ?? Empirically, sparse coding appears to be a better performer Replacing k-means with sparse coding in bag-of-features yielded significant improvements However, K-Means is a little more efficient because sparse coding needs to do a convex optimization for ?? repeatedly in the learning algorithm

18. 18 Sparse Coding Originally developed for use in neural encoding/decoding Now adapted to Deep Learning Unsupervised learning method that finds set of basis vector with which to later order input vectors. Huge success with CV, audio In audio, the basis vectors almost resemble phonemes Inputs are linear combination of basis vectors

19. 19 Hierarchical Feature Learning Sparse Coding V2 1 2 V1

20. 20 Overview of Implementation How to learn layer of features without requiring labels Learning a generative model of input with one layer of latent variables Use vectors of latent variable activities as data for training second layer This is done with a Restricted Boltzman Machine Combine stack of models into a single multilayer generative model

21. 21 Overview of Implementation cont. After training whole stack of generative models, each of which is one hidden layer, compose them all into one generative model with multiple hidden layers Each layer, as with V1 and V2, is a combination of the prior layers basis vectors; we get a higher level of abstraction each time Each time we add another layer we get a new and better lower bound on log of probability of training data

22. 22 RBM DBN-DNN (BP) Deep Belief Net W3 W4 = 0 W3 RBM Copy W2 W3 RBM W2 Copy W1 W2 W1 W1 D D D Rectified Linear Neurons work best.

23. 23 Random Forest NN Used to be feasible; with larger data sets and larger NN to train, it takes too long Utilization of Dropout Each time we present training data, randomly omit hidden units with a probability of (0.5) Random sampling from 2^H hidden units

24. 24 Conclusion RBM Convolutional Nets and replicated feature approach Dropout algorithm in-depth