[Review] Understanding the difficulty of training deep feedforward neural networks

Info

Author:
Xavier Glorot
Yoshua Bengio

Year:
2010

Agenda

Abstract

1. Deep Neural Networks
2. Experimental Setting and Datasets
   1. Online Learning on an Infinite Dataset: Shapeset-3x2
   2. Finite Datasets
   3. Experimental Setting
3. Effect of Activation Functions and Saturation During Training
   1. Experiments with the Sigmoid
   2. Experiments with the Hyperbolic Tangent
   3. Experiments with the Softsign
4. Studying Gradients and their Propagations
   1. Effect of the Cost Function
   2. Gradients at Initialisation
      1. Theoretical Considerations and a New Normalised Initialisation
      2. Gradient Propagation Study
   3. Back-propagation Gradients During
5. Error Curves and Conclusions

Evernote!!
https://www.evernote.com/l/ATRQQEYUnXtBTreAuqXoqRWKJjkZn7mNBV0

Nice Material