人工智能-深度强化学习Tips-for-Training-Deep-Network课件.pptx

1、Tips for Training Deep NetworkOutput Training Strategy:Batch Normalization Activation Function:SELU Network Structure:Highway NetworkBatch NormalizationFeature ScalingThe means of all dimensions are 0,and the variances are all 1 For each dimension i:In general,gradient descent converges much faster

2、with feature scaling than without it.How about Hidden Layer?Layer 1Feature ScalingFeature Scaling?Feature Scaling?Smaller learning rate can be helpful,but the training would be slower.Difficulty:their statistics change during the training Batch normalizationInternal Covariate Shift BatchSigmoid=Sigm

3、oidSigmoidBatchBatch normalizationNote:Batch normalization cannot be applied on small batch.Batch normalizationSigmoidSigmoidSigmoidHow to do backpropogation?Batch normalizationBatch normalization At testing stage:We do not have batch at testing stage.Ideal solution:Practical solution:AccUpdatesBatc

4、h normalization-Benefit BN reduces training times,and make very deep net trainable.Because of less Covariate Shift,we can use larger learning rates.Less exploding/vanishing gradients Especially effective for sigmoid,tanh,etc.Learning is less affected by initialization.BN reduces the demand for regul

5、arization.To learn more Batch Renormalization Layer Normalization Instance Normalization Weight Normalization Spectrum NormalizationActivation Function:SELUReLU Rectified Linear Unit(ReLU)Reason:1.Fast to compute2.Biological reason3.Infinite sigmoid with different biases4.Vanishing gradient problemR

6、eLU-variant also learned by gradient descentReLU-variantExponential Linear Unit(ELU)Scaled ELU(SELU)SELUPositive and negative valuesThe whole ReLU family has this property except the original ReLU.Saturation regionELU also has this property Slope larger than 1Only SELU also has this property SELUKKk

7、kwawawaz11z1wkwKw1akaKa zfa=0=1=0Do not have to be Gaussian=0SELUKKkkwawawaz11z1wkwKw1akaKa zfa=0=1=1=1targetAssume GaussianDemo93 頁的證明SELU is actually more general.最新激活神經元最新激活神經元:SELF-NORMALIZATION NEURAL NETWORK(SELU)MNISTCIFAR-10DemoHighway Network&Grid LSTM xf1a1f2a2f3a3f4yx1h0fh1x2fx3h2fx4h3fy4

8、t is layert is time stepApplying gated structure in feedforward networkFeedforward v.s.Recurrent1.Feedforward network does not have input at each step2.Feedforward network has different parameters for each layerht-1rzytxtht-1hxt1-htresetupdateNo input xt at each stepat-1 is the output of the(t-1)-th

9、 layerat is the output of the t-th layerNo output yt at each stepNo reset gateat-1atat-1Highway Network Residual NetworkResidual Network Highway NetworkHighway NetworkDeep Residual Learning for Deep Residual Learning for Image RecognitionImage Recognitionhttp:/arxiv.org/abs/1512.03http:/arxiv.org/ab

10、s/1512.03385385Training Very Deep Training Very Deep NetworksNetworkshttps:/arxiv.org/pdf/150https:/arxiv.org/pdf/1507.06228v2.pdf7.06228v2.pdf+copycopyGate controllerInput layeroutput layerInput layeroutput layerInput layeroutput layerHighway Network automatically determines the layers needed!Highw

11、ay NetworkGrid LSTMLSTMyxchthcGridLSTMchhcMemory for both time and depthbabatimedepthGridLSTMht-1ct-1alblhtctal-1bl-1GridLSTMalblht+1ct+1al-1bl-1GridLSTMht-1ct-1al+1bl+1htctGridLSTMal+1bl+1ht+1ct+1Grid LSTMGridLSTMchhcbabahzzizfzohctanhcababef3D Grid LSTMhchcbaefba3D Grid LSTM Images are composed of pixels3 x 3 images