学习视频课件RNN training.pptx

1、Training Recurrent Neural NetworkHung-yi LeeGoalx1x2x3y1y2y3WiWhWoinitWhWhWiWoWiWo0Backpropagation through time(BPTT)Review:BackpropagationlixlijlilijxzwzwCCForward PassBackward Pass11llza 11lTlllWz1211llllbaWz yzxLCL111lxlajlj12j12ilijwLayerlLayer1l111bxWz 11za LTLLLWz11liError signallizReview:Back

2、propagationli12n1Cyx Lz1 Lz2 Lnz2yCxnxyCLayer L21L1z1mLayer L-1TWL yxCL1-L1L2z1LmzBackward Pass 11lTlllWz yzxLCL LTLLLWz11lixlijlilijxzwzwCCError signalanBackpropagation through TimexnynA very deep neural networkoutput:yn Input:init,x1,x2,xnUNFOLD:xn-1an-1xn-2x1a1initan-2 ynCny1nCnnyC2nnyC3anBackpro

3、pagation through TimexnynA very deep neural networkoutput:yn Input:init,x1,x2,xnUNFOLD:xn-1an-1xn-2x1a1initan-2 ynC123 nz1 nz2 nz312311nz12nz13nzanBackpropagation through TimexnynA very deep neural networkoutput:yn Input:init,x1,x2,xnUNFOLD:xn-1an-1xn-2x1a1initan-2 ynCanxnynxn-1an-1xn-2x1a1Some weig

4、hts are shared.initan-2jikikjijikjBackpropagation through TimeA very deep neural networkoutput:yn Input:init,x1,x2,xnUNFOLD:Initialize w1,w2 by the same value(The values of w1,w2 should always be the same.)the same memorypointerpointerBPTT x1x2x3y1y2y3initx4y4Backward Pass:Forward Pass:Compute a1,a2

5、,a3,a4 a1a2a3a4Unfortunately,it is not easy to train RNN.The error surface is rough.w1w2CostSource:http:/jmlr.org/proceedings/papers/v28/pascanu13.pdfThe error surface is either very flat or very steep.If n=1000:Toy Example101wy101wy201wy301wynCost Cn1111n=10n=100n=1000Only extremely large and small

6、 valueanBackpropagation through Timexnynxn-1an-1xn-2x1a1initan-2Gradient Vanishing/Exploding For simplicity,assume linear activation function Gradient Vanishing/Exploding 05101520253035-100-10-1-0.1-0.0100.010.111010005101520253035-100-10-1-0.1-0.0100.010.111010005101520253035-100-10-1-0.1-0.0100.01

7、0.111010005101520253035-100-10-1-0.1-0.0100.010.111010005101520253035-100-10-1-0.1-0.0100.010.111010005101520253035-100-10-1-0.1-0.0100.010.11101001 step2 steps5 steps10 steps20 steps50 stepsPossible SolutionsClipped Gradientw1w2Cost Clipped gradienttheano.tensor.clip(x,min,max)Source:http:/jmlr.org

8、/proceedings/papers/v28/pascanu13.pdfSource:http:/www.cs.toronto.edu/fritz/absps/momentum.pdfGradient descentMomentum Nesterovs Accelerated Gradient(NAG)NAGMethods:ValleyNAG Momentum Nesterovs Accelerated Gradient(NAG)MovementGradientLast MovementGradient=0Gradient=0RMSPropLarger Learning RateSmalle

9、r Learning RateReview:AdagradUse first derivative to estimate second derivativeRMSPropError Surface can be even more complex when training RNN.Larger Learning RateSmaller Learning RateRMSPropRoot Mean Square of the gradients with previous gradients being decayed x1x2+Input4 times of parametersLSTM c

10、an address the gradient vanishing problem.LSTMxtzzizfzoytxt+1zzizfzoyt+1ht+1Extension:“peephole”htLSTMxtzzizfzoytxt+1zzizfzoyt+1ht+1htabab11X WTX WTConstant Error Carrousel(CEC)Other Simpler Variants GRU:Cho,Kyunghyun,et al.Learning Phrase Representations using RNN EncoderDecoder for Statistical Mac

11、hine Translation“,EMNLP,2014 SCRN:Mikolov,Tomas,et al.Learning longer memory in recurrent neural networks“,ICLR 2015Better Initialization Vanilla RNN:Initialized with Identity matrix+ReLU Quoc V.Le,Navdeep Jaitly,Geoffrey E.Hinton,“A Simple Way to Initialize Recurrent Networks of Rectified Linear Units“,2015Concluding Remarks Be careful when training RNN Possible solutions:Clipping the gradients Advanced optimization technology NAG RMSprop Try LSTM(or other simpler variants)Better initialization