GradientDescent梯度下降法课件.ppt

1、vGradientvDirectional DerivativesvGradient descent(GD):AKA steepest descent(SD)Goal:Minimize a function iteratively based on gradientFormula for GD:Normalized versionWith momentumGradient Descent(GD)Step size or learning rateQuiz!Vanilla GDorExample of Single-Input FunctionsvIf n=1,GD reduces to the

2、 problem of going left or right.vExamplevAnimation:http:/ nownownownextxxxxxxfxxf21222vEach point/region with zero gradient has a basin of attractionBasin of Attraction in 1DvExample of Two-input Functions“Peaks”Functions(1/2)If n=2,GD needs to find a direction in 2D plane.Example:“Peaks”function in

3、 MATLABAnimation:gradientDescentDemo.m222222153123151013,yxyxyxeeyxxexyxfGradients is perpendicularto contours,why?3 local maxima3 local minima“Peaks”Functions(2/2)Gradient of the“peaks”functiondz/dx=-6*(1-x)*exp(-x2-(y+1)2)-6*(1-x)2*x*exp(-x2-(y+1)2)-10*(1/5-3*x2)*exp(-x2-y2)+20*(1/5*x-x3-y5)*x*exp

4、(-x2-y2)-1/3*(-2*x-2)*exp(-(x+1)2-y2)dz/dy=3*(1-x)2*(-2*y-2)*exp(-x2-(y+1)2)+50*y4*exp(-x2-y2)+20*(1/5*x-x3-y5)*y*exp(-x2-y2)+2/3*y*exp(-(x+1)2-y2)d(dz/dx)/dx=36*x*exp(-x2-(y+1)2)-18*x2*exp(-x2-(y+1)2)-24*x3*exp(-x2-(y+1)2)+12*x4*exp(-x2-(y+1)2)+72*x*exp(-x2-y2)-148*x3*exp(-x2-y2)-20*y5*exp(-x2-y2)+

5、40*x5*exp(-x2-y2)+40*x2*exp(-x2-y2)*y5-2/3*exp(-(x+1)2-y2)-4/3*exp(-(x+1)2-y2)*x2-8/3*exp(-(x+1)2-y2)*x vEach point/region with zero gradient has a basin of attractionBasin of Attraction in 2DRosenbrock FunctionRosenbrock functionMore about this functionAnimation:http:/ on how to optimize this funct

6、ion 2221001,xyxyxfJustification for using momentum termsProperties of Gradient DescentvPropertiesNo guarantee for reaching global optimumFeasible for differentiable objective functions(which can have finite number of non-differential points)Performance heavily dependent on starting point and step si

7、ze vVariantsUse adaptive step sizesNormalize the gradient by its lengthUse the momentum term to reduce zig-zag pathsUse line minimization at each iterationQuiz!Comparisons of Gradient-based OptimizationvGradient descent(GD)Treat all parameters as nonlinearvHybrid learning of GD+LSEDistinguish betwee

8、n linear and nonlinearvConjugate gradient descentTry to reach the minimizing point by assume the objective function is quadraticvGauss-Newton(GN)methodLinearize the objective function to treat all parameters as linearvLevenberg-Marquardt(LM)methodSwitch smoothly between SD and GNvSynonymsLinearizati

9、on methodExtended Kalman filter methodvConcept:General nonlinear model:y=f(x,q)linearization at q=qnow:y=f(x,qnow)+a1(q1-q1,now)+a2(q2-q2,now)+.LSE solution:qnext=qnow+(ATA)-1ATBGauss-Newton MethodvFormulaqnext=qnow+(ATA+lI)-1ATBvEffects of ll small Gauss-Newton methodl big Gradient descentvHow to u

10、pdate lGreedy policy Make l smallCautious policy Make l bigLevenberg-Marquardt MethodvCan we use GD to find the minimum of f(x)=|x|?vWhat is the gradient of the sigmoid function?Can you express the gradient using the original function?vWhat are the basins of attraction of the following curve?Exercises xexf11f(x)xglobal minimuminflection pointlocal minimumglobal maximumQuiz:The gradient of a function of eax can be expressed as the function itself.Why?