x-j)&ddlZddZ d dZdS) Nct||\}}n||kr||fn||f\}}||zd||z z||z z z } | dz||zz } | dkrs| } ||kr|||z || z| z ||z d| zzz zz } n|||z || z| z ||z d| zzz zz } tt| ||S||zdz S)a]Cubic interpolation between (x1, f1, g1) and (x2, f2, g2). Use two points and their gradient to determine a cubic function and get the minimum point between them in the cubic curve. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp59: formula 3.59 Args: x1, f1, g1: point1's position, value and gradient. x2, f2, g2: point2's position, value and gradient. bounds: bounds of interpolation area Returns: min_pos: the minimum point between the specified points in the cubic curve. Nrg@)sqrtminmax) x1f1g1x2f2g2bounds xmin_bound xmax_boundd1 d2_squared2min_poss m/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/paddle/incubate/optimizer/line_search_dygraph.py_cubic_interpolaters $!' JJ-/2XX"bB8 J b1R=BG, ,BARIA~~ ^^   88BGb2"r'AF:J(KLLGGBGb2"r'AF:J(KLLG3w ++Z888Z'3..-C6??& .>c |} |}||||\} } d}tj| |}tjd|j|||f\}}}}d}d}|| kr | |||z|zzks |dkr)| |kr#||g}|| g}|| g}||g}ntj|| |zkr |g}| g}| g}d}n|dkr#||g}|| g}|| g}||g}nz|d||z zz}|dz}|}t||||| |||f}|}| }| }|}||||\} } |dz }| |}|dz }|| k || kr d|g}|| g}|| g}d}|d|d krd nd \}}|s||| krutj|d|dz | z| krnIt|d|d|d|d|d|d}d t|t|z z} tt||z |t|z | kr|s&|t|ks|t|krrtj|t|z tj|t|z krt|| z }nt|| z}d}nd}nd}||||\} } |dz }| |}|dz }| |||z|zzks | ||kr@|||<| ||<| ||<|||<|d|dkrd nd \}}ntj|| |zkrd}nD|||||z zdkr,||||<||||<||||<||||<|||<| ||<| ||<|||<|s|| ku||}||} ||} | | ||fS) a4 Implements of line search algorithm that satisfies the strong Wolfe conditions using double zoom. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp60: Algorithm 3.5 (Line Search Algorithm). Args: obj_func: the objective function to minimize. ```` accepts a multivariate input and returns a scalar. xk (Tensor): the starting point of the iterates. alpha (Scalar): the initial step size. d (Tensor): search direction. loss (scalar): the initial loss grad (Tensor): the initial grad c1 (Scalar): parameter for sufficient decrease condition. c2 (Scalar): parameter for curvature condition. tolerance_change (Scalar): terminates if the change of function value/position/parameter between two iterations is smaller than this value. max_ls(int): max iteration of line search. alpha_max (float): max step length. Returns: loss_new (Scaler): loss of obj_func at final alpha. grad_new, (Tensor): derivative of obj_func at final alpha. alpha(Tensor): optimal step length, or 0. if the line search algorithm did not converge. ls_func_evals (Scaler): number of objective function called in line search process. Following summarizes the essentials of the strong Wolfe line search algorithm. Some notations used in the description: - `func` denotes the objective function. - `obi_func` is a function of step size alpha, restricting `obj_func` on a line. obi_func = func(xk + alpha * d), where xk is the position of k'th iterate, d is the line search direction(decent direction), and a is the step size. - alpha : substitute of alpha - a1 is alpha of last iteration, which is alpha_(i-1). - a2 is alpha of current iteration, which is alpha_i. - a_lo is alpha in left position when calls zoom, which is alpha_low. - a_hi is alpha in right position when calls zoom, which is alpha_high. Line Search Algorithm: repeat Compute obi_func(a2) and derphi(a2). 1. If obi_func(a2) > obi_func(0) + c_1 * a2 * obi_func'(0) or [obi_func(a2) >= obi_func(a1) and i > 1], alpha= zoom(a1, a2) and stop; 2. If |obi_func'(a2)| <= -c_2 * obi_func'(0), alpha= a2 and stop; 3. If obi_func'(a2) >= 0, alpha= zoom(a2, a1) and stop; a1 = a2 a2 = min(2 * a2, a2) i = i + 1 end(repeat) zoom(a_lo, a_hi) Algorithm: repeat aj = cubic_interpolation(a_lo, a_hi) Compute obi_func(aj) and derphi(aj). 1. If obi_func(aj) > obi_func(0) + c_1 * aj * obi_func'(0) or obi_func(aj) >= obi_func(a_lo), then a_hi <- aj; 2. 2.1. If |obi_func'(aj)| <= -c_2 * obi_func'(0), then alpha= a2 and stop; 2.2. If obi_func'(aj) * (a2 - a1) >= 0, then a_hi = a_lo a_lo = aj; end(repeat) r)dtypeFTg{Gz? )r)rr)rrg?) absrclonepaddledot to_tensorrrr)!obj_funcxkalphadlossgradgtdc1c2tolerance_changemax_lsd_normloss_newgrad_new ls_func_evalsgtd_newt_prevf_prevg_prevgtd_prevdonels_iterbracket bracket_f bracket_g bracket_gtdmin_stepmax_steptmpinsuf_progresslow_poshigh_posepss! r _strong_wolferH6slUUWW[[]]F ::<222:"      h'   !!%Xb%33( ,,q//1 a F  f&e*8$ 8$ N"+A,)B-"?"?VGXI+w'' :gaj71:- . . 7:J J J # AJ aL N AJ aL N   "S\\CLL01 s7||e#US\\%9 : :S @ @ &#g,,!6!6%3w<<:O:O:ec'll233fjCLL(77 LL3.EELL3.E!&!%"N%Xb%33( ,,q//1  rEzC// 0 09W---!&GH "*Ih "*.."2"2Ih $+K !#A,)A,66F GXXz'""rcCi//GH-0@@AQFF$+G$4!&/&8 (#&/&8 (#(3G(< H% %GG !)Ig !)!1!1Ig #*K SI+w''X G E!H!H Xum 33r)N)rrrr)r$rrHrrrJsW !/!/!/!/X   s4s4s4s4s4s4r