ёi'y0SSKJr SSKrSSKrSSKJr SSKJr SSKJ r J r J r SSK J r SSKrSSKJr S S KJr \ (aSS KJr SS KJr SS KJr SSKJr S SKJr /r"SS\ 5r"SS\ 5rSrSrSSjr SSjr!"SS\5r"g)) annotationsN) defaultdict)reduce) TYPE_CHECKINGNoReturn TypedDict) NotRequired) framework) Optimizer)Sequence)Tensor)GradientClipBase)WeightDecayRegularizer)_ParameterConfigc\rSrSr%S\S'S\S'S\S'S\S'S\S 'S\S 'S\S 'S\S 'S\S 'S\S'S\S'Srg) _LbfgsState*int func_evalsn_iterrdalphaz list[Tensor]old_ykold_skroH_diagprev_flat_gradfloat prev_losszNotRequired[list[Tensor]]alN__name__ __module__ __qualname____firstlineno____annotations____static_attributes__r#V/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/optimizer/lbfgs.pyrr*s?O K I M   N!!r+rc \rSrSr%S\S'Srg)_LbfgsStateDict8rstater#Nr$r#r+r,r.r.8s r+r.c[RR5(a2[R"S5S:wa[ R "S5 ggg)z-Check and warn about TF32 acceleration statusNVIDIA_TF32_OVERRIDE0zWarning! TF32 Tensor Cores are enabled by default on some NVIDIA GPUs for faster computation, but may compromise numerical precision in specific cases, particularly with the L-BFGS optimizer.To disable it, set: NVIDIA_TF32_OVERRIDE=0N)paddledeviceis_compiled_with_cudaosgetenvwarningswarnr#r+r,check_tf32_overrider;<sD  ++-- II, - 4  9  5 .r+c$X-RSS9$)z NOTE: This is a temporary workaround for unstable result computed by `paddle.dot`, which will be reverted when the problem is fixed." axis)sumxys r,dotrDIs E;;B; r+c:UbUupxO X::aX4OX04upxX%-SX- -X- - - n U S-X%-- n U S:a_U R5n X::aX3U- X[-U - XR- SU --- -- n OXU- X+-U - X%- SU --- -- n [[X5U5$Xx-S- $)a-Cubic interpolation between (x1, f1, g1) and (x2, f2, g2). Use two points and their gradient to determine a cubic function and get the minimum point between them in the cubic curve. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp59: formula 3.59 Args: x1, f1, g1: point1's position, value and gradient. x2, f2, g2: point2's position, value and gradient. bounds: bounds of interpolation area Returns: min_pos: the minimum point between the specified points in the cubic curve. r rg@)sqrtminmax) x1f1g1x2f2g2bounds xmin_bound xmax_boundd1 d2_squared2min_poss r,_cubic_interpolaterWQs$!' J-/X"B8 1=BG, ,BAIA~ ^^  8G2"'AF:J(KLLGG2"'AF:J(KLLG3w+Z88'3..r+c UR5R5n UR5nU"XU5upSn[X5nSXEU4unnnnSnSnUU :aXXr-U--:d US:a%U U:aUU/nUU /nUU R5/nUU/nO[U5U*U-::a U/nU /nU /nSnOUS:aUU/nUU /nUU R5/nUU/nOaUSUU- --nUS-nUn[ UUUUU UUU4S9nUnU nU R5nUnU"XU5upUS- n[X5nUS- nUU :aMUU :Xa SU/nXL/nX]/nSnWSUS::aS OS unnU(GdUU :Ga[WSUS- 5U -U :aGO[ USUSWSUSUSUS5nS [U5[ U5- -n [ [U5U- U[ U5- 5U :axU(dU[U5:dU[ U5::aP[U[U5- 5[U[ U5- 5:a[U5U - nO[ U5U -nSnOSnOSnU"XU5upUS- n[X5nUS- nXXr-U--:d U UU:a6UUU'U UU'U R5WU'UUU'USUS::aS OS unnOj[U5U*U-::aSnO2UUUUU- -S:a UUUU'UUUU'WUUU'UUUU'UUU'U UU'U R5WU'UUU'U(d UU :aGMWUnUUn WUn XX.4$) a{ Implements of line search algorithm that satisfies the strong Wolfe conditions using double zoom. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp60: Algorithm 3.5 (Line Search Algorithm). Args: obj_func: the objective function to minimize. ```` accepts a multivariate input and returns a scalar. xk (Tensor): the starting point of the iterates. alpha (Scalar): the initial step size. d (Tensor): search direction. loss (scalar): the initial loss grad (Tensor): the initial grad c1 (Scalar): parameter for sufficient decrease condition. c2 (Scalar): parameter for curvature condition. tolerance_change (Scalar): terminates if the change of function value/position/parameter between two iterations is smaller than this value. max_ls(int): max iteration of line search. alpha_max (float): max step length. Returns: loss_new (Scaler): loss of obj_func at final alpha. grad_new, (Tensor): derivative of obj_func at final alpha. alpha(Tensor): optimal step length, or 0. if the line search algorithm did not converge. ls_func_evals (Scaler): number of objective function called in line search process. Following summarizes the essentials of the strong Wolfe line search algorithm. Some notations used in the description: - `func` denotes the objective function. - `obi_func` is a function of step size alpha, restricting `obj_func` on a line. obi_func = func(xk + alpha * d), where xk is the position of k'th iterate, d is the line search direction(decent direction), and a is the step size. - alpha : substitute of alpha - a1 is alpha of last iteration, which is alpha_(i-1). - a2 is alpha of current iteration, which is alpha_i. - a_lo is alpha in left position when calls zoom, which is alpha_low. - a_hi is alpha in right position when calls zoom, which is alpha_high. Line Search Algorithm: repeat Compute obi_func(a2) and derphi(a2). 1. If obi_func(a2) > obi_func(0) + c_1 * a2 * obi_func'(0) or [obi_func(a2) >= obi_func(a1) and i > 1], alpha= zoom(a1, a2) and stop; 2. If |obi_func'(a2)| <= -c_2 * obi_func'(0), alpha= a2 and stop; 3. If obi_func'(a2) >= 0, alpha= zoom(a2, a1) and stop; a1 = a2 a2 = min(2 * a2, a2) i = i + 1 end(repeat) zoom(a_lo, a_hi) Algorithm: repeat aj = cubic_interpolation(a_lo, a_hi) Compute obi_func(aj) and derphi(aj). 1. If obi_func(aj) > obi_func(0) + c_1 * aj * obi_func'(0) or obi_func(aj) >= obi_func(a_lo), then a_hi <- aj; 2. 2.1. If |obi_func'(aj)| <= -c_2 * obi_func'(0), then alpha= a2 and stop; 2.2. If obi_func'(aj) * (a2 - a1) >= 0, then a_hi = a_lo a_lo = aj; end(repeat) reference: https://github.com/pytorch/pytorch r rFTg{Gz? )rPr=)rr )r rg?)absrIclonerDrWrH)!obj_funcxkrrlossgradgtdc1c2tolerance_changemax_lsd_normloss_newgrad_new ls_func_evalsgtd_newt_prevf_prevg_prevgtd_prevdonels_iterbracket bracket_f bracket_g bracket_gtdmin_stepmax_steptmpinsuf_progresslow_poshigh_posepss! r, _strong_wolfer{us|pUUW[[]F ::222:"      h' !%b3 h"1 a F f&e*$ $ N"+A,)B-"?VGXw' wqzGAJ& '& 03C C # AJ aL N AJ aL N  "S\CL01 s7|e#US\%9 :S @#g,!6%3w<:Ous7|+,s53w<3G/HHL3.EL3.E!&!%"N%b3 h"1  rzC// 09W--!&GH "*Ih "*.."2Ih $+K !#A,)A,6F GX7|sSy(GH-0@@AQF$+G$4!&/&8 (#&/&8 (#(3G(< H% %GG !)Ig !)!1Ig #*K Mw'R G E!H!H u 33r+c^\rSrSrSrSSU4SjjjrSSjrSSjrSrSr Sr S r S r \ RSS j5rSSS jjrS rU=r$)LBFGSiea The L-BFGS is a quasi-Newton method for solving an unconstrained optimization problem over a differentiable function. Closely related is the Newton method for minimization. Consider the iterate update formula: .. math:: x_{k+1} = x_{k} + H_k \nabla{f_k} If :math:`H_k` is the inverse Hessian of :math:`f` at :math:`x_k`, then it's the Newton method. If :math:`H_k` is symmetric and positive definite, used as an approximation of the inverse Hessian, then it's a quasi-Newton. In practice, the approximated Hessians are obtained by only using the gradients, over either whole or part of the search history, the former is BFGS, the latter is L-BFGS. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp179: Algorithm 7.5 (L-BFGS). Args: learning_rate (float, optional): learning rate .The default value is 1. max_iter (int, optional): maximal number of iterations per optimization step. The default value is 20. max_eval (int|None, optional): maximal number of function evaluations per optimization step. The default value is max_iter * 1.25. tolerance_grad (float, optional): termination tolerance on first order optimality The default value is 1e-5. tolerance_change (float, optional): termination tolerance on function value/parameter changes. The default value is 1e-9. history_size (int, optional): update history size. The default value is 100. line_search_fn (string|None, optional): either 'strong_wolfe' or None. The default value is strong_wolfe. parameters (list|tuple|None, optional): List/Tuple of ``Tensor`` names to update to minimize ``loss``. \ This parameter is required in dygraph mode. The default value is None. weight_decay (int|float|WeightDecayRegularizer|None, optional): The strategy of regularization. \ It can be a int or float value as coeff of L2 regularization or \ :ref:`api_paddle_regularizer_L1Decay`, :ref:`api_paddle_regularizer_L2Decay`. If a parameter has set regularizer using :ref:`api_paddle_ParamAttr` already, \ the regularization setting here in optimizer will be ignored for this parameter. \ Otherwise, the regularization setting here in optimizer will take effect. \ Default None, meaning there is no regularization. grad_clip (GradientClipBase|None, optional): Gradient clipping strategy, it's an instance of \ some derived class of ``GradientClipBase`` . There are three clipping strategies \ ( :ref:`api_paddle_nn_ClipGradByGlobalNorm` , :ref:`api_paddle_nn_ClipGradByNorm` , \ :ref:`api_paddle_nn_ClipGradByValue` ). Default None, meaning there is no gradient clipping. name (str|None, optional): Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. The default value is None. Return: loss (Tensor): the final loss of closure. Examples: .. code-block:: python >>> import paddle >>> import numpy as np >>> paddle.disable_static() >>> np.random.seed(0) >>> np_w = np.random.rand(1).astype(np.float32) >>> np_x = np.random.rand(1).astype(np.float32) >>> inputs = [np.random.rand(1).astype(np.float32) for i in range(10)] >>> # y = 2x >>> targets = [2 * x for x in inputs] >>> class Net(paddle.nn.Layer): ... def __init__(self): ... super().__init__() ... w = paddle.to_tensor(np_w) ... self.w = paddle.create_parameter(shape=w.shape, dtype=w.dtype, default_initializer=paddle.nn.initializer.Assign(w)) ... ... def forward(self, x): ... return self.w * x ... >>> net = Net() >>> opt = paddle.optimizer.LBFGS(learning_rate=1, max_iter=1, max_eval=None, tolerance_grad=1e-07, tolerance_change=1e-09, history_size=100, line_search_fn='strong_wolfe', parameters=net.parameters()) >>> def train_step(inputs, targets): ... def closure(): ... outputs = net(inputs) ... loss = paddle.nn.functional.mse_loss(outputs, targets) ... print('loss: ', loss.item()) ... opt.clear_grad() ... loss.backward() ... return loss ... opt.step(closure) ... >>> for input_np, target_np in zip(inputs, targets): ... input = paddle.to_tensor(input_np) ... target = paddle.to_tensor(target_np) ... train_step(input, target) c >[5 UcUS-S-nXlX lX0lX@lXPlX`lXpl[U[R5(a[S[U5-5e[[5Ul[ TU]ESUU U U S9 [UR$S[5(dUR$UlO([)UR*5HupU SUlM SUlg)Nz^parameters argument given to the optimizer should be an iterable of Tensors or dicts, but got ?) learning_rate parameters weight_decay grad_clipnamerparams)r;rmax_itermax_evaltolerance_gradrc history_sizeline_search_fn isinstancer4r TypeErrortyperdictr0super__init___parameter_list_params enumerate _param_groups _numel_cache)selfrrrrrcrrrrrridx param_group __class__s r,rLBFGS.__init__s   !|q(H*  , 0(, j&-- 0 0<>B:>NO  !&  !%  $..q1488//DL$-d.@.@$A *84 %B!r+cx0nURR5Hup#URX#05 M SU0$)aPReturns the state of the optimizer as a :class:`dict`. Return: state, a dict holding current optimization state. Its content differs between optimizer classes. Examples: .. code-block:: python >>> import paddle >>> paddle.disable_static() >>> net = paddle.nn.Linear(10, 10) >>> opt = paddle.optimizer.LBFGS( ... learning_rate=1, ... max_iter=1, ... max_eval=None, ... tolerance_grad=1e-07, ... tolerance_change=1e-09, ... history_size=100, ... line_search_fn='strong_wolfe', ... parameters=net.parameters(), >>> ) >>> def train_step(inputs, targets): ... def closure(): ... outputs = net(inputs) ... loss = paddle.nn.functional.mse_loss(outputs, targets) ... opt.clear_grad() ... loss.backward() ... return loss ... ... opt.step(closure) ... >>> inputs = paddle.rand([10, 10], dtype="float32") >>> targets = paddle.to_tensor([2 * x for x in inputs]) >>> n_iter = 0 >>> while n_iter < 20: ... loss = train_step(inputs, targets) ... n_iter = opt.state_dict()["state"]["func_evals"] ... print("n_iter:", n_iter) r0)r0itemsupdate)r packed_statekvs r, state_dictLBFGS.state_dicts@\ JJ$$&DA    ''&&r+cnURc[SURS5UlUR$)Nc&XR5-$N)numel)totalps r,LBFGS._numel..+s !2r+r)rrr)rs r,_numel LBFGS._numel's4    $ &2DLL!!D    r+c/nURHdnURc'[R"U5R S/5nOURR S/5nUR U5 Mf [R "USS9$)Nr=rr>)rr_r4 zeros_likereshapeappendconcat)rviewsrviews r,_gather_flat_gradLBFGS._gather_flat_grad0snAvv~((+33RD9vv~~rd+ LL   }}U++r+c >SnURHvnUR/:wa[SUR5OSn[R"UR X#X5-R UR5U-5U5nX5- nMx X0R5:Xdeg)Nrc X-$rr#rAs r,r!LBFGS._add_grad..>sr+r )rshaperr4assignaddrr)rr directionoffsetrrs r, _add_gradLBFGS._add_grad;sA;<77b=F-qww7aE v~6>>qwwG%O A OF&&&r+c`URVs/sHoR5PM sn$s snfr)rr[)rrs r, _clone_paramLBFGS._clone_paramHs"#'<<0>> import paddle >>> paddle.disable_static() >>> inputs = paddle.rand([10, 10], dtype="float32") >>> targets = paddle.to_tensor([2 * x for x in inputs]) >>> net = paddle.nn.Linear(10, 10) >>> opt = paddle.optimizer.LBFGS( ... learning_rate=1, ... max_iter=1, ... max_eval=None, ... tolerance_grad=1e-07, ... tolerance_change=1e-09, ... history_size=100, ... line_search_fn='strong_wolfe', ... parameters=net.parameters(), >>> ) >>> def closure(): ... outputs = net(inputs) ... loss = paddle.nn.functional.mse_loss(outputs, targets) ... print("loss:", loss.item()) ... opt.clear_grad() ... loss.backward() ... return loss ... >>> opt.step(closure) rrrr Nrrrrrrrr!r)dtypeg|=r"r= strong_wolfez only 'strong_wolfe' is supportedc*>TRTXU5$r)r)rBrrrrs r,r\LBFGS.step..obj_funcs#'#=#= '1$r+)$r4no_grad enable_gradrrrrrcrrr0 setdefaultr rrZrIgetneg to_tensorrsubtractmultiplyrDlenpoprrangerrr[rHr@ RuntimeErrorrr{r)%rrrrrrrcrrr0 orig_lossr^ current_evalsropt_condrrrrrrrr!rrCsysnum_oldr"qirbe_ir`rhx_initr\s%`` r,step LBFGS.stepVsR^^ ((*73G ..M}}H}}H!00N#44 !00N,,LJJE   \1 -   Xq ) I#DM , 1 $ ..0I }}**,>H 7 < #AIIg&EYYx(FYYx(F4BYYx(F"YY'78N +.IF8#! h1$ ?a'! AFFB#--cIF"**>:A 6#3#3E#IJAQBEzv;,6"JJqM"JJqMFF1I a( a( #(+"$c!Qi"&kG5('+f|&;d tB" A"7Q;B7 #F1Iq 1BqE 91 aeeF1I"Q%,@&A1E8 #OOAv66A"7^"6!9a02a58 aeeF1IA,F&GK,")%.__%6NMM)^< ?a'Cy}}':':'>ti'(+;;!,X%C8#F#J"E'N$E(O$E(OE$K$E(O&4E" #!*E+ gjK21a js2C:W+OW+4WB!W+)'W+ W( $W++ W:c[S5e)z}Empty method. LBFGS optimizer does not use this way to minimize ``loss``. Please refer 'Examples' of LBFGS() above for usage.zeLBFGS optimizer does not use this way to minimize loss. Please refer 'Examples' of LBFGS() for usage.)NotImplementedError)rr^startup_programr no_grad_sets r,minimizeLBFGS.minimize6s" s  r+) rrrrrrrr0rcr) rNgHz>& .>dNNNNN)rr rrrz int | Nonerr rcr rrr str | Nonerz4Sequence[Tensor] | Sequence[_ParameterConfig] | Nonerz%float | WeightDecayRegularizer | NonerzGradientClipBase | NonerrreturnNone)rr.)rr)rr)NNN)rr)r%r&r'r(__doc__rrrrrrrrr non_static_onlyrrr* __classcell__)rs@r,r}r}es Xx ## $"&%)KO>B-11!1!1! 1!  1!  1!1!#1!I1!<1!+1!1! 1!1!f2'h!, '1$]]@HL   r+r}r)g-C6?g?r)# __future__rr7r9 collectionsr functoolsrtypingrrrtyping_extensionsr r4baser optimizerr collections.abcrrpaddle.nn.cliprpaddle.regularizerrr__all__rr.r;rDrWr{r}r#r+r,r s# #55)  (/9+  ") "i   !/X   m4`W IW r+