x-j)ddlmZddlmZmZddlZddlZddlm Z ddl m Z m Z m Z er ddlmZddlmZ d%d&d$ZdS)') annotations) TYPE_CHECKINGLiteralN) strong_wolfe)_value_and_gradient&check_initial_inverse_hessian_estimatecheck_input_type)Callable)Tensor2Hz>& .>r?float32objective_funcCallable[[Tensor], Tensor]initial_positionr max_itersinttolerance_gradfloattolerance_change initial_inverse_hessian_estimate Tensor | Noneline_search_fnLiteral['strong_wolfe']max_line_search_itersinitial_step_lengthdtypeLiteral['float32', 'float64']name str | Nonereturn0tuple[bool, int, Tensor, Tensor, Tensor, Tensor]c   dvrtd dd} t|d| tj|jd |}n t|d | t |tj|} tj|} t| \}}tj d gd d }tj d gdd }tj d gd d }tj d gd d }fd} fd}tj j ||||||| ||| g||| ||| fS)a Minimizes a differentiable function `func` using the BFGS method. The BFGS is a quasi-Newton method for solving an unconstrained optimization problem over a differentiable function. Closely related is the Newton method for minimization. Consider the iterate update formula: .. math:: x_{k+1} = x_{k} + H_k \nabla{f_k} If :math:`H_k` is the inverse Hessian of :math:`f` at :math:`x_k`, then it's the Newton method. If :math:`H_k` is symmetric and positive definite, used as an approximation of the inverse Hessian, then it's a quasi-Newton. In practice, the approximated Hessians are obtained by only using the gradients, over either whole or part of the search history, the former is BFGS, the latter is L-BFGS. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp140: Algorithm 6.1 (BFGS Method). Args: objective_func: the objective function to minimize. ``objective_func`` accepts a 1D Tensor and returns a scalar. initial_position (Tensor): the starting point of the iterates, has the same shape with the input of ``objective_func`` . max_iters (int, optional): the maximum number of minimization iterations. Default value: 50. tolerance_grad (float, optional): terminates if the gradient norm is smaller than this. Currently gradient norm uses inf norm. Default value: 1e-7. tolerance_change (float, optional): terminates if the change of function value/position/parameter between two iterations is smaller than this value. Default value: 1e-9. initial_inverse_hessian_estimate (Tensor, optional): the initial inverse hessian approximation at initial_position. It must be symmetric and positive definite. If not given, will use an identity matrix of order N, which is size of ``initial_position`` . Default value: None. line_search_fn (str, optional): indicate which line search method to use, only support 'strong wolfe' right now. May support 'Hager Zhang' in the future. Default value: 'strong wolfe'. max_line_search_iters (int, optional): the maximum number of line search iterations. Default value: 50. initial_step_length (float, optional): step length used in first iteration of line search. different initial_step_length may cause different optimal result. For methods like Newton and quasi-Newton the initial trial step length should always be 1.0. Default value: 1.0. dtype ('float32' | 'float64', optional): data type used in the algorithm, the data type of the input parameter must be consistent with the dtype. Default value: 'float32'. name (str, optional): Name for the operation. For more information, please refer to :ref:`api_guide_Name`. Default value: None. Returns: output(tuple): - is_converge (bool): Indicates whether found the minimum within tolerance. - num_func_calls (int): number of objective function called. - position (Tensor): the position of the last iteration. If the search converged, this value is the argmin of the objective function regarding to the initial position. - objective_value (Tensor): objective function value at the `position`. - objective_gradient (Tensor): objective function gradient at the `position`. - inverse_hessian_estimate (Tensor): the estimate of inverse hessian at the `position`. Examples: .. code-block:: python :name: code-example1 >>> # Example1: 1D Grid Parameters >>> import paddle >>> # Randomly simulate a batch of input data >>> inputs = paddle. normal(shape=(100, 1)) >>> labels = inputs * 2.0 >>> # define the loss function >>> def loss(w): ... y = w * inputs ... return paddle.nn.functional.square_error_cost(y, labels).mean() >>> # Initialize weight parameters >>> w = paddle.normal(shape=(1,)) >>> # Call the bfgs method to solve the weight that makes the loss the smallest, and update the parameters >>> for epoch in range(0, 10): ... # Call the bfgs method to optimize the loss, note that the third parameter returned represents the weight ... w_update = paddle.incubate.optimizer.functional.minimize_bfgs(loss, w)[2] ... # Use paddle.assign to update parameters in place ... paddle. assign(w_update, w) .. code-block:: python :name: code-example2 >>> # Example2: Multidimensional Grid Parameters >>> import paddle >>> def flatten(x): ... return x. flatten() >>> def unflatten(x): ... return x.reshape((2,2)) >>> # Assume the network parameters are more than one dimension >>> def net(x): ... assert len(x.shape) > 1 ... return x.square().mean() >>> # function to be optimized >>> def bfgs_f(flatten_x): ... return net(unflatten(flatten_x)) >>> x = paddle.rand([2,2]) >>> for i in range(0, 10): ... # Flatten x before using minimize_bfgs ... x_update = paddle.incubate.optimizer.functional.minimize_bfgs(bfgs_f, flatten(x))[2] ... # unflatten x_update, then update parameters ... paddle.assign(unflatten(x_update), x) )rfloat64z?The dtype must be 'float32' or 'float64', but the specified is . minimize_bfgsrrr Nrrint64shape fill_valuer Fboolc|k|zS)N) kdone is_convergenum_func_callsxkvalueg1Hkrs i/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/paddle/incubate/optimizer/functional/bfgs.pycondzminimize_bfgs..condsI $&&ctj|| }dkrt||\} }} } ntdd|| z }| |z} | |z } || z}| }tj| d} tj| d} tj| | tjjdkfdfd}|| z| zz }|| z| zz }tjtj||||| z| zz}|d z }tj |tj }tj |tj }tj||kz|kz|tj||tj|| dkz|||||||||gS) Nr)fr6pkrrr zNCurrently only support line_search_fn = 'strong_wolfe', but the specified is ''rgc4tjdgdS)Nrg@@r,)paddlefullr*sr:z-minimize_bfgs..body..sFKqcfEJJJr<cdz S)Nrr1)rhok_invsr:rDz-minimize_bfgs..body..s C(Nr<r)p)rBmatmulrNotImplementedError unsqueezedotstaticnnr;tlinalgnormnpinfassign)r2r3r4r5r6r7r8r9r?alphag2 ls_func_callsskykrhok Vk_transposeVkgnormpk_normrFIr rrrrrrs @r:bodyzminimize_bfgs..bodys?mB## # ^ + +.: /$7 /// +E5"mm&raorrr  -'RZ "W "W   b! $ $  b! $ $:b"%%}$$ O J J J J " " " "   4"9rttvv-- RTTVV# # M&- b992 > >Ri"$$&&  ! Q ""2"00-$$R26$22 EN* +w9I/I JD     dK((( desl+T2224nb%RHHr<)r;r_ loop_vars) ValueErrorr rBeyer-r rSdetachrrCrLrM while_loop)rrrrrrrrrr r"op_namer9r6r7r8r5r2r3r4r;r_r^s` ``` ```` @r:r)r)$sF *** Ve V V V   G%'97CCC #)!,E:::A'/+,(( , .    //OPPP 7 8 8B '..00 1 1B#NB77IE2[sqHHHN  1#!7;;;A ;aSU& A A AD+QCEHHHK'''''7I7I7I7I7I7I7I7I7I7I7I7Ir M  dKUBK  E2r 99r<) r rrNrr rrN)rrrr rrrrrrrrrrrrrrr r!r"r#r$r%) __future__rtypingrrnumpyrQrB line_searchrutilsrr r collections.abcr r r)r1r<r:rls#""""")))))))) %%%%%% ((((((  "6:.