ёiSSKJr SSKrSSKrSSKJr SSKrSSKJr SSK J r SSKJ r SSKJrJrJr SSKJr SSKJr SSKJrJrJr SS KJr SS KJrJrJrJ r \(aSS KJ!r! /r"S&S jr#S&S jr$S&Sjr%Sq&Sr'Sr(S'Sjr)Sr*"SS5r+"SS\+5r,Sr-"SS5r."SS\.5r/"SS\.5r0Sq1Sr2Sq3S r4"S!S"\.5r5\RlS(S#j5r7S$r8S%r9\.r:\/r;\0r<\5r=g))) annotationsN) TYPE_CHECKING)_C_ops)core framework unique_name)check_variable_and_dtype)DataType)Variable check_typedefault_main_program)get_complete_pp_mesh) LayerHelperin_dynamic_modein_dynamic_or_pir_mode in_pir_mode)Tensorc[5(a[R"X5$[S 0[ 5D6n[ US/SQS5 [ US[S5 Uc1[R"SRURS/55nURURX RSS9nURSSU0SU0S U0S 9 U$) aM Limits the L2 norm of the input :math:`x` within :math:`max\_norm`. If the L2 norm of :math:`x` is less than or equal to :math:`max\_norm`, :math:`out` will be the same as :math:`x`. If the L2 norm of :math:`x` is greater than :math:`max\_norm`, :math:`x` will be linearly scaled to make the L2 norm of :math:`out` equal to :math:`max\_norm`, as shown in the following formula: .. math:: out = \frac{max\_norm * x}{norm(x)} where :math:`norm(x)` represents the L2 norm of :math:`x`. Args: x(Tensor): The input of clip_by_norm and data type is float32. The number of dimensions must be between [1, 9]. max_norm(float): The maximum norm value. name(str, optional): For detailed information, please refer to :ref:`api_guide_Name`. Usually name is no need to set and None by default. Returns: Tensor: The output of clip_by_norm with shape as input. The data type is float32. Examples: .. code-block:: python >>> import paddle >>> from paddle.nn import clip >>> input = paddle.to_tensor([[2.0, 2.0], [2.0, 2.0]], dtype='float32') >>> reward = clip.clip_by_norm(x=input, max_norm=1.0) >>> print(reward) Tensor(shape=[2, 2], dtype=float32, place=Place(cpu), stop_gradient=True, [[0.50000000, 0.50000000], [0.50000000, 0.50000000]]) clip_by_normX)float16float32uint16max_norm.tmpF)typenamedtype persistableOutrinputsattrsoutputs)r)rrrrlocalsr r floatrgenerate_with_ignorable_keyjoinrcreate_variablerr append_op)xrrhelperouts N/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/nn/clip.pyrr+sT""1//  468 4F 30.xen= |66 HHfkk5) *   VV$gg5 ! C  Qx8$  Jc[5(a[R"U5$[S0[ 5D6nUR UR S9nURSSU00SU0S9 U$)aS Merge by adding duplicated rows in the input SelectedRows object. Args: x(Tensor): The input selected rows to be merge. name(basestring|None): Name of the output. Returns: Tensor, merged output. Examples: .. code-block:: python >>> import paddle >>> import paddle.base as base >>> b = paddle.static.default_main_program().global_block() >>> var = b.create_var( ... name="X", dtype="float32", persistable=True, ... type=base.core.VarDesc.VarType.SELECTED_ROWS) >>> y = paddle.nn.clip.merge_selected_rows(var) merge_selected_rowsrrr!r")r2)rrr2rr&"create_variable_for_type_inferencerr+r,rr-r.s r/r2r2qsr0))!,,  ;&( ;F  3 3!'' 3 BC  "Qx  Jr0cr[5(a[R"U5$[US[S5 UR [ RRR:wa [S5e[S0[5D6nURURS9nURSSU0SU00S9 U$) a Get tensor data from input with SelectedRows type, and outputs a Tensor. .. code-block:: text input x is SelectedRows: x.rows = [0, 5, 5, 4, 19] x.height = 20 x.value = [[1, 1] [2, 2] [2, 2] [3, 3] [6, 6]] Output is DenseTensor: out.shape = [5, 2] out.data = [[1, 1], [2, 2], [2, 2], [3, 3], [6, 6]] Args: x(SelectedRows): Input with SelectedRows type. The data type is float32, float64, int32 or int64. name(str, optional): The default value is None. Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name` . Returns: Variable: DenseTensor transformed from SelectedRows. The data type is same with input. Examples: .. code-block:: python >>> import paddle >>> import paddle.base as base >>> from paddle.base import core >>> paddle.enable_static() >>> scope = core.Scope() >>> block = paddle.static.default_main_program().global_block() >>> x_rows = [0, 5, 5, 4, 19] >>> height = 20 >>> x = scope.var('X').get_selected_rows() >>> x.set_rows(x_rows) >>> x.set_height(height) >>> x = block.create_var(name="X", dtype="float32", persistable=True, type=base.core.VarDesc.VarType.SELECTED_ROWS) >>> z = paddle.nn.clip.get_tensor_from_selected_rows(x) r,get_tensor_from_selected_rowszGThe type of 'x' in get_tensor_from_selected_rows must be SELECTED_ROWS.r3rr!rr#r%r$)r7)rrr7r r rrVarDescVarType SELECTED_ROWS TypeErrorrr&r4rr+r5s r/r7r7sX}}33A66q#x!@Avv%%333 U   EFH EF  3 3!'' 3 BC  ,Qx   Jr0Fc[U5S::de[U5S:Xa'[US[5(de[nUSqU$[$)Nr)len isinstancebool'_clip_by_global_norm_using_mp_type_flagargs old_values r/"_clip_by_global_norm_using_mp_typerFsO t9>> 4yA~$q'4((((; 26q'/66r0c&UR[RRR:Xd2UR[RRR :XaB[ 5(a3UR[RRR5$UR[R:XdUR[R:Xa.[ 5(aUR[R5$U$N) rrr9r:FP16BF16rFastypeFP32r FLOAT16BFLOAT16FLOAT32r,s r/_cast_to_mp_type_if_enabledrQs 4<<'',,, 77dll**// / , . .xx ,,1122 8###qww(2C2C'C , . .xx(())r0cUR5(aUR5(dgUR5(dUR5(a[UR5S:wagg)NFrT)_is_initializedis_distis_denser?shape)grad clip_inputs r/_can_inplace_clip_gradrYsN    ! !)C)C)E)E $--//s4::!/C r0c[U5n[5(a[R"U5$Sn[ US/SQU5 [ U40[ 5D6nURUR5nSU0nSU0nURXUS9 U$)z) Return the squared L2 norm of a tensor. squared_l2_normr,)rfloat64rrrr!rr#r%) rQrrr[r rr&r4rr+)r,op_typer-r.r#r%s r/_squared_l2_normr_s $A&A%%a((G 3;W -FH -F  3 3AGG >> import paddle >>> paddle.enable_static() >>> BATCH_SIZE = 128 >>> CLIP_MAX = 2e-6 >>> CLIP_MIN = -1e-6 >>> prog = paddle.static.Program() >>> with paddle.static.program_guard(main_program=prog): ... image = paddle.static.data(name='x', shape=[None, 784], dtype='float32') ... hidden1 = paddle.static.nn.fc(image, size=128, activation='relu') ... hidden2 = paddle.static.nn.fc(hidden1, size=64, activation='relu') ... predict = paddle.static.nn.fc(hidden2, size=10, activation='softmax') ... label = paddle.static.data(name='y', shape=[1], dtype='int64') ... cost = paddle.nn.functional.cross_entropy(input=predict, label=label) ... avg_cost = paddle.mean(cost) >>> prog_clip = prog.clone() >>> prog_clip.block(0).var(hidden1.name)._set_error_clip( ... paddle.nn.clip.ErrorClipByValue( ... max=CLIP_MAX, min=CLIP_MIN)) NcV[U5nUcU*nO [U5nXlX lgrH)r'maxmin)rfrxrys r/__init__ErrorClipByValue.__init__Es*Cj ;$C*Cr0c>SURSSURS3$)Nz ByValue, min=f, max=ryrxres r/rgErrorClipByValue.__str__Ns!txxl&! ==r0cURR5nURS5 URSU/5 UR SU/5 UR SUR 5 UR SUR5 g)Ncliprr!ryrx)descr+set_type set_input set_output _set_attrryrx)rfrkrl clip_op_descs r/rm ErrorClipByValue._append_clip_opQsnzz++- f%sYK0 {3udhh/udhh/r0rxryrH) rprqrrrs__doc__rzrgrmrtror0r/rvrvs$L>0r0rvcUnURRURR5S- 5nUR5Vs/sH oDU;dM UPM snH\nUR X%5n[ USS5nUb [ U[5(d [S5eUcMKURX5 M^ gs snf)Nr> error_clipzIVariable's error_clip should be an instance of BaseErrorClipAttr or None.) ropop_sizeoutput_arg_names_var_recursivegetattrr@rar<rm)rkcontext grad_to_varop_descngrad_nfwd_varrs r/error_clip_callbackrZsKjjmmEJJ..0145G%668M8[TU]5 grH)superrz)rf __class__s r/rzClipGradBase.__init__ls r0c[erHrcres r/rgClipGradBase.__str__orir0c[erHrcrf params_gradss r/ _dygraph_clipClipGradBase._dygraph_cliprs!!r0c[erHrcrs r/ _pir_clipClipGradBase._pir_clipvrir0c[erHrcrs r/ _static_clipClipGradBase._static_clipyrir0c[5(aURU5$[5(aURU5$UH+up#[ USS5cM[ R "S5 O URU5$)Ngradient_clip_attrz'set_gradient_clip' will be ineffective, because you have set 'need_clip' in 'ParamAttr'. So, 'set_gradient_clip' is redundant and you can remove it.)rrrrrwarningswarnr)rfrpgs r/__call__ClipGradBase.__call__|sy   %%l3 3 ]]>>,/ /$12D9EMM> %$$\2 2r0c[erHrcrfrparamrWs r/_process_contextClipGradBase._process_contextrir0c[erHrc)rfrrWs r/_create_operatorsClipGradBase._create_operatorsrir0ro)rlist[tuple[Tensor, Tensor]]returnr)rprqrrrsrzrgimperative_baseno_gradrrrrrrrt __classcell__rs@r/rrksX"""""373 $3$"""r0rc^\rSrSr%SrS\S'S\S'S SU4SjjjrSSjr\R"5S5r S r S r S r S rU=r$)ClipGradByValueia\ Limit the value of multi-dimensional Tensor :math:`X` to the range [min, max]. - Any values less than min are set to ``min``. - Any values greater than max are set to ``max``. The multi-dimensional Tensor :math:`X` is not passed from this class, but the gradients of all parameters set in ``optimizer``. If ``need_clip`` of specific param is ``False`` in its ``ParamAttr``, then the gradients of this param will not be clipped. Gradient clip will takes effect after being set in ``optimizer`` , see the document ``optimizer`` (for example: :ref:`api_paddle_optimizer_SGD`). Note: ``need_clip`` of ``ClipGradByValue`` HAS BEEN DEPRECATED since 2.0. Please use ``need_clip`` in ``ParamAttr`` to specify the clip scope. Args: max (float): The maximum value to clip by. min (float, optional): The minimum value to clip by. if not set by user, it will be set to ``-max`` automatically. In this case, ``max`` must be greater than :math:`0`. Examples: .. code-block:: python >>> import paddle >>> x = paddle.uniform([10, 10], min=-1.0, max=1.0, dtype='float32') >>> linear = paddle.nn.Linear(in_features=10, out_features=10, ... weight_attr=paddle.ParamAttr(need_clip=True), ... bias_attr=paddle.ParamAttr(need_clip=False)) >>> out = linear(x) >>> loss = paddle.mean(out) >>> loss.backward() >>> clip = paddle.nn.ClipGradByValue(min=-1, max=1) >>> sdg = paddle.optimizer.SGD(learning_rate=0.1, parameters=linear.parameters(), grad_clip=clip) >>> sdg.step() r'rxryc~>[TU]5 Uc US:deU*n[U5Ul[U5Ulg)N)rrzr'rxry)rfrxryrs r/rzClipGradByValue.__init__s<  ;9 9$C::r0c>SURSSURS3$)NzClip Gradient By Value, min = r}r~rres r/rgClipGradByValue.__str__s!/|6$((1NNr0c/nUHiup4UcM [USS5SLaURX445 M.[R"X@RUR S9nURX545 Mk U$)N need_clipTFr,ryrx)rappendpaddlerryrxrfrparams_and_gradsrrnew_grads r/rClipGradByValue._dygraph_clipsp DAyq+t,5 ''/{{QHH$((CH  # #QM 2! r0c /n0n[R"S5 UHupEUcM [USS5SLaURXE45 M.URR R XE/5 [R"XPRURS9nSSS5 URUW45 URX4R'M SSS5 [X#5 U$!,(df  NR=f!,(df  N,=f)N gradient_cliprTFr) r name_scoperrrkprogram_optimized_guardrrryrxr_correct_clip_op_role_varrfrrparam_new_grad_name_dictrrrs r/rClipGradByValue._static_clips#%  ! !/ 2$91k40E9$++QF3WW__55qf=%{{QHH$((KH> ''H 63;==(0%3 ""2M >=3 2s$AC92*C(7C9( C6 2C99 DcgrHrors r/r ClipGradByValue._process_context r0cZ[R"X RURS9nX4$)Nr)rrryrxrfrrWrs r/r!ClipGradByValue._create_operatorss";;88Br0rrH)rxr'ryz float | NonerNonerstr)rprqrrrsr__annotations__rzrgrrrrrrrtrrs@r/rrsV%N J JO     $ r0rc^\rSrSr%SrS\S'SU4SjjrSSjrSr\ R"5S5r S r S r S rS rS rU=r$)ClipGradByNormiao Limit the l2 norm of multi-dimensional Tensor :math:`X` to ``clip_norm`` . - If the l2 norm of :math:`X` is greater than ``clip_norm`` , :math:`X` will be compressed by a ratio. - If the l2 norm of :math:`X` is less than or equal to ``clip_norm`` , nothing will be done. The multidimensional Tensor :math:`X` is not passed from this class, but the gradients of all parameters set in ``optimizer``. If ``need_clip`` of specific param is ``False`` in its ``ParamAttr``, then the gradients of this param will not be clipped. Gradient clip will takes effect after being set in ``optimizer`` , see the document ``optimizer`` (for example: :ref:`api_paddle_optimizer_SGD`). The clipping formula is: .. math:: Out = \left\{ \begin{array}{ccl} X & & if (norm(X) \leq clip\_norm) \\ \frac{clip\_norm*X}{norm(X)} & & if (norm(X) > clip\_norm) \\ \end{array} \right. where :math:`norm(X)` represents the L2 norm of :math:`X`. .. math:: norm(X) = ( \sum_{i=1}^{n}|x\_i|^2)^{ \frac{1}{2}} Note: ``need_clip`` of ``ClipGradByNorm`` HAS BEEN DEPRECATED since 2.0. Please use ``need_clip`` in ``ParamAttr`` to specify the clip scope. Args: clip_norm(float): The maximum norm value. Examples: .. code-block:: python >>> import paddle >>> x = paddle.uniform([10, 10], min=-1.0, max=1.0, dtype='float32') >>> linear = paddle.nn.Linear(in_features=10, out_features=10, ... weight_attr=paddle.ParamAttr(need_clip=True), ... bias_attr=paddle.ParamAttr(need_clip=False)) >>> out = linear(x) >>> loss = paddle.mean(out) >>> loss.backward() >>> clip = paddle.nn.ClipGradByNorm(clip_norm=1.0) >>> sdg = paddle.optimizer.SGD(learning_rate=0.1, parameters=linear.parameters(), grad_clip=clip) >>> sdg.step() r' clip_normcB>[TU]5 [U5UlgrH)rrzr'r)rfrrs r/rzClipGradByNorm.__init__+s y)r0c"SURS3$)Nz!Gradient Clip By Norm, clip_norm=r}rres r/rgClipGradByNorm.__str__/s24>>!2DEEr0c/nUHSup4UcM [USS5SLaURX445 M.[X@RS9nURX545 MU U$)NrTFr,r)rrrrrs r/_clip_gradientsClipGradByNorm._clip_gradients2sf DAyq+t,5 ''/#a..AH  # #QM 2! r0c$URU5$rHrrs r/rClipGradByNorm._dygraph_clip>s##L11r0c$URU5$rHrrs r/rClipGradByNorm._pir_clipBs##L11r0c/n[R"S5 0nUHupEUcM [USS5SLaURXE45 M.URR R XE/5 [XPRS9nSSS5 WRX4R'URXF45 M SSS5 [UW5 U$!,(df  NR=f!,(df  N-=f)NrrTFr) rrrrrkrrrrrrrs r/rClipGradByNorm._static_clipEs  ! !/ 2') $$91k40E9$++QF3WW__55qf=+a..IH>3;==(0 '' 6%3 ""24LM >=3 2s$AC#2C6C# C C## C1cgrHrors r/rClipGradByNorm._process_contextWrr0c.[X RS9nX4$)Nr)rrrs r/r ClipGradByNorm._create_operatorsZs$@r0r)rr'rrr)rprqrrrsrrrzrgrrrrrrrrrtrrs@r/rrsW4l*F  222 $ r0rc[U5S:Xa[$[U5S:Xa[US[5(de[nUSqU$Nrr>)r?&_allow_pure_fp16_global_norm_clip_flagr@rArCs r/!_allow_pure_fp16_global_norm_cliprbI 4yA~554yA~*T!Wd";";;;: 15a.r0c[U5S:Xa[$[U5S:Xa[US[5(de[nUSqU$r)r?&_allow_pure_bf16_global_norm_clip_flagr@rArCs r/!_allow_pure_bf16_global_norm_cliprprr0c^\rSrSr%SrS\S'S\S'S\S'SSU4S jjjrSS jr\R"5S 5r S r S r Sr SrSrU=r$)ClipGradByGlobalNormi{a Given a list of Tensor :math:`t\_list` , calculate the global norm for the elements of all tensors in :math:`t\_list` , and limit it to ``clip_norm`` . - If the global norm is greater than ``clip_norm`` , all elements of :math:`t\_list` will be compressed by a ratio. - If the global norm is less than or equal to ``clip_norm`` , nothing will be done. The list of Tensor :math:`t\_list` is not passed from this class, but the gradients of all parameters set in ``optimizer``. If ``need_clip`` of specific param is ``False`` in its ``ParamAttr``, then the gradients of this param will not be clipped. Gradient clip will takes effect after being set in ``optimizer`` , see the document ``optimizer`` (for example: :ref:`api_paddle_optimizer_SGD`). The clipping formula is: .. math:: t\_list[i] = t\_list[i] * \frac{clip\_norm}{\max(global\_norm, clip\_norm)} where: .. math:: global\_norm = \sqrt{\sum_{i=0}^{N-1}(l2norm(t\_list[i]))^2} Note: ``need_clip`` of ``ClipGradyGlobalNorm`` HAS BEEN DEPRECATED since 2.0. Please use ``need_clip`` in ``ParamAttr`` to specify the clip scope. Args: clip_norm (float): The maximum norm value. group_name (str, optional): The group name for this clip. Default value is ``default_group``. auto_skip_clip (bool, optional): skip clipping gradient. Default value is ``False``. Examples: .. code-block:: python >>> import paddle >>> x = paddle.uniform([10, 10], min=-1.0, max=1.0, dtype='float32') >>> linear = paddle.nn.Linear(in_features=10, out_features=10, ... weight_attr=paddle.ParamAttr(need_clip=True), ... bias_attr=paddle.ParamAttr(need_clip=False)) >>> out = linear(x) >>> loss = paddle.mean(out) >>> loss.backward() >>> clip = paddle.nn.ClipGradByGlobalNorm(clip_norm=1.0) >>> sdg = paddle.optimizer.SGD(learning_rate=0.1, parameters=linear.parameters(), grad_clip=clip) >>> sdg.step() r'rr group_namerAauto_skip_clipc>[TU]5 [U5UlX l[ U[ 5(deX0lSUlSUl g)NF) rrzr'rrr@rAr _async_add_nshould_comm_on_shard_dim)rfrrrrs r/rzClipGradByGlobalNorm.__init__sL y)$.$////, !(-%r0c"SURS3$)Nz)Gradient Clip By GlobalNorm, global_norm=r}rres r/rgClipGradByGlobalNorm.__str__s:4>>!:LMMr0c./n/n/n/nSn[U5S:a%[US5S:aUSSRnOSnUGHupU cM [USS5SLaMU n [5(a1U R 5(a[ U 5n U R 5n OHU R[RRR:Xa[ U 5n [U 5n [U 5n UbU RU:waSn[U R5n [U RR 5[U R 5:a#Sn["R$"XU R&5n ["R$"XU R&5n U R([*R,:XdU R([*R.:XaUR1U 5 GMU R([*R2:XaUR1U 5 GMUR1U 5 GM [U5[U5-[U5-S:XaU$Sn [U5S:aSOSn/n[U5S:a(U "U5nUR1UR5U55 [U5S:a@U "U5nUS:XaUR1U5 O UR1UR5U55 [U5S:aU "U5nUR1U5 U "U5nU(aUb["R6"5nU(aSUR8;aUR;S5S :aUR=S5R?S5nURA5n["RB"U["RDRFUS 9 ["RH"UURUR&5nURJ(aU[MUS 5(aD[*RNRCURA5URPS 9RS5 URJ(aU[MUS 5(aD[*RNRCURA5URTS 9RS5 [*RV"U5n[*RX"S /XRZS9nSnUR\(d+Sn[*R^"U[*R`"UUS9S9nOUU:aSn[*R^"UUS9nUGH4upU cM [USS5SLaUR1X45 M/U(GaWR(U R(:waUR5U R(5OUnURU R:wGa5[U RR 5[URR 5:aSUR&nSnUHnURc5(aMSn O U(aURA5nO[eS5e[U R5n [U RR 5[U R 5:a"["R$"UU UR&5n[*RNR%UU RUR&5n[gU U5(a&U RiU5 UR1X45 GM[*Rj"U U5nUR1UU45 GM"UR1X45 GM7 U$)NTrrFcJ[R"U5R5$rHrstacksumvar_lists r/ async_add_n7ClipGradByGlobalNorm._dygraph_clip..async_add_n <<)--/ /r0r\rppr>)rgroupsharding_group)rmp_grouprVr fill_valuer,yzLReshard a sharded tensor from a local mesh to a global mesh is not supported)6r? process_meshrris_selected_rowsr2_get_tensor_from_selected_rowsrrr9r:r;r7r_rset process_idsdistreshard placementsrrrbfloat16rrrKget_mesh dim_names get_dim_sizeget_submesh_with_dim get_group _local_value all_reduceReduceOpSUM shard_tensorr hasattr distributedrwaitrsqrtfullrrdividemaximum is_replicatedrdrY multiply_multiply)rfrrsum_square_listsum_square_list_fp16sum_square_list_fp32flag_auto_hybrid_ppsrc_meshrr merge_grad sum_squarepp_meshr sum_dtypeglobal_norm_varglobal_norm_var_fp16global_norm_var_fp32global_norm_var_fp64g_meshpp_groupglobal_norm_var_localmax_global_normrclip_varrXr( is_replicate placementrs r/r"ClipGradByGlobalNorm._dygraph_clipsw!!" | q Sa%9A%=#Aq)66HH DAyq+t,5J  Q%7%7%9%903 'FFH 4<<//===03 ::F )*5J#((B&+#.q~~>q~~112S9L9L5MM*.'!%"Z-B-B"J"\\**?*?   FNN2##v6$++J7!!V^^3$++J7&&z2Q!X  &' (&' (    0"%_!5!9Iy  # $q (#./C#D  " "#7#>#>y#I J # $q (#./C#D I%&&';<&&';'B'B9'MN  ! ##.#?  " "#7 8%o6 8#7]]_FF,,,''-1"66t<FFtL)8(D(D(F%)}}((" #'"3"3)#00#..#  ( (WT;K-L-L    ) ),,.d6I6I * df  ( (WT:-F-F    ) ),,.dmm * df ++o6 ++#Y>>  ""I}}!..?oFH .I}}/JH DAyq+t,5 ''/ ~~0OOAGG,! **ann<1>>556"//;;:&0%:%: '+ )3I#,#:#:#<#*. *GZ5J5J*J&,%7%7%?%?& 8M8M& *!Z88KK +$++QF3%q*=H$++QM: ''/k!n r0c /n/n/n/n/n/n/n/n /n /n Sn [5n SnUGHunnUR5(dMU RUR5R5 SUR5RR ;dMqUcUR5RnMUR5Rn[UR 5[UR 5:aUn[UR 5[UR 5::aGMe [ U 5S:a SSKJn Sn UceUGHunnUcM [USS5SLaMUn[5(a+UR5(a[U5n[U5n[U5nU (aUR5RU:waq[R R#UUW"UR5R$UR5RUR5R&55nUR((aUR*S(aUR,[.R0:XdUR,[.R2:XaUR5U5 GMpUR,[.R6:XaUR5U5 GMUR5U5 GMUR8(aUR,[.R0:XdUR,[.R2:XaUR5U5 GMUR,[.R6:XaUR5U5 GMIUR5U5 GM]UR,[.R0:XdUR,[.R2:XaU R5U5 GMUR,[.R6:XaU R5U5 GMU R5U5 GM [ U5[ U5-[ U5-[ U5-[ U5-[ U5-[ U 5-[ U 5-[ U 5-S:XaU$Sn[ U5[ U5-[ U 5-S:aS OS n/n/n/n[ U5S:a(U"U5nUR5UR;U55 [ U5S:a(U"U5nUR5UR;U55 [ U 5S:a(U"U 5nUR5UR;U55 [ U5S:a@U"U5nUS :XaUR5U5 O UR5UR;U55 [ U5S:a@U"U5nUS :XaUR5U5 O UR5UR;U55 [ U 5S:a@U"U 5nUS :XaUR5U5 O UR5UR;U55 [ U5S:aU"U5nUR5U5 [ U5S:aU"U5nUR5U5 [ U 5S:aU"U5nUR5U5 Sn[ U5S:aU"U5n[ U5S:a U"U5nO9UR((a(UR<(a[R>"S/US S 9nUR((aUR<(a[R@RCWURDRF[HRJRL5n[R@RCUURNRF[HRJRL5nUcUnOUU-n[ U5S:a U"U5nO9UR((a(URP(a[R>"S/US S 9nUR((aiURP(aX[R@RCWURDRF[HRJRL5nUcUnOUU-n[RR"U5n[R>"S/UR,URTS 9n Sn!URV(d+Sn![RX"U [RZ"UU S 9S 9n"OUU :aSn![RX"U US 9n"UGHvunnUcM [USS5SLaUR5UU45 M1U!(Ga*W"R,UR,:waU"R;UR,5OU"n#U (aU#R5RUR5R:wa[R R#U#UR5RW"U#R5R$U#R5RU#R5R&55n#[R\"UU#5n$UR5UU$45 GMcUR5UU45 GMy U$)NFrr>) to_placementsTr no_fusioncJ[R"U5R5$rHrrs r/r3ClipGradByGlobalNorm._pir_clip..async_add_nrr0r\rrrr)/r$is_dist_dense_tensor_typeadd dist_attrr!r%r?/paddle.distributed.auto_parallel.placement_typerTrris_selected_row_typer2r7r_rr5r' dims_mapping partial_dimsr  optimize_attrrr rMrNrrOis_distributedrKhas_dist_paramr8rr0ridr&r1r2rhas_not_dist_paramr7rrr9r:r=)%rfrrno_fusion_sum_squareno_fusion_sum_square_fp16no_fusion_sum_square_fp32sum_square_distsum_square_dist_fp16sum_square_dist_fp32sum_square_not_distsum_square_not_dist_fp16sum_square_not_dist_fp32auto_parallel_pp pp_meshespp_stage0_meshrrp_meshrTrCrDrrFno_fusion_global_normglobal_norm_distglobal_norm_not_distrHrIrJrGglobal_norm_dist_varglobal_norm_not_dist_varrNrrOrXrs% r/rClipGradByGlobalNorm._pir_clips  "$&!$&!!!!#% #%  E  DAq**,, akkm889 22>>>%-)*)C)C!"!;!;~99:S"..>.4N"6#5#56#*66;  ! y>A   $ !- -- DAqyq+t,5J}}!7!7!9!903 ::F )*5J ((*77>I#//77"!",,.;;",,.;;",,.;; 11??;/$$(8(88!''8+<+<<-44Z@%%)9)99-44Z@(// ;!!$$(8(88!''8+<+<<(// ;%%)9)99(// ;#**:6$$(8(88!''8+<+<<,33J?%%)9)99,33J?'..z:s!z $ %+, -+, -/" #&'  ( &'  ( %&  '*+ ,*+ ,   0 '(/"#%&'   !#! ( )A -#./H#I ! ( ()=)D)DY)O P # $q (#./C#D  # #$8$?$? $J K ' (1 ,#./G#H ' '(<(C(CI(N O ( )A -#./H#I I%%,,-AB%,,(// : # $q (#./C#D I% ''(<= ''(<(C(CI(NO ' (1 ,#./G#H I%$++,@A$++(// : # $q (#./C#D ! ( ()= >  ! ##.#?  # #$8 9 " #a '#.#? ' '(< = $ % ))*?@O  1 $#./?#@  * *t/B/B#);;cs$   ( (T-@-@#)==#;#;$d&9&9&<&O>O$ $*==#;#;$dmm&6&6 8I8I$ &"6"14H"H # $q ('23G'H $  * *t/F/F'-{{cs( $  ( (T-D-D'-}}'?'?(##&& !!( $ &":"14L"L ++o6 ++#_22t~~  ""I}}!..?oFH .I}}/JH DAqyq+t,5 ''A/ ~~0OOAGG,! %",,.;;{{}112"(!3!3!;!;" 22%&002??&002??&002??"J"??1j9 ''H 6 ''A/?!B r0c  ^/n/n/n/n/nU4Sjn[R"S5 UGH~upU cM [USS5SLaMU n URRR X/5 U R [RRR:Xa[U 5n [U 5n [U 5n U R[RRR:XaUR!U 5 OU R[RRR":XaUR!U 5 OUU R[RRR$:XaUR!U 5 OUR!U 5 SSS5 GM ['U5S:a['U5S:a [)S5e['U5['U5-['U5-S:Xa2['U5['U5-['U5-S:Xa UsSSS5 $WRRR UW /5 ['U5S:aSOS n /n ['U5S:aWU"U5nU(dU(d[+5(d!U R!UR-U 55 OU R!U5 ['U5S:aWU"U5nU(dU(d[/5(d!U R!UR-U 55 OU R!U5 ['U5S:a@U"U5nU S :XaU R!U5 O U R!UR-U 55 ['U5S:aU"U5nU R!U5 ['U 5S :aU"U 5OU Sn [0R2"U S 9n [0R4"S /U RTR6S 9n[0R8"U[0R:"UU S 9S 9nSSS5 0nUGHupU cM [USS5SLaUR!X45 M/URRR X/5 [=U 5nUR[RRR:XaDWR[RRR:waUR-S5nOxUR[RRR":XaDWR[RRR":waUR-S5nOWn[?5RA5nURCSUUS.SU0S9 UU La-URCSSU0SU 0URU RS.S9 SSS5 U RDUURD'UR!X45 GM SSS5 [GUW5 U$!,(df  GM!=f!,(df  GN#=f!,(df  Ny=f!,(df  NR=f)Nc>TR(a$[R"U5R5$[R"U5$rH)r rrradd_n)rrfs r/_add_n1ClipGradByGlobalNorm._static_clip.._add_ns3  ||H-1133||H--r0rrTFrz1FP16 and BF16 are not supported at the same time.r\rr>rPrrrr)elementwise_mulrYr!r]castr)in_dtype out_dtyper8)$rrrrkrrrrr9r:r;r2r7r_rrIrrJrLr?rdrrKrrr7r8rr9r:rQr current_blockr+rr)rfrrr>r?sum_square_list_bf16r@rzrrrCrDrFrGrHglobal_norm_var_bf16rIglobal_norm_var_other_dtyperN scale_varrnew_g scale_inputrks` r/r!ClipGradByGlobalNorm._static_clips!!! .  ! !/ 2$91k40E9 WW__55qf=vv!5!5!C!CC%8%; %B:%N !1*!=J!''4<<+?+?+D+DD,33J?#))T\\-A-A-F-FF,33J?#))T\\-A-A-F-FF,33J?'..z:>= %('(1,5I1JQ1N)G O$*+,*+, O$*+,*+, $M3 2P111a&9),_)=)AIy "$+,q0+12F+G(,*@BB'..077 B(../CD+,q0+12F+G(,*@BB'..077 B(../CD+,q0+12F+G( I-'../CD'..077 B'!+282I/#**+FG?+a/?+(+ #)++"@"(++#)//#~~# #MM%nn/J i:p(* $$91k40E9$++QF3WW__55qf=7:E t||';';'@'@@%OOt||/C/C/H/HH&/&6&6y&A  t||';';'@'@@%OOt||/C/C/H/HH&/&6&6z&B &/ 12@@BEOO.%*=!&$ A~!'$'<%*AJ,1KK-.WW# (5>H4566(0 ''/Y%C3^ ""24LMS>=B:9@>=Q3 2s_AY20D1X<!BY20'Y2GY3A!Y2EY!.8Y2< Y  Y2 Y Y2! Y/ +Y22 ZcURU;ad/XR'URXRS-'[R"S/URURS9XRS-'O*URXRS-:Xd [ S5eUnUR [RRR:Xa[U5n[U5nO:[5(a+UR5(a[U5n[U5n[U5nXRR!U5 Xlg)N _clip_valuer>r_clipz>All parameters' 'clip_norm' of a same group should be the same)rrrr8r ValueErrorrrr9r:r;r2r7rr\r_rr)rfrrrWrClocal_norm_vars r/r%ClipGradByGlobalNorm._process_context:s ??' )')GOO $7;~~GOOm3 417c2GOOg- .>>W__}-L%MM T 99 ,,:: :,T2J6zBJ ]]t88::,T2J6zBJ)*5 ''7 r0c8SnURS-nX@R;aU"URUR5n[R"US9nURURS-n[R"U[R "XeS9S9nUR S:XdeXpRU'[5(a&[R"X RU5nX4$URRSX RUS.S U0S 9 X4$) NcJ[R"U5R5$rHrrs r/r;ClipGradByGlobalNorm._create_operators..async_add_nUrr0_scalerPrr)r>r|r}r!r]) rrrr7r9r:rVrr=rkr+)rfrrWrgroup_scale_namegroup_norm_varrOgroup_scale_vars r/r&ClipGradByGlobalNorm._create_operatorsTs 0 ??X5 << /(doo)FGN#[[>:N||DOOg$=>H$mm..8>O#((D0 00->> import paddle >>> paddle.enable_static() >>> def network(): ... image = paddle.static.data(name='image', shape=[ ... None, 28], dtype='float32') ... param_attr1 = paddle.ParamAttr("fc1_param") ... fc1 = paddle.static.nn.fc(image, size=10, weight_attr=param_attr1) ... param_attr2 = paddle.ParamAttr("fc2_param") ... fc2 = paddle.static.nn.fc(fc1, size=10, weight_attr=param_attr2) ... loss = paddle.mean(fc2) ... return loss >>> # network 1: clip all parameter gradient >>> with paddle.static.program_guard(paddle.static.Program(), paddle.static.Program()): ... loss = network() ... paddle.nn.clip.set_gradient_clip( ... paddle.nn.ClipGradByGlobalNorm(clip_norm=2.0)) ... sgd = paddle.optimizer.SGD(learning_rate=1e-3) ... sgd.minimize(loss) >>> # network 2: clip parameter gradient by name >>> with paddle.static.program_guard(base.Program(), paddle.static.Program()): ... loss = network() ... paddle.nn.clip.set_gradient_clip( ... paddle.nn.ClipGradByValue(min=-1.0, max=1.0), ... param_list=["fc1_param", "fc2_param"]) ... sgd = paddle.optimizer.SGD(learning_rate=1e-3) ... sgd.minimize(loss) >>> # network 3: clip parameter gradient by value >>> with paddle.static.program_guard(base.Program(), paddle.static.Program()): ... loss = network() ... param_var1 = paddle.static.default_main_program().global_block().var("fc1_param") ... param_var2 = paddle.static.default_main_program().global_block().var("fc2_param") ... paddle.nn.clip.set_gradient_clip( ... paddle.nn.ClipGradByValue(min=-1.0, max=1.0), ... param_list=[param_var1, param_var2]) ... sgd = paddle.optimizer.SGD(learning_rate=1e-3) ... sgd.minimize(loss) >>> # network 4: use 'set_gradient_clip' and 'optimize(grad_clip=clip)' together >>> with paddle.static.program_guard(base.Program(), paddle.static.Program()): ... loss = network() ... clip1 = paddle.nn.ClipGradByValue(min=-1.0, max=1.0) ... clip2 = paddle.nn.ClipGradByNorm(clip_norm=1.0) ... # Set the gradient clipping strategy: clip1 ... paddle.nn.clip.set_gradient_clip(clip1) ... # Set the gradient clipping strategy: clip2 ... sgd = paddle.optimizer.SGD(learning_rate=1e-3, grad_clip=clip2) ... sgd.minimize(loss) ... # 'set_gradient_clip' will not take effect when setting has a conflict, ... # and the gradient clipping strategy will be 'clip2' zCaution! 'set_gradient_clip' is not recommended and may be deprecated in future! We recommend a new strategy: set 'grad_clip' when initializing the 'optimizer'. This method can reduce the mistakes, please refer to documentation of 'optimizer'.z<'clip' should be an instance of ClipGradBase's derived classNr op_namescope optimizerz'minimize' has been invoked before, this will make 'set_gradient_clip' be ineffective! Please invoke 'set_gradient_clip' before 'minimize'.c3B# UHn[U[5v M g7frH)r@r.0elems r/ $set_gradient_clip..s 8ZT:dC Zsc3V# UHn[U[R5v M! g7frH)r@r Parameterrs r/rrs Lz$ 3 344s')zK'param_list' should be a list of Parameter or basestring(parameter's name).)rrr@rr<rr rkops all_attrsattrall_parametersallvarcopydeepcopyr)r param_listrrrrs r/set_gradient_cliprrs:t MM 1 dL ) ) J  002mmA"" R\\^ + rww @ 1  MMW  #]]1%446  8Z 888=GHZTmmA&**40Z H LL L L Y  #'==#6  Is1)E#c P0nUHup#UcM URRRX#/5 [R"S5 [ USS5nUcUsSSS5 sSSS5 s $[ U[5(d [S5eURXUS9 SSS5 SSS5 M /n0nUHup#UcM URRRX#/5 [R"S5 WRX#S9upxURXgR'URXx/5 SSS5 SSS5 M [XV5 U$!,(df  N=f!,(df  GM}=f!,(df  NK=f!,(df  M=f)Nrrz8clip attribute should be an instance of GradientClipBase)rrrW)rrW)rkrrrrrr@rr<rrrrr) param_gradsrrr clip_attrresrrrs r/append_gradient_clip_opsrsfG 9  GGOO , ,aV 4   1#7>I " 2 1 5 4 i66N  & &wa & H 2 5 4 " C! 9  GGOO , ,aV 4   1'999JOE3;== $ZZ 0 JJ( ) 2 5 4 c< J5 2 1 5 4( 2 1 5 4sSE2 E! E230E!#E2-F>GGM**WW]3A6 9"0<#KLL <9$r0rH)rWrrXr)NN)> __future__rrrtypingrrpaddle.autogradautogradrpaddle.distributedr5r&r paddle.baserrrpaddle.base.data_feederr paddle.base.libpaddler paddle.common_ops_importr r r "paddle.distributed.utils.moe_utilsrpaddle.frameworkrrrrr__all__rr2r7rBrFrQrYr_rarvrrrrrrrrrdygraph_not_supportrrrGradientClipBaseGradientClipByValueGradientClipByNormGradientClipByGlobalNormror0r/rs+#  )!44<*OOC CL#L<~+0' 7  .""90(90x6"'"'"TZlZzj\jZ*/&*/&t <t n }7}7@!N=4 %#/r0