͑i:SSKJr SSKrSSKJr SSKJr SSKJrJ r J r SSK r SSK r SSK JrJr SSKJrJr SSKJr SS KJrJrJr SS KJr S S KJr \(aSS K Jr SSKJr SSK J!r! "SS\ 5r""SS\5r#Sr$"SS5r%"SS\%5r&g)) annotationsN) defaultdict)Enum) TYPE_CHECKINGAny TypedDict)_C_ops _legacy_C_ops)core unique_name) check_type)Operator_dygraph_tracer in_pir_mode)in_dynamic_mode)amp_global_state)Tensor)OptimizerWithMixedPrecision) Optimizercf\rSrSr%S\S'S\S'S\S'S\S'S\S 'S\S 'S\S 'S \S 'Srg)_ScaleStateDict)rscalefloat incr_ratio decr_ratiointincr_every_n_stepsdecr_every_n_nan_or_inf incr_count decr_countbooluse_dynamic_loss_scalingN)__name__ __module__ __qualname____firstlineno____annotations____static_attributes__r%V/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/amp/grad_scaler.pyrr)s/ !$$"&&r,rc \rSrSrSrSrSrSrg)OptimizerState4rrr%N)r&r'r(r)INITUNSCALEDSTEPPEDr+r%r,r-r/r/4s DHGr,r/c&S[R0$)Nstate)r/r2r%r,r-_refresh_optimizer_stater7:s ^(( ))r,c\rSrSrSrSSSjjrSSjrSSjrSrSr SSjr SS jr SS jr SS jr SS jrSS jrSSjrS SjrS!SjrS"SjrS!SjrS#SjrS$SjrS%SjrSrg)& AmpScaler>a$ AmpScaler is used for Auto-Mixed-Precision training/inferring in imperative mode. It controls the scaling of loss, helps avoiding numerical overflow. The object of this class has seventeen methods `scale()`, `unscale_()`, `minimize()` and `get`/`set` api of parameters. `scale()` is used to multiply the loss by a scale ratio. `unscale_()` is used to unscale the gradients of parameters, multiplies the gradients of parameters by 1/(scale ratio) `minimize()` is similar as `optimizer.minimize()`, performs parameters updating, and it will update the loss_scaling. Commonly, it is used together with `amp_guard` to achieve Auto-Mixed-Precision in imperative mode. Args: enable(bool, optional): Enable loss scaling or not. Default is True. init_loss_scaling (float, optional): The initial loss scaling factor. Default is 2**15. incr_ratio(float, optional): The multiplier to use when increasing the loss scaling. Default is 2.0. decr_ratio(float, optional): The less-than-one-multiplier to use when decreasing the loss scaling. Default is 0.5. incr_every_n_steps(int, optional): Increases loss scaling every n consecutive steps with finite gradients. Default is 1000. decr_every_n_nan_or_inf(int, optional): Decreases loss scaling every n accumulated steps with nan or inf gradients. Default is 2. use_dynamic_loss_scaling(bool, optional): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamically. Default is True. Returns: An AmpScaler object. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') >>> model = paddle.nn.Conv2D(3, 2, 3) >>> optimizer = paddle.optimizer.SGD( ... learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.AmpScaler(init_loss_scaling=1024) >>> data = paddle.to_tensor(data) >>> with paddle.amp.amp_guard(): ... conv = model(data) ... loss = paddle.mean(conv) ... scaled = scaler.scale(loss) ... scaled.backward() ... scaler.minimize(optimizer, scaled) c N[5(a[5nU(d [S5eU(aURR 5(ddURR 5(dEURR 5(d&[R"SURS35 SnXl SUl SUl SUl UR(GaUS:dS5eUS:dS5eX l X0l X@lXPlX`lSUlSUlXpl ['5(at[(R*R,R/S S /[0R2"S 5[(R4R6R9URS 9S 9Ul g[(R:"[<R>"S/5RA[<RB55Ul"[(R:"[<R>"S/5RA[<RB55Ul#[(R:"[<R>"S/5RA[<RB55Ul$[(R:"[<R>"S/5RA[<RB55Ul%[(R:"[<R>"S/5RA[<RB55Ul&[(R:"[<R>"UR/5RA[<RN55Ul SUl([S[T5Ul+gg)Nz;current_tracer is None, maybe it is not in imperative mode.zWAmpScaler can only be enabled on CUDAPlace, XPUPlace and CustomPlace, current place is z, so it makes no effect.F?zThe incr_ratio must be > 1.0.zThe decr_ratio must be < 1.0.rfloat32r loss_scaling)value)dtypeshapename initializer),rr ValueError_expected_place is_gpu_place is_xpu_placeis_custom_placewarningswarn_enable_use_dynamic_loss_scaling_init_loss_scaling_scale _incr_ratio _decr_ratio_incr_every_n_steps_decr_every_n_nan_or_inf _incr_count _decr_countrpaddlepirr create_persistable_valuer generatennrCConstantInitializer to_tensornparrayastypebool_ _found_inf_temp_found_inf_value_false_temp_found_inf_fp16_temp_found_inf_bf16_temp_found_inf_fp32r=_cache_found_infrr7_optimizer_states) selfenableinit_loss_scalingrrrr r$tracers r-__init__AmpScaler.__init__os   $&F Q&&3355))6688))99;; mntoEoEnFF^_ ).&"% <<<# D%D D## D%D D#&7 #) ) '9 $,C ) D  D -E *}}$jjooFF##$--n= & 5 5 I I"55!J! G #)"2"2HHaSM((2#4:3C3CHHaSM((240-3,<,<HHaSM((2-)-3,<,<HHaSM((2-)-3,<,<HHaSM((2-)%..HHd5567>>rzzJ )-%)45M)N&U r,c[US[R[RR4S5 UR (aj[ 5RS:waRUR(aASUlSUlSUl [R"S[ 5RS35 [5(GaLUR[RR :waUR#S5nUR(dU$[R$R'XR(5nUR+5nUR+5nUR,(aUR,(a[R.R0RR3UR,R4UR,R75UR,R95UR,R:5UlU$UR (aUR=5(dU$XR(-$) a Multiplies a Tensor by the scale factor and returns scaled outputs. If this instance of :class:`AmpScaler` is not enabled, output are returned unmodified. Args: var (Tensor): The Tensor to scale. Returns: The scaled Tensor or original Tensor. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') >>> model = paddle.nn.Conv2D(3, 2, 3) >>> optimizer = paddle.optimizer.SGD( ... learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.AmpScaler(init_loss_scaling=1024) >>> data = paddle.to_tensor(data) >>> with paddle.amp.amp_guard(): ... conv = model(data) ... loss = paddle.mean(conv) ... scaled = scaler.scale(loss) ... scaled.backward() ... scaler.minimize(optimizer, scaled) varzAmpScaler.scale()float16Fr<z6It is not recommended to use dynamic loss scaling for z&, so GradScaler is disable by default.r=)r rUrrVValuerKr amp_dtyperLrMrIrJrr@r DataTypeFLOAT32r^r multiplyrNget_defining_op dist_attrbase libpaddlecreate_op_dist_attribute process_meshoperandsresultschunk_id_is_initialized)rgrn scale_out multiply_op src_var_ops r-rAmpScaler.scales<    ]]FJJ,, -    LL ",, 9.. DL-2D *&)D # MMHIYI[IeIeHfgMN  ==yyDMM111jj+11  ..sKK@I#335K,,.J$$)=)=KK))--FF#--::#--668#--557",,55 % ||3#6#6#8#8J[[  r,cV[5(a[U[RRR R 5(deURUlURUlURUl USUl UR(aRURUl URUl URUlURUlSUlSUlUR$"U0UD6$UR&(dUR$"U0UD6$UR([+U5nUS[,R.LaUR1U5 SupV[3US5(aGUR5SUR65 UR$"U0UD6upVUR9S5UlO4UR6(aSUlOUR$"U0UD6upVSUlUR(aUR=5 [?[@5UlXV4$) a This function is similar as `Optimizer.minimize()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Finally, the loss scaling ratio is updated. Args: optimizer(Optimizer): The optimizer used to update parameters. args: Arguments, which will be forward to `Optimizer.minimize()`. kwargs: Keyword arguments, which will be forward to `Optimizer.minimize()`. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') >>> model = paddle.nn.Conv2D(3, 2, 3) >>> optimizer = paddle.optimizer.SGD( ... learning_rate=0.01, ... parameters=model.parameters() ... ) >>> scaler = paddle.amp.AmpScaler(init_loss_scaling=1024) >>> data = paddle.to_tensor(data) >>> with paddle.amp.amp_guard(): ... conv = model(data) ... loss = paddle.mean(conv) ... scaled = scaler.scale(loss) ... scaled.backward() ... scaler.minimize(optimizer, scaled) rNr6)NN_set_auxiliary_var found_infTF)!r isinstancerUstaticamp decoratorrrLrMrN _loss_scaling _scaled_lossrQrRrOrP_num_good_steps_num_bad_stepsminimizerKrfidr/r2_unscalehasattrrr`_get_auxiliary_varre_updaterr7)rg optimizerargskwargsoptimizer_state optimize_ops params_gradss r-rAmpScaler.minimizesT == !!++GG 372P2PI /+/+B+BI (&*kkI #%)!WI "--040H0H -112)-(8(8 %(,(8(8 %,0 )+/ (%%t6v6 6||%%t6v6 600I? 7 #~':': : MM) $%1" 92 3 3  ( (doo F)2););T)LV)L &L$-$@$@$MD !(,%-6-?-?-P-P* (-%  ) ) LLN!,-E!F))r,cUR(dgUR[U5nUS[RLa [ S5eUS[R La [ S5e[USS5(Ga7[URS[5(Ga/n/n/n/nURHnUSHnUR5cMURUR55 UR5R[R:Xa!URUR55 MUR5R[R :Xa!URUR55 MURUR55 M M GO[#5(a.[$R&R)UR*5unnnOUR*Vs/sH&nUR5cMUR5PM( nnUVs/sH%nUR[R:XdM#UPM' nnUVs/sH%nUR[R :XdM#UPM' nnUVs/sH%nUR[R,:XdM#UPM' nnUR.Ul[3U5(a][4R6"UUR8UUR:5 [<R>"UR0UR:5Ul[3U5(a][4R6"UUR8UUR@5 [<R>"UR0UR@5Ul[3U5(a][4R6"UUR8UURB5 [<R>"UR0URB5Ul[RUS'gs snfs snfs snfs snf)aG Unscale the gradients of parameters, multiplies the gradients of parameters by 1/(loss scaling ratio). If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: optimizer(Optimizer): The optimizer used to update parameters. Returns: The unscaled parameters or original parameters. Nr6zMunscale_() has already been called on this optimizer since the last update().z(unscale_() is being called after step(). _param_groupsrparams)"rKrfrr/r3 RuntimeErrorr4getattrrrdict _grad_ivarappendr@rUrobfloat16rr eagerget_grads_lists_parameter_listr=rar`lenr check_finite_and_unscalerNrbr bitwise_orrcrd) rgrr param_gradsparam_grads_fp16param_grads_bf16param_grads_fp32groupparams r-rAmpScaler._unscalebs|| 00I? 7 #~'>'> >_ W %)?)? ?IJ J 9ot 4 4  # #A &: : K! ! ! "00"8_E'')5#**5+;+;+=> ++-33v~~E,33E4D4D4FG"--/55H,33E4D4D4FG,33E4D4D4FG-1  JJ..y/H/HI $$$"+!:!:!:'')'E$$&!:"-$!,{{fnn4!,!$"-$!,{{foo5!,!$"-$!,{{fnn4!,!$ ::   2 2   ))   %//!:!:DO   2 2   ))   %//!:!:DO   2 2   ))   %//!:!:DO$2#:#: i $ $ $s0P3P35"P8P8'"P= P="Q?Qc NUR(dgUR(aSUlURS-UlURUR:Xar[ S[ UR5S[ UR5S[ UR535 URUR-UlSUlgSUlURS-UlURUR:Xa%URUR-UlSUlg)z Updates the loss_scaling. Nrrz$Found inf or nan, current scale is: z, decrease to: *) rKrerSrTrRprintrrNrPrQrOrgs r-rAmpScaler._updates||    D #//!3D 4#@#@@:5;M:No^cdhdodo^p]qqrsxy}zJzJtKsLM#kkD,<,<< #$   !D #//!3D 4#;#;;"kkD,<,<< #$ r,cUR$)zd Enable loss scaling or not. Returns: bool: enable loss scaling return True else return False. )rKrs r- is_enableAmpScaler.is_enables||r,cUR$)z Whether to use dynamic loss scaling. Returns: bool: if fixed loss_scaling is used return False, if the loss scaling is updated dynamically return true. )rLrs r-is_use_dynamic_loss_scaling%AmpScaler.is_use_dynamic_loss_scalings---r,cUR$)z` Return the initial loss scaling factor. Returns: float: the initial loss scaling factor. )rMrs r-get_init_loss_scalingAmpScaler.get_init_loss_scalings&&&r,cXl[R"[R"UR/5R [R 55Ulg)z Set the initial loss scaling factor by `new_init_loss_scaling`. Args: new_init_loss_scaling(int): The new_init_loss_scaling used to update initial loss scaling factor.s N)rMrUr[r\r]r^r=rN)rgnew_init_loss_scalings r-set_init_loss_scalingAmpScaler.set_init_loss_scalings@#8&& HHd--. / 6 6rzz B  r,cUR$)z Return the multiplier to use when increasing the loss scaling. Returns: float: the multiplier to use when increasing the loss scaling. rOrs r-get_incr_ratioAmpScaler.get_incr_ratio r,c*US:dS5eXlg)z Set the multiplier to use when increasing the loss scaling by `new_incr_ratio`, `new_incr_ratio` should > 1.0. Args: new_incr_ratio(float): The new_incr_ratio used to update the multiplier to use when increasing the loss scaling. r<z!The new_incr_ratio must be > 1.0.Nr)rgnew_incr_ratios r-set_incr_ratioAmpScaler.set_incr_ratio#H%HH#)r,cUR$)z Get the less-than-one-multiplier to use when decreasing the loss scaling. Returns: float: the less-than-one-multiplier to use when decreasing the loss scaling. rPrs r-get_decr_ratioAmpScaler.get_decr_ratiorr,c*US:dS5eXlg)a  Set the less-than-one-multiplier to use when decreasing the loss scaling by `new_incr_ratio`, `new_decr_ratio` should < 1.0. Args: new_decr_ratio(float): The new_decr_ratio used to update the less-than-one-multiplier to use when decreasing the loss scaling. r<z!The new_decr_ratio must be < 1.0.Nr)rgnew_decr_ratios r-set_decr_ratioAmpScaler.set_decr_ratio&rr,cUR$)z Return the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Returns: int: the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. rQrs r-get_incr_every_n_steps AmpScaler.get_incr_every_n_steps0s'''r,cXlg)a> Set the num `n` by `new_incr_every_n_steps`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Args: new_incr_every_n_steps(int): The new_incr_every_n_steps used to update the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Nr)rgnew_incr_every_n_stepss r-set_incr_every_n_steps AmpScaler.set_incr_every_n_steps9s $: r,cUR$)z Return the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Returns: int: the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. rRrs r-get_decr_every_n_nan_or_inf%AmpScaler.get_decr_every_n_nan_or_infBs,,,r,cXlg)aU Set the num `n` by `new_decr_every_n_nan_or_inf`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Args: new_decr_every_n_nan_or_inf(int): The new_decr_every_n_nan_or_inf used to update the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Nr)rgnew_decr_every_n_nan_or_infs r-set_decr_every_n_nan_or_inf%AmpScaler.set_decr_every_n_nan_or_infKs )D%r,c UR(aiURR5URURUR UR URURURS.$0$)ay Returns the state of the scaler as a `dict`, If this instance is not enabled, returns an empty dict. Returns: A dict of scaler includes: scale (tensor): The loss scaling factor. incr_ratio(float): The multiplier to use when increasing the loss scaling. decr_ratio(float): The less-than-one-multiplier to use when decreasing the loss scaling. incr_every_n_steps(int): Increases loss scaling every n consecutive steps with finite gradients. decr_every_n_nan_or_inf(int): Decreases loss scaling every n accumulated steps with nan or inf gradients. incr_count(int): The number of recent consecutive unskipped steps. decr_count(int): The number of recent consecutive skipped steps. use_dynamic_loss_scaling(bool): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamically. Default is True. )rrrrr r!r"r$) rKrNnumpyrOrPrQrRrSrTrLrs r- state_dictAmpScaler.state_dictVsp4||**,".."..&*&>&>+/+H+H".."..,0,J,J   r,cUR(dg[U5S:Xa [S5eUSSUl[R "[ R"UR/5R[ R55Ul USUl USUl USUl USUlUS UlUS UlUS Ulg) z Loads the scaler state. Args: state_dict(dict): scaler state. Should be an object returned from a call to `AmpScaler.state_dict()`. NrzdThe input state dict is empty, possibly because it was saved from a disabled instance of GradScaler.rrrrr r!r"r$)rKrrrMrUr[r\r]r^r=rNrOrPrQrRrSrTrL)rgrs r-load_state_dictAmpScaler.load_state_dictts||  z?a :  #-W"5a"8&& HHd--. / 6 6rzz B  &l3%l3#-.B#C (23L(M%%l3%l3)34N)O&r,)rerTrRrPrKr`rSrQrOrMrfrNrcrbrdrarLN)Tg@@?irTrhr#rirrrrrrrr rr$r#returnNonernrrrrz'Optimizer | OptimizerWithMixedPrecisionrrrrrz2tuple[list[Operator], list[tuple[Tensor, Tensor]]]rr#rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr)r&r'r(r)__doc__rkrrrrrrrrrrrrrrrrrrr+r%r,r-r9r9>s#.d#*"&'()-JOJO!JO JO  JO  JO"%JO#'JO JOXH!T[*:[*[* [* < [*zf;P2.'   * *(:- D+. D  D <Pr,r9c^\rSrSrSrSSU4SjjjrSU4SjjrSU4SjjrSSjrSSjr U4Sjr SU4S jjr SU4S jjr SU4S jjr S U4S jjrSU4S jjrS!U4SjjrSU4SjjrS"U4SjjrS#U4SjjrS$U4SjjrS#U4SjjrS%U4SjjrS&U4SjjrS'U4SjjrSrU=r$)( GradScaleria GradScaler is used for Auto-Mixed-Precision training in dynamic graph mode. It controls the scaling of loss, helps avoiding numerical overflow. The object of this class has nineteen methods `scale()`, `unscale_()`, `minimize()`, `step()`, `update()` and `get`/`set` api of parameters. `scale()` is used to multiply the loss by a scale ratio. `unscale_()` is used to unscale the gradients of parameters, multiplies the gradients of parameters by 1/(scale ratio) `minimize()` is similar as `optimizer.minimize()`, performs parameters updating, and it will update the loss_scaling, it equal to `step()` + `update()`. `step()` is similar as `optimizer.step()`, which performs parameters updating. `update` is used to update the loss_scaling. Commonly, it is used together with `paddle.amp.auto_cast` to achieve Auto-Mixed-Precision in dynamic graph mode. Args: enable(bool, optional): Enable loss scaling or not. Default is True. init_loss_scaling (float, optional): The initial loss scaling factor. Default is 65536.0. incr_ratio(float, optional): The multiplier to use when increasing the loss scaling. Default is 2.0. decr_ratio(float, optional): The less-than-one-multiplier to use when decreasing the loss scaling. Default is 0.5. incr_every_n_steps(int, optional): Increases loss scaling every n consecutive steps with finite gradients. Default is 2000. decr_every_n_nan_or_inf(int, optional): Decreases loss scaling every n accumulated steps with nan or inf gradients. Default is 1. use_dynamic_loss_scaling(bool, optional): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamically. Default is True. Returns: An GradScaler object. Examples: .. code-block:: python >>> import paddle >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.minimize(optimizer, scaled) # update parameters >>> optimizer.clear_grad() c 0>[TU]UUUUUUU5 g)N)superrk) rgrhrirrrr r$ __class__s r-rkGradScaler.__init__s'       # $ r,c">[TU]U5$)a Multiplies a Tensor by the scale factor and returns scaled outputs. If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: var (Tensor): The tensor to scale. Returns: The scaled tensor or original tensor. Examples: .. code-block:: python >>> import paddle >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.minimize(optimizer, scaled) # update parameters >>> optimizer.clear_grad() )rr)rgrnrs r-rGradScaler.scales<w}S!!r,c,>[TU]"U/UQ70UD6$)a This function is similar as `optimizer.minimize()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Finally, the loss scaling ratio is updated. Args: optimizer(Optimizer): The optimizer used to update parameters. args: Arguments, which will be forward to `optimizer.minimize()`. kwargs: Keyword arguments, which will be forward to `optimizer.minimize()`. Examples: .. code-block:: python >>> import paddle >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.minimize(optimizer, scaled) # update parameters >>> optimizer.clear_grad() )rr)rgrrrrs r-rGradScaler.minimizes Nw ;D;F;;r,cUR(dUR5$UR[U5nUS[R La [ S5eUS[RLaURU5 [US5(aCURSUR5 UR5 URS5Ul O0UR(aSUl OUR5 SUl [R US'UR(d[[ 5Ulgg)a This function is similar as `optimizer.step()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Args: optimizer(Optimizer): The optimizer used to update parameters. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.step(optimizer) # update parameters >>> scaler.update() # update the loss scaling ratio >>> optimizer.clear_grad() r6z7step() has already been called since the last update().rrTFN)rKsteprfrr/r4rr2rrrr`rrerLrr7)rgrrs r-r GradScaler.step"s>||>># #00I? 7 #~'='= =I  7 #~':': : MM) $ 92 3 3  ( (doo F NN $-$@$@$MD !(,% (-%#1#9#9 --%01I%JD ".r,cUR(dgUR(a$UR5 [[5Ulg)a` Updates the loss_scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.step(optimizer) # update parameters >>> scaler.update() # update the loss scaling ratio >>> optimizer.clear_grad() N)rKrLrrr7rfrs r-updateGradScaler.update^s12||   ) ) LLN%01I%JD "r,c">[TU]U5$)a} Unscale the gradients of parameters, multiplies the gradients of parameters by 1/(loss scaling ratio). If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: optimizer(Optimizer): The optimizer used to update parameters. Returns: The unscaled parameters or original parameters. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.unscale_(optimizer) # unscale the parameter >>> scaler.step(optimizer) >>> scaler.update() >>> optimizer.clear_grad() )rr)rgrrs r-unscale_GradScaler.unscale_~sBw **r,c >[TU]5$)ah Enable loss scaling or not. Returns: bool: enable loss scaling return True else return False. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> enable = scaler.is_enable() >>> print(enable) True )rrrgrs r-rGradScaler.is_enables2w ""r,c >[TU]5$)a Whether to use dynamic loss scaling. Returns: bool: if fixed loss_scaling is used return False, if the loss scaling is updated dynamically return True. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> use_dynamic_loss_scaling = scaler.is_use_dynamic_loss_scaling() >>> print(use_dynamic_loss_scaling) True )rrrs r-r&GradScaler.is_use_dynamic_loss_scaling2w244r,c >[TU]5$)a Return the initial loss scaling factor. Returns: float: the initial loss scaling factor. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> init_loss_scaling = scaler.get_init_loss_scaling() >>> print(init_loss_scaling) 1024 )rrrs r-r GradScaler.get_init_loss_scalings2w,..r,c$>[TU]U5 g)aM Set the initial loss scaling factor by `new_init_loss_scaling`. Args: new_init_loss_scaling(float): The new_init_loss_scaling used to update initial loss scaling factor. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_init_loss_scaling()) 1024 >>> new_init_loss_scaling = 1000 >>> scaler.set_init_loss_scaling(new_init_loss_scaling) >>> print(scaler.get_init_loss_scaling()) 1000 N)rr)rgrrs r-r GradScaler.set_init_loss_scalings8 %&;[TU]5$)a Return the multiplier to use when increasing the loss scaling. Returns: float: the multiplier to use when increasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> incr_ratio = scaler.get_incr_ratio() >>> print(incr_ratio) 2.0 )rrrs r-rGradScaler.get_incr_ratio2w%''r,c$>[TU]U5 g)ac Set the multiplier to use when increasing the loss scaling by `new_incr_ratio`, `new_incr_ratio` should > 1.0. Args: new_incr_ratio(float): The new_incr_ratio used to update the multiplier to use when increasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_incr_ratio()) 2.0 >>> new_incr_ratio = 3.0 >>> scaler.set_incr_ratio(new_incr_ratio) >>> print(scaler.get_incr_ratio()) 3.0 N)rr)rgrrs r-rGradScaler.set_incr_ratio+8 ~.r,c >[TU]5$)a Get the less-than-one-multiplier to use when decreasing the loss scaling. Returns: float: the less-than-one-multiplier to use when decreasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> decr_ratio = scaler.get_decr_ratio() >>> print(decr_ratio) 0.5 )rrrs r-rGradScaler.get_decr_ratioIrr,c$>[TU]U5 g)a Set the less-than-one-multiplier to use when decreasing the loss scaling by `new_incr_ratio`, `new_decr_ratio` should < 1.0. Args: new_decr_ratio(float): The new_decr_ratio used to update the less-than-one-multiplier to use when decreasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_decr_ratio()) 0.5 >>> new_decr_ratio = 0.1 >>> scaler.set_decr_ratio(new_decr_ratio) >>> print(scaler.get_decr_ratio()) 0.1 N)rr)rgrrs r-rGradScaler.set_decr_ratiodr r,c >[TU]5$)a Return the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Returns: int: the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> incr_every_n_steps = scaler.get_incr_every_n_steps() >>> print(incr_every_n_steps) 1000 )rrrs r-r!GradScaler.get_incr_every_n_stepss2w-//r,c$>[TU]U5 g)a Set the num `n` by `new_incr_every_n_steps`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Args: new_incr_every_n_steps(int): The new_incr_every_n_steps used to update the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_incr_every_n_steps()) 1000 >>> new_incr_every_n_steps = 2000 >>> scaler.set_incr_every_n_steps(new_incr_every_n_steps) >>> print(scaler.get_incr_every_n_steps()) 2000 N)rr)rgrrs r-r!GradScaler.set_incr_every_n_stepss8 &'=>r,c >[TU]5$)a" Return the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Returns: int: the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> decr_every_n_nan_or_inf = scaler.get_decr_every_n_nan_or_inf() >>> print(decr_every_n_nan_or_inf) 2 )rrrs r-r&GradScaler.get_decr_every_n_nan_or_infrr,c$>[TU]U5 g)a Set the num `n` by `new_decr_every_n_nan_or_inf`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Args: new_decr_every_n_nan_or_inf(int): The new_decr_every_n_nan_or_inf used to update the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_decr_every_n_nan_or_inf()) 2 >>> new_decr_every_n_nan_or_inf = 3 >>> scaler.set_decr_every_n_nan_or_inf(new_decr_every_n_nan_or_inf) >>> print(scaler.get_decr_every_n_nan_or_inf()) 3 N)rr)rgrrs r-r&GradScaler.set_decr_every_n_nan_or_infs< +,GHr,c >[TU]5$)a` Returns the state of the scaler as a `dict`, If this instance is not enabled, returns an empty dict. Returns: A dict of scaler includes: scale (tensor): The loss scaling factor. incr_ratio(float): The multiplier to use when increasing the loss scaling. decr_ratio(float): The less-than-one-multiplier to use when decreasing the loss scaling. incr_every_n_steps(int): Increases loss scaling every n consecutive steps with finite gradients. decr_every_n_nan_or_inf(int): Decreases loss scaling every n accumulated steps with nan or inf gradients. incr_count(int): The number of recent consecutive unskipped steps. decr_count(int): The number of recent consecutive skipped steps. use_dynamic_loss_scaling(bool): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamically. Default is True. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> scaler_state = scaler.state_dict() )rrrs r-rGradScaler.state_dictsDw!##r,c$>[TU]U5 g)a Loads the scaler state. Args: state_dict(dict): scaler state. Should be an object returned from a call to `GradScaler.state_dict()`. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> scaler_state = scaler.state_dict() >>> scaler.load_state_dict(scaler_state) N)rr)rgrrs r-rGradScaler.load_state_dicts4  +r,)rerf)Tg@rrirTrrr)rrrr)rrrrrrrrrrrr)r&r'r(r)rrkrrr r rrrrrrrrrrrrrrrr+ __classcell__)rs@r-rrs1j#*"&'()-  !       "% #'    ("@'<:'<'< '< < 'r=s##  ().HH,'G;')'T *P PP Pfc ,c ,r,