Αi1(SSKJr SSKrSSKJrJrJr SSKrSSK r SSK J r SSK J r SSK Jr SSKJr SS KJr S S KJrJrJrJr \(a#SS KJr SS KJr SSKJr SSKJr "SS\5r /r!Sr"Sr#"SS\ 5r$SSjr%g)) annotationsN) TYPE_CHECKINGAny TypedDict) framework)PyLayer)core)get_rng_state_tracker)utils)check_recompute_necessarycustom_state_managerdetach_variableswitch_rng_state_tracker)Callable) NotRequired)Group)Layerc4\rSrSr%S\S'S\S'S\S'Srg) _Ctx+rmp_groupzNotRequired[bool]offload partitionN)__name__ __module__ __qualname____firstlineno____annotations____static_attributes__rs/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/distributed/fleet/recompute/recompute_hybrid.pyrr+s""$$r#rcURnURnUS:aU$[R"U5nUS:wdS5eXB-S:XdSUSUS35eUR 5nXB-nXc-nXv-nXWU$)Nr rzcan't recompute zero elementz The capacity of the activation (z#) cannot be divisible by mp_degree())nranksrankpaddlenumelflatten_) tensorr mp_degreemp_rank tensor_numeldata part_sizestartends r$_split_activationr44sImmG1} <<'L 1 <<<   #q ( *<.8[\e[ffgh ( ?? D)I  E  C c?r#cRURnURnUS:aU$[UR5nUS==UR-ss'[R "X@R 5nURRUR5U5nUR5 U$)Nr r) r'r(listshaper)emptydtype process_group all_gathercudawait)r,rr-r. tensor_shapeouttasks r$_merge_activationrAIsImmG1}  %LOx&O ,,|\\ 2C  ! ! , ,V[[]C @DIIK Jr#c8\rSrSrSr\S5r\S5rSrg)_HPRecomputeFunctionXab Compared with paddle.distributed.fleet.utils.recompute, there are the following differences: 1. In order to support PipeLineParallel, the input of recompute is modified to ensure that the input can be tuple type. 2. Offload support for activation 3. Support MP segmentation of activation to further reduce cuda memory 4. Adapt to the random state of MP c^XlXl[R"5Ul[ 5R 5Ul[RR5Ul [R"5Ul U"5UlX`lXplX0lX@lXPl/Ul/Ul/Ul/n [R."5n S[R."5;dXS[R."5;d?U R1S5S[R2R55;d SU S35e[6R8"5n U R:[<R>R@:XaSOSUl!U R:[<R>RD:XaS Ul#ObU R:[<R>RH[<R>R@4;aS Ul#O[KS U R:35eU RLUl'U RQ5uUl)Ul*[RV"5 U"U0U D6n SSS5 [YU5GHYup[RZ"U5(GaUR\nU(agUR,R_UR`5 [cURe5U5Rg5nU(aURi5OUnOU(aURi5OUnUUl.U R_U5 UR*R_U5 UR(R_S5 [6Rj"5(aU(aURmU5 GM8GM;GM>UR(R_U5 GM\ URn"U 6 [RZ"W 5(aX-/- nU $X-- n[qU 5$!,(df  GN=f) Nzgpu:zxpu::rz2Recompute with RNG is not support current device: .FTO2O1zunsupported amp level: )9 run_functionkwargsr) get_rng_state fwd_rng_stater get_states_trackerfwd_rng_state_trackernprandom get_statefwd_numpy_stategetstatefwd_random_statefwd_custom_statecustom_get_state_funccustom_set_state_funcrrrinputstensor_indices tensor_shapes get_devicesplitdeviceget_all_custom_device_typer_dygraph_tracer _amp_levelr AmpLevelO0is_fw_autocastrH amp_levelrI ValueError _amp_dtype amp_dtype_get_amp_op_listamp_white_listamp_black_listno_grad enumerate is_tensor stop_gradientappendr7r4detachclonecpuin_dynamic_modemark_non_differentiablesave_for_backwardtuple)ctxrJ all_outputsrrrrWrXargsrK tensor_inputs cur_devicetraceroutputsiargstates r$forward_HPRecomputeFunction.forwardasF( #002$9$;$N$N$P! ii113%046$9!$9!  !   &&( f'') )**,,$Q'}}779: N @ |1 M  N :**,&&$--*:*::E     0 0 0 CM   4==#3#3T]]5E5E"F F CM6v7H7H6IJK K)) 171H1H1J.C. ^^ "D3F3G oFA$$))%%,,SYY7 1 h!eg.5)--/)C'.#'')CC$)!$$S)""))!, !!$',,..5//449. !!#&;&> }-   G $ $ 9 $KN  "K> !S s # P P,c [RRR5 [ UR 5nUR nURn[ UR55n[RR5Rn[U5HupxUR(aPXWRn [XWUR 5R#5R%XG5XW'XUlUR&(aXWR)U5OXWX('M [*R,"5n SU l[1UR2UR4UR6UR8UR:UR<UR>5 UR@(a[RBREUR@URFURHURJURLS9 [O[QU55n URR"U 0URTD6n SSS5 O0[O[QU55n URR"U 0URTD6n SSS5 [WW [XRZR\5(aU 4n [_U 5[_U5:Xde/n /n[a[_U 55Hkn[WX[XRZR\5(dM0XR(aMEU RcX5 URcX5 Mm [_U 5S:Xa [eS5e[RfRiX5 [QSW 55nUsSSS5 $!,(df  GN:=f!,(df  GND=f!,(df  g=f)NT)enablecustom_white_listcustom_black_listlevelr9rzInone of output has stop_gradient=False, this recompute() is not necessaryc3# UH@n[U[RR5(dM.UR 5v MB g7fN) isinstancer eagerTensor _grad_ivar).0inps r$ 0_HPRecomputeFunction.backward..s7*Cc4::#4#45!  *s -A A )5r)basedygraphguardr6rYrZr[ saved_tensor distributed ParallelEnv device_idrmrrorArrqreshape_rr<rr` _has_gradrrMrOrSrUrVrWrXrdamp auto_castrjrkrerhrrwrJrKrr rrlenrangerp RuntimeErrorautogradbackward)rxrzrYrZr[tensorsrridxrr}detached_inputsr~forward_outputs_with_gradbackward_inputsgradss r$r_HPRecomputeFunction.backwards  [[ & & (#**%F //N--M3++-.G**668BBI#N3==#J44E)'*cllC!-"23J 05AJ,25++GJOOI.7: 4..0F#F *!!))##$$$$))))%%--"11*-*<*<*-*<*<!mm!mm .+:%-*H"%"2"2,#03 #'6eFm&DO!..N3::NG/2'4::#4#455"*w<3t9, ,,(* % O3w<(wz4::+<+<==#J444-44WZ@#**473 ),-2"_ OO $ $%> P*E [) (F3) (sKFO*A%N61N$9N69BOO$A6O$ N3 .N66 O O OrN) rrrr __doc__ staticmethodrrr"rr#r$rCrCXs4d"d"LNNr#rCc URSS5nUcS5eURSS5nURSS5n[R"5R(a [ U5 [ R c[ RbeS SjnS SjnO [ R n[ Rn/n [R"UU UUUUU/UQ70UD6 [U 5S :XaU S $U HDn [R"U 5(dM [R"U 5(aM=S U lMF [!U 5$) a recompute intermediate activations to save the memory in hybrid parallel scene. # NOTE(shenliang03)The current hybrid parallel recompute has limitations. # It cannot handle the following situations: # 1. The calculation output of recompute, there are tensors that do not require gradients. # 2. The forward output tensor has no gradient. This problem can be solved temporarily by detach(). # 3. Here, we only use float dtype to distinguish whether a gradient is needed in output tensor Parameters: ctx(dict): include 'mp_group', 'offload', and 'partition' keys. the key 'mp_group' (Group), represents the activations are splitted in which group. the key 'offload' (bool, optional, default=False), represents whether to offload to cpu. the key 'partition' (bool, optional, default=False), represents whether to split activations in the mp_group. function(paddle.nn.Layer): layer of sequence of layers that describes part of forward pass of the model whose intermediate activations will be released to save memory in forward stage and will be recomputed in backward stage for gradient calculation. *args(Tensor): inputs(tuple) to the function. **kwargs(Dict): inputs(dict) to the function. Returns: Output of function on args and kwargs. rNz8ctx must contains mp_group and mp_group can not be None.rFrcgrrxs r$"recompute_hybrid..Atr#cgrrrs r$rrBrr#r rTr)getrr`rrrrWrXrCapplyrr)rnr is_float_tensorrorw) rxfunctionrzrKrrrrWrXryoutputs r$recompute_hybridrs=4wwz4(H  B ggi'G U+I  ",,!$'119#99AAA 3 3 4 J J 4 J JK      ;11~!F''0E0Ef0M0M'+$"[!!r#) rxrrzLayer | Callable[..., Any]rzrrKrreturnr)& __future__rrQtypingrrrnumpyrPr)rpaddle.autogradrpaddle.frameworkr $meta_parallel.parallel_layers.randomr meta_parallel.pp_utilsr recomputerrrrcollections.abcrtyping_extensionsr&paddle.distributed.communication.groupr paddle.nnrr__all__r4rArCrrr#r$rs# 00 #!H*(-<%y% * 7DA" A"3A"