Αi |SSKrSSKrSSKrSSKrSSKrSSKJr SSKrSSK r SSK J r SSK J r SSKJr SSKJr SSKJrJr SSKJr S S KJrJr S S KJrJrJr \R<R>r\R<RA5r!\RDRFRH\RDRFRJ\RDRFRL\RDRFRN\RDRFRP/r)S /r*/SQr+/SQr,SSjr-Sr.Sr/Sr0Sr1Sr2Sr3Sr4Sr5Sr6Sr7Sr8Sr9Sr:SSjr;SSjrS"r?S#r@S$rAS%rBSS&jrCS'rDS(rES)rFS*rGSS+jrHS,rISS-jrJS.rKS/rLS0rMS1rNS2rOSS3jrPS4rQS5rRS6rSS7rTSS8jrUS9rVS:rWS;rXS<rYS=rZS>r[S?r\S@r]SAr^SBr_SCr`SDraSErbSFrcSGrdSHreSIrfSJrgSKrhSLri"SMSN5rjSOrkSPrlSQrmSRrnSSroSTrpSUrqSVrrSWrsSXrtSYruSZrvS[rwS\rxS]ryS^rzS_r{S`r|Sar}Sbr~ScrSdrSerSfrSSgjrSShjrSirSjr\"\5rSkrSlrSmrSnrSorSprSqrSrrSs\St\Su\4SvjrSSwjrSxrSyrSzrSS{jrSS|jrS}\S~\4SjrSrg)N)reduce) use_pir_api)pir)wrap_decorator)core)is_belong_to_optimizer is_parameter)Variable) ProcessMeshmerge_process_meshes)DistTensorSpecOperatorDistAttrTensorDistAttr expand_v2)sumsqrt fill_constantelementwise_maxelementwise_divstack reduce_sum)zbuiltin.combinez builtin.split pd_op.pylayerzcf.yieldz cf.tuple_pushz cf.tuple_popzcf.stack_createzcf.has_elementscD[R"U5nSUlUR(d`UR U5 [R "5n[R "S5nURU5 URU5 U$UR U5 U$)NFz>%(levelname)s %(asctime)s %(filename)s:%(lineno)d] %(message)s) logging getLogger propagatehandlerssetLevel StreamHandler Formatter setFormatter addHandler) log_levelnamelogger log_handler log_formats m/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/distributed/auto_parallel/static/utils.py get_loggerr+Gs   t $FF ?? "++- && L    ,+& M  " McDU[U5*:aU[U5:agg)NTF)len)listindexs r*is_valid_list_indexr1Ws! T us4y0r,cUS:waggNTFmappings r* is_dim_shardr8^"}r,cUS:Xaggr3r5r6s r*is_dim_replicater;er9r,c4Ucg[SU55(dg[[U55H(nXS:dX[UR5:dM( g [[UR55HnUR U5S:dM g g)NFc3B# UHn[U[5v M g7fN) isinstanceint).0ds r* &verify_dims_mapping..os8r5)process_mesh_listcompatible_process_meshrJs r*compute_compatible_process_meshris>" &&)  #'/*:*6'* #"r,c^[U5[U5:XdeSn/n[[U55H2n[XX5(deURXX5 M4 [ U5nUcg[[U55HnXPUX:wdMXPUX'SnM U$NFT)r.rFr1rNr^)ra index_listchangedr\rKrds r*)compute_compatible_and_update_dim_mappingrns  !S_ 44 4GL 3() *"#4#7GGGG-0?@+ Remove auto parallel suffix from distributed attribute name. )striprrprqs r*remove_distributed_attr_suffixrus ::d..0 11r,cSSKJn UcU"5nUR5(dS5eURHnURR 5HBnUR U5nURU5nUcM*UR5(aMA g URHBnURW5nURU5n U cM*UR5(aMA g M g)Nrget_default_distributed_contextz8Distributed attributes must be initialized before check.FT) dist_contextrxis_initialized_for_programblocksvarsvaluesget_dist_tensor_for_graph get_tensor_dist_attr_for_programis_validopsget_dist_op_for_graphget_op_dist_attr_for_program) programryrxblocktensor dist_tensortensor_dist_attropdist_op op_dist_attrs r*"check_distributed_attr_for_programrs=68  2 2 4 4B 4jj'')F&@@HK+LL  !,{7K7K7M7M *))B"88@G'DDRHL(73C3C3E3E   r,c[R"5nUR5 SSKJnJn UcU"5n[ USS9 O!U"5nU"U5 [ USS9 U"U5 UR5 g)z This function reuses the original program output ability with a distributed context. Using lock can avoid multiple threads change the default distributed context simultaneously. r)rxset_default_distributed_contextNT)flush) threadingLockacquireryrxrprintrelease)rrylockrxroriginal_default_contexts r*print_program_with_dist_attrrse >> DLLN 68  gT"#B#D ' 5 gT"'(@ALLNr,cdX0;d SUSU35eURU5n[X5n[X5Vs/sHoeSSPM nn[X5H nXgUU'M UVs/sHn[X5PM nnUV s/sHoU PM n n [ U 5$s snfs snfs sn f)af Given a rank and the processes mesh the rank belongs to, compute the communication peers of the rank based on the give axis in the mesh. Example: 16 processes managed in a 4-Dimensional mesh with shape of [2, 2, 2, 2]. the rank communication peers of rank 0 (included) are following: in axis 0: [0, 1] in axis 1: [0, 2] in axis 2: [0, 4] in axis 3: [0, 8] zrank [z] is NOT in processes group N)r0_linear_idx2coordinaterF_coordinate2linear_idxsorted) processesrGaxisrank rank_relative coordinaterKcoordinates_in_groupranks_in_group_relativeidxranks_in_groups r*_get_comm_groupr)s   29+> OOD)M'=J383EF3EaqM3EF5; ()Q% /.J u1.1HH0Gn0GNH . !!G IsB#-B(B-cBURU5n[X5nXR$)z Given a rank and the processes mesh the rank belongs to, compute the index of the rank in given axis. Example: 27 processes managed in a 3-Dimensional mesh with shape of [3, 3, 3]. the index of rank 22 are: in axis 0: 1 in axis 1: 1 in axis 2: 2 )r0r)rrGrrrrs r*_get_idx_in_axisrLs%OOD)M'=J  r,cX[U5[U5:Xd SUSU35e[[U55H2nXS:d SUSU35eXX:aM$SUSUSU35e USnUSn[[U5S - SS5HnXCX-- nX0U-nM U$) a[ convert a coordinate in multidimensional mesh space into a scala idx in linear space. it use Row-major order for dimension conversion. so it has: [most_significant_dim, ..., least_significant_dim] assume: the size of i-th dimension to be: S[i] the index of j-th dimension is: I[j] linear_idx of a n dimensional coordinate is: I[n-1] * (S[n-2] * S[n-3] * S[n-4] * .... S[0]) + I[n-2] * ( S[n-3] * S[n-4] * .... S[0]) + I[n-3] * ( S[n-4] * .... S[0]) + ... I[1] * ( S[0]) + I[0] zCcoordinate should have the same size as mesh shape, but got shape: z, coordinate: rzindex in dimension [z"] is least than zero. coordinate: z"index beyond extent in dimension [z ]. shape: r4r )r.rF) mesh_shaperrKbase linear_idxs r*rr_s: z?c*o - Mj\Yghrgst -3z? #}! "1#%G | T !}z}, 0:j\XbWc d , $ b>DBJ3z?Q&B /Z]**  1 0 r,cUS:d SUS35eU[R"U5:d SUSU35eSnS/[U5-n[[ [U555H nX- n[ XPU-5X4'X U-nM" U$)a mapping a linear scala into multidimensional mesh space, return it coordinate in that space. it is the inverse function of _coordinate2linear_idx. assume: the size of i-th dimension to be: S[i] the index of j-th dimension is: I[j] the coordinate given linear_idx is: I[0] = linear_idx % S[0] I[0] = (linear_idx / S[0]) % S[1] I[0] = (linear_idx / (S[0] * S[1])) % S[2] .... rzlinear index [z] is least than zeroz5linear index beyond the extent of mesh shape. shape: z, linear index: rr4)npprodr.reversedrFr@)rrrrrKoffsets r*rrs& ?MnZL8LMM?  + + ? |K[\f[gh + DJ'J eC O, -"F]23  1 . r,cNSnURH_nX$R;dMURUR:XdM0[URURR U55n O Ub#UR[ WRU5$URS$Nr)process_meshes process_idsrGrr0r)ry target_meshrrmeshs r*_get_corresponding_rankrs J++ ## # k6G6G(G/ D,,2248J  ,&& "4::z :  &&q))r,chURnURnURRn[U5[U5:XdSUSUS35e/n[ [U55HCnX&S:XdX6S:XaUR X&5 M(UR X&XCU-5 ME U$)Nzvariable shape [z] and dim_mapping [z] is NOT match !r4)rGrIrJr.rFrN)var dist_attr var_shaper7r new_shapers r*_get_unshard_dist_shapers I$$G  ! ! ' 'D y>S\ ) 9+%8 AQR )IS^$ >R 7<2#5   Y^ ,   Y^d3<.@@ A % r,cJSSKJn UcU"5nUR5HnUR(dMUR U5n[ XE5nUR RU5 URnS/[U5-nXulURXE5 M g)Nrrwr4) ryrx list_varsis_datarrdesc set_shaperIr. set_tensor_dist_attr_for_program)dist_main_progdist_startup_progryrxrr inverse_shaperUs r*make_data_unshardrs=68 '') ;;;+LL  4CJM HH  } -*77K$[!11K,7 )  9 9# P*r,c<SSSS.nU(dU$[U[5(d[S[U5S35eUR 5HKup#US;a[ SUS35e[U[ 5(d[ S[U5S35eX1U'MM U$)z(Update default addition_info with inputsr)epochbatch batch_sizez7The type of 'addition_info' should be 'dict', but got ''.z[The key of 'addition_info' should be one of the ['epoch', 'batch', 'batch_size'], but got 'z7The value of 'addition_info' should be 'int', but got ')r?dict TypeErrortypeitems ValueErrorr@) addition_infoadd_infoitemvalues r*_update_addition_inforsQa8H   t , , ]+,B 0  )..0KD;; BBFrKeS))  $U }B0#TN1r,cDU(dU$[U[5(akUHcn[U[5(d[S[ U5S35e[ R RU5(aMV[SUS35e U$[S[ U5S35e)z!Validity check of input file pathz0The type of file path should be 'str', but got 'rzThe file path 'z' does not exist.z1The type of file path should be 'list', but got ') r?r/rXrrospathexistsr) file_pathfiles r*_check_valid_pathr s  It $ $DdC(( $T |2/77>>$'' ?4&8I!JKK Y( ,  r,cU(d [S5e[U[5(d[S[ U5S35eUR 5Hsup[U[ 5(d[S[ U5S35e[U[RR5(aM][S[ U5S35e U$)Nz'param_dict' cannot be None.z4The type of 'param_dict' should be 'dict', but got 'rz:The type of key of 'param_dict' should be 'str', but got 'zDThe type of value of 'param_dict' should be 'DenseTensor', but got ') rr?rrrrrXpaddler DenseTensor) param_dictr&rs r*_check_param_dictr s 788  D ) ) Z() -  &++-KDdC(( $T |2/eV[[%<%<== $U }B0.r,cU(dU$[U[5(d[S[U5S35eUR 5Hup[U[ 5(d[S[U5S35e[U[5(d[S[U5S35e/SQn[ UR55U:wdM[SUR5S35e U$) Nz3The type of 'dist_attr' should be 'dict', but got 'rz@The type of param name of 'dist_attr' should be 'str', but got 'z=The type of distributed attribute should be 'dict', but got '' process_shape process_grouprIrOziThe key of distributed attribute should be '['process_shape', 'process_group', 'dims_mapping']', but got .) r?rrrrrXr/keysr)rr&rattrs r*_check_dist_attrr7s   4 ( ( Y( ,  %??,KDdC(( $T |2/eT** $U }A/D EJJL!T) $zz|nA/%-.r,cSSKJn [U[RR 5(de[U[ 5(deUcU"5n[U5nU(d[XU5 [XU5 g[S5e)a Save model parameter state, optimizer state, distributed attribute and additional information of each rank. Args: program(Program): The program to be saved. checkpoint_path(str): The path of the checkpoint file to be saved. dist_attr_path(str): The path of distributed attribute file to be saved. addition_info(dict, optional): Additional information, key should be selected in ['epoch', 'batch', 'batch_size']. Default values are 0, when 'addition_info' is None. Default: None. is_integrated(bool, optional): Whether to integrate param before save. Default: False. dist_context(DistributedContext ,optional): collect related distributed information for program Returns: None Examples: .. code-block:: python >>> import os >>> from paddle.distributed.auto_parallel.static.utils import save_distributed_checkpoint >>> step = 16000 >>> global_batch_size = 32 >>> path = os.path.join("./output", "step_%d" % step) >>> os.makedirs(path, exist_ok=True) >>> program = paddle.static.Program() >>> add_info = {'batch': step, "batch_size": global_batch_size} >>> save_distributed_checkpoint(program, path, path, add_info) rrwNz/Integrating parameter has not been implemented.) ryrxr?rstaticProgramboolr_save_distributed_state_dict_save_distributed_attributeNotImplementedError)rcheckpoint_pathdist_attr_pathr is_integratedryrxs r*save_distributed_checkpointrZsyP> gv}}44 5 55 5 mT * ** *68 )-8M $W_M#G\J" =  r,c[U5(dS5e[U5(dS5e[U5n[U5nUSnUSnXCU4$)ab Load parameter, optimizer, distributed attribute and addition_info. Args: checkpoint_path(list[str]): model parameter file path, must be in order of rank id. dist_attr_path(list[str]): distributed attribute file path, must be in order of rank id. Returns: param_dict(dict): parameters' value of all ranks. dist_attr(dict): parameters' distributed attribute. addition_info(dict): additional information user saved in last training. Notes: The return, 'addition_info', is belonging to the first file of checkpoint_path by default. Examples: .. code-block:: python >>> # doctest: +SKIP('Depends on external files.') >>> from paddle.distributed.auto_parallel.static.utils import load_distributed_checkpoint >>> ckpt_path = [ ... './model_state_rank0.pdmodel', ... './model_state_rank1.pdmodel', ... ] >>> dist_attr_path = [ ... './dist_attr_rank0.pdattr', ... './dist_attr_rank1.pdattr', ... ] >>> param_dict, dist_attr, add_info = load_distributed_checkpoint(ckpt_path, dist_attr_path) !'checkpoint_path' cannot be None. 'dist_attr_path' cannot be None.modelr)r_load_distributed_state_dict_load_distributed_attribute)rrstate_dict_inforrrs r*load_distributed_checkpointrsl@ _ - -+ - ^ , ,P.PP ,2?CO+N;I )J#O4M - //r,c\SSKJn [U[RR 5(de[ U5(dS5e[ U5(dS5eUcU"5n[U5n[U5n[X#5nUSnUSn [XU5n [X5 U $)a Load parameter, optimizer, distributed attribute and addition_info into model. Args: checkpoint_path(list[str]): model parameter file path, must be in order of rank id. dist_attr_path(list[str]): distributed attribute file path, must be in order of rank id. program(Program): the program to be updated with checkpoint_path. dist_context(DistributedContext ,optional): collect related distributed information for program Returns: addition_info(dict): user saved in last train. Notes: The return, 'addition_info', is belonging to the first file of checkpoint_path by default. Examples: .. code-block:: python >>> # doctest: +SKIP('Depends on external files.') >>> from paddle.distributed.auto_parallel.static.utils import load_checkpoint_into_program >>> exe.run(startup_program) >>> ckpt_path = [ ... './model_state_rank0.pdmodel', ... './model_state_rank1.pdmodel', ... ] >>> dist_attr_path = [ ... './dist_attr_rank0.pdattr', ... './dist_attr_rank1.pdattr', ... ] >>> load_checkpoint_into_program(ckpt_path, dist_attr_path, main_program) rrwrrrr) ryrxr?rrrrrr get_dist_attrmerge_and_slice_parameterload_parameter_into_program) rrrryrxall_state_dict_infoall_pre_dist_attrall_cur_dist_attrall_param_dictrsliced_param_dicts r*load_checkpoint_into_programrsF> gv}}44 5 55 5 _ - -+ - ^ , ,P.PP ,68 6G3NC%g<(1N'8M1+< 1; r,c[U[5(deU(a)[U[RR5(deU(dgUR U5 g)z Load parameters into program. Args: param_dict(dict): parameters' name and value. program(Program): the program to be updated N)r?rrrrset_state_dict)rrs r*rrsJ j$ ' '' ' z'6==+@+@AAA A  :&r,c<[RR5n[RR USUS35n[ X5[RR5S.n[R"XT5 [R"SUS35 g)z,Save distributed attribute of all parametersdist_attr_rankz.pdattr)r world_sizez(Already saved distributed attribute to 'rN) r distributedget_rankrrjoinrget_world_sizesaverinfo)rrryrank_iddist_attr_namedist_attr_dicts r*rrs  ))+GWW\\. 9Nw5((779N KK/ LL;N;K2NOr,c0nUH[n[R"U5nUSnU[U5:XdS5eUSR5HupVXQ;dM XaU'M M] U$)z:Load parameters' distributed attribute from dist_attr_pathr zMThe number of 'dist_attr_path' must be equal to the last training world size.r)rloadr.r)rtotal_dist_attrdist_attr_filerpre_world_sizer&rs r*rrsxO(KK/ "<0^!44 [ 4$G,224JD*(,%5 ) r,cH[RR5n[RR USUS35nUR 5[RR5US.n[R"XT5 [R"SUS35 g)zSave parameters' state_dictmodel_state_rankz.pdmodel)rr rzAlready saved model to 'rN) rrrrrr state_dictrrrr)rrrrckpt_file_namers r*rr's    & & (DWW\\+D6:N##%((779&J  KK + LL+O+$>!-!7!7 ' #P ' B'6==#8#89999  :)r?rndarray)rAvs r*rC,merge_and_slice_parameter..s!2 /4!Jq"** % %us')zoThe value of 'dist_param_dict' is parameter's value of all ranks, and its type should be 'list(numpy.ndarray)'.z$Start to merge and slice parameters.rrIrr4rz Parameters 'z)' are not found in last training process.z,' are not found in current training process.)rr?rrrrXrr/rErrrrNrrrr0r.set_merge_parameter_with_dist_attr_slice_parameter_with_dist_attrpopwarningswarn)dist_param_dict pre_dist_attr cur_dist_attrr&rparam_not_in_preparam_not_in_curr0pre_attrcur_attrrr0param pre_parampre_dims_mappingcur_dims_mappingcomplete_param sliced_params r*rrs M * *M,MM * ot , , B4CXBYYZ[ ,',,. $$$99=d AG %&&c2 /42 / / @ /  LL78!&&(  (  # #H -  * *  ((113G_-33G  ))+G0l=%8L r,c*SSKJn [U5S:Xa<Sn[USS5HupgUSS:wd USX6:wdMSn O U(agU(dUR X45 gSnU[U5:aUR XSU5un n n U S:wa_U S:Xa[ R"XSU4U S9n O[ R"XUS4U S9n URU5 [UU U U5 gUS- nU[U5:aMgg) a Merge partial parameters to a complete one. Returns: None Examples: .. code-block:: python >>> import numpy as np >>> from paddle.distributed.auto_parallel.static.utils import _merge_parameter >>> partition_param_list = [(np.array([[[1.11, 1.12]]]), [[0, 1],[0, 1],[0, 2]])] >>> param = np.array([[[1.13, 1.14]]]) >>> partition_index = [[0, 1],[0, 1],[2, 4]] >>> complete_shape = [2, 2, 4] >>> _merge_parameter(partition_param_list, param, partition_index, complete_shape) >>> print(partition_param_list) [(array([[[1.11, 1.12, 1.13, 1.14]]]), [[0, 1],[0, 1],[0, 4]])] rrJTrFNr4r) rLrKr.r"rNcompute_concat_infor concatenater9rO) rRrCrUrQrKis_complete_datarrrK concat_axis first_order new_partition new_params r*rOrOsI2#  A%"#7#:1#=>ICAw!|tAw.*==#( ?   ##U$<= #*++ --$'*O  b !# "-03U;+!I!#Q 7 :;+!I%((+ (!"   FA5#*++r,cUcUR5nURGHnUR5H n[XS9 M UR 5[ ;aM>UR bMM/n/n/nUR5HnUR 5nUcPUR[R"55 Sn UR5R n U b U Rn OURU5 URn U cMX;dMURU 5 M UR5Hzn U R 5nUc&UR[R"55 M<URU5 URU;dM_URUR5 M| [U5S:dGM[U5S:XaUSn O [U5n [R "U UU5UlGM g)N)rrr)r)rr{_complete_op_dist_attrr&partition_skip_op_listroperands_sourcerNr Attributeget_defining_oprJresultsr.r create_op_dist_attribute) rrr sub_blockmeshes operand_attrs result_attrsoperandtmp_attr value_meshtmp_op_dist_attrr,rs r*rgrgYs }$$&iiI "7 <% 779. .  << FML--/",,.#!((9!%J'.'>'>'@'J'J$'3%5%B%B !((2!)!6!6J)j.FMM*-0**,!++-# '' 8 ''1,,F: h&;&;<'6{Qv;!#!!9D/7D";;!   Kr,c /n[UR5U- n[R"XUUS9nUS:XaU$UH!nUR [ XaUS- 55 M# U$)a Slice a complete parameter. Returns: sliced_param_list(list): sliced parameters with 'partition_index_list' Examples: .. code-block:: python >>> import numpy as np >>> from paddle.distributed.auto_parallel.static.utils import _slice_parameter >>> complete_param = np.array([[[1.11, 1.12, 1.13, 1.14, 1.15, 1.16]]]) >>> rank = 2 >>> complete_shape = [1, 1, 6] >>> dims_mapping = [-1, -1, 0] >>> process_shape = [3] >>> process_group = [0, 1, 2] >>> sliced_param_list = _slice_parameter(complete_param, [[], [], [2, 4]], 3) >>> print(sliced_param_list) [array([[[1.11, 1.12]]]), array([[[1.13, 1.14]]]), array([[[1.15, 1.16]]])] r^r)r.rGrsplitextendrX)rGrZrbr[rrHrCs r*rXrXst2 ~## $v -D88T2L{  U&1* E  r,cSSKJn URXX#U5nSn[U5H>upX(S:XaU n O XX(-n U S:XaXhSn O XhSS-U -n XyU --U -nM@ U$)a Get sliced_param's index of current rank in all sliced parameters list. Returns: sliced_param_index(int): the index of sliced param in sliced_param_list Examples: .. code-block:: python >>> import numpy as np >>> from paddle.distributed.auto_parallel.static.utils import _get_sliced_param_index >>> complete_param = np.array([[[1.11, 1.12, 1.13, 1.14, 1.15, 1.16]]]) >>> rank = 2 >>> complete_shape = [1, 1, 6] >>> dims_mapping = [-1, -1, 0] >>> process_shape = [3] >>> process_group = [0, 1, 2] >>> slice_param = _slice_parameter(complete_param, [[], [], [2, 4]], 3) >>> print(slice_param) [array([[[1.11, 1.12]]]), array([[[1.13, 1.14]]]), array([[[1.15, 1.16]]])] >>> index = _get_sliced_param_index(rank, complete_shape, dims_mapping, ... process_shape, process_group) >>> print(index) 2 rrJrr4)rLrKrNr") rrQrIrrrKrUr\rKrG slice_shaper0s r*rYrYs>#77 l=On- ?b K#AAK ! #&q)E$'*Q.;>E/K3GH5P. r,c2SSKJn /nUHQnURX`XU5nU(a2[[ U55HnXXR Xx5 M MOUnMS [ [SUU55nUV s/sHn [U 5PM nn U$s sn f)a Get split indices of every dimension. Returns: split_indices_list(list): the split indices of every dimension of the parameter Examples: .. code-block:: python >>> import numpy as np >>> from paddle.distributed.auto_parallel.static.utils import _get_split_indices >>> complete_param = np.array([[[1.11, 1.12, 1.13, 1.14, 1.15, 1.16]]]) >>> complete_shape = [1, 1, 6] >>> dims_mapping = [-1, -1, 0] >>> process_shape = [3] >>> process_group = [0, 1, 2] >>> index = _get_split_indices(complete_shape, dims_mapping, process_shape, process_group) >>> print(index) [[], [], [2, 4]] rrJc:[[U5U1- S1- 5$r)r/r6xys r*$_get_split_indices..sc!fslaS01r,) rLrKrNrFr.rxr/mapr) rQrIrrrKsplit_indices_listrTrUdimr~s r*rWrWs2# #;; \-  S12"'../CD3"1 ! 1   .@@-?&)-?@ As;Bc[URS55n[UR;=(a? U[[R 5:H=(d U[[R 5:H$)Nop_role)r@r OP_ROLE_KEY attr_namesOpRoleForwardLossrrs r* is_forward_oprsO"'')$%G "-- ' 3v~~&&E'S5E*Er,c[UR;=(a: [UR5[5[[R 5-$r>)rrr@ all_attrsrBackwardrs r*is_backward_opr> "-- ' C {#- FOO-r,c[UR;=(a: [UR5[5[[R 5-$r>)rrr@rrOptimizers r*is_optimize_oprrr,c[UR;=(aD [UR5[5[[R R 5-$r>)rrr@rrrLRSchedrs r*is_lr_sched_opr$sD "-- ' %C {#- FOO # #$-%%r,c[UR;=(aT [UR5[5[[R 5[[R 5-:H$r>)rrr@rrrrrs r* is_loss_opr*sL "-- ' 2C {#- fnn FKK 0 0-22r,c[UR;ag[UR5[5nU[[R 5-=(a U[[R 5-$)NF)rrr@rrrrrs r*is_loss_grad_opr0sM"--'",,.-.G S) ) HgFKK8H.HHr,cURRS5=(a* URRS5RS5$)N op_namescopez/gradient_clip)rr*r startswithrs r*is_gradient_clip_opr7s; 77  N + # 1j!"#r,cURRS5=(a SURRS5;$)Nr/auto_parallel/reshard)rr*rrs r* is_reshard_opr=s8 77    C "bggll>&B BCr,c8URRS5$)N_p)rendswithrs r* is_prim_oprCs 77  D !!r,c$URS5$)Nring_id)r*rs r* is_comm_oprGs ;;y !!r,c/nURHTn[U5(dM[URR 55S:XdS5eUR U5 MV [U5S:XdS5eUS$)Nrz#loss op should only output loss varz"num of loss op is not equal to oner)rrr.routput_arg_namesrN)rloss_opsrs r* get_loss_oprKszHii b>>rww//12a7 5 7 OOB   x=A CCC  A;r,c [5nX%l[U[[R 45(a[ U5UlO?[U[R 5(aX5lO[US[U535eURS5(a"URS5 URS5 URS5(a USUl URX5 U$)Nz8 must be a instance of ProcessMesh or list, but receive mark_annotatedrIrJchunk_id)rrIr?r/rr3r rJrrrgetrrr)ryrrIrJkwargsrs r*set_var_dist_attrrXs%'$0!,rzz 233(3L(A% L$"2"2 3 3(4%nTUYZfUgTh i  zz"##''7''7 zz*$*:$6!11#H r,c bUceUce[5nURR5HnURXb5 M URR 5HnUR Xr5 M XlURS5(a USUlURX5 g)Nr) rrinput_arg_namesset_input_dims_mappingrset_output_dims_mappingrJrrset_op_dist_attr_for_program)new_oprJ ref_mappingctxrnew_op_dist_attr input_varnameoutput_varnames r*6naive_set_dist_op_attr_for_program_by_mesh_and_mappingrms  ## #  "" "')446 // K7 ++66800M9%1! zz*$*:$6!$$V>r,c 0Uce[5nURR5HJnURR U5nUR U5R nURXW5 ML URR5HJnURR U5nUR U5R nURX5 ML Xl SU;a USUl SU;a USUl URX5 g)N is_recomputer)rrrrrrrIrrrrJrrr) rrJrrrrrr7rs r**naive_set_dist_op_attr_for_program_by_meshrs  ## #')446 ll}-66s;HH// G7!++668ll~.66s;HH00I9 %1!(.~(>%V$*:$6!$$V>r,c SnURnURRnUR5S:XdUR5S:XagUR 5n/nSU;aUR S5n/nUR 5HnURU5nUR(aM'URU5n [U 5S:a;[U SS5H)upU S:XaM UR5SU SU S 35e [U 5S:dMURU S 5 M UR5GH1nURU5nUR(aM(URU5n Xu;aq[U 5S:a;[U SS5H)upU S:XaM UR5SU SU S 35e [U 5S:aURU S 5 MMU S S:XdUR5S W S 35e[U 5S :a;[U S S5H)upU S:XaM UR5S U SU S 35e URU S5 GM4 [!U5n U cS5eUR 5HYnURU5nUR(aM'URU5n [U 5S:dMIXS :wdMSXS 'SnM[ UR5HpnURU5nUR(aM'URU5n Xu;a#[U 5S:aXS :waXS 'SnM\M^M`XS:wdMjXS'SnMr U$)NFrGsliceXShaperr4zD only the batch dimension (0-dim) can be sharded, but the dimension z is sharded by z part.rz^ only the batch dimension (1-dim) of XShape can be sharded, but the dimension 0 is sharded by r zN only the batch dimension (1-dim) of XShape can be sharded, but the dimension #There is no compatible dim mapping.T)r serial_oprr output_namesoutputrget_serial_inputr get_input_dims_mappingr.r"rNrget_serial_outputget_output_dims_mappingr^) rrmrop_descrxshape_arg_namesbatch_dim_mappingsarg_name serial_tensorrIrr7rds r*+update_op_dims_mapping_by_default_dist_implrsG$$L$$G||~ GLLNg$='')L<">>(3++-00:  % % #::8D | q ),qr*: ; "}||~&&jknjoo~@GHHNO}!< |  !  % %l1o 6.,,.11(;  % % #;;HE  +< 1$$-l12.>$?LC"b="<<>**nornstCDKCLLRS=%@< A%")),q/:& ?b( <<>"#ABIAJJPQ (< 1$$-l12.>$?LC"b="<<>**xy|x}~MNUMVV\]=%@  % %l1o 6-/0<  $ $%5 6  $ $%<%F G$//1$#;;HE < $8888  .% >>OP " .- .$ " ,/C C!"9"CD DqD  2<=(8+LL'>g&F # >  8#DD33HOE'*A*KK335%$(%#;;HE "l 2  0 01 G % Na I, s  H$8 H)cSSKJn URn[R"U5n/nUc[ R R5O[URS55n[U5HMn[R"U5n U"5U l URX5un n n n n URU 5 MO U$)z2Get all distributed main programs by dist_context.r)DistributedOperatorContextGPU)ryrclustercopydeepcopyrrrr.get_all_devicesrF_dist_op_context_get_dist_programrN) serial_program_infory parallelizerrrcopied_parallelizerall_dist_main_programranksrused_dist_contextrdist_startup_programdist_main_programs r* get_all_distributed_main_programr-s9!))G-- 5 ? ))+ ((/ 0 < MM,7-G-I* 1 1' M   $$%67  ! r,cp\rSrSrS Sjr\S5r\S5r\S5r\S5r \S5r S r g) SerialProgramInfoiJNc@XlX lX0lX@lXPlgr>)_train_program_startup_program_loss _optimizer_cluster)self train_programstartup_programloss optimizerrs r*__init__SerialProgramInfo.__init__Ks, / # r,cUR$r>)rrs r*rSerialProgramInfo.train_programTs"""r,cUR$r>)rrs r*r!SerialProgramInfo.startup_programXs$$$r,cUR$r>)rrs r*rSerialProgramInfo.loss\s zzr,cUR$r>)rrs r*rSerialProgramInfo.optimizer`s r,cUR$r>)rrs r*rSerialProgramInfo.clusterds }}r,)rrrrrr>) __name__ __module__ __qualname____firstlineno__rpropertyrrrrr__static_attributes__r5r,r*rrJsnGK ##%%r,rcSnSSKJn UR5nUR5 SnSSSSSS S S .n/n/S QnUGHn0n UR 5R n UR 5R GHn Sn U RU;aXU RR5'M4U R(a$[XRSR5OS n [U RS55[[R 5:Xa{SU R;ajU RSSnXR#5;aX^nUR%USU S9nU(a U"XU 5n OUR%XS9nU(a SU"XU 5-n O[U RS55[[R&5:XaYU RUR#5;aX[RO U RnUR%U5nU(a U"XU 5n XU RR5'GM UR)U 5 GM U$)Nc(Sn[US5nUSnSnSnURS5nSnUGH5n SU ;aSOSnX;dMU SU RU5S- n U RS5n U RS 5n U S:a U S:aX:dS 5eU U S-U RS 5n U Vs/sHn[UR 55PM n nSnU[ S U S5- nUR S :Xa U S:XaSOSn URHRnUR5U :XdMURU5H"nUUnU[ SUR5- nM$ GM3 GM8 US:aUS:dS5eXe- U-nU$! Us$=fs snf)Nrop_timeconfig z (Variable)z(listr[]zGet shape failed.,c X-$r>r5r}s r*rDget_standalone_cost_data.._compute_runtime..squr, c_embeddingweightwidsc X-$r>r5r}s r*rrsQUr,zGet input size failed.) floatrwfindr@rtrr input_nameslowerinputrG)op_costrr|runtime op_configtotal_static_input_sizetotal_actual_input_size parsed_infovariablerarg_name_lowershape_left_boundaryshape_right_boundaryrGr~ dtype_factorrr0ractual_runtimes r*_compute_runtime2get_standalone_cost_data.._compute_runtimejs GI./GH% "#"#ood+ D , 4 :K !%&? ((;a(?!@&*iin#'+yy~$'!+,q0,B'' 'C'!+.B%*277AQWWY7 '62DeQ+OO'77m+-9u#!#H~~'>9(*(:H"&x.C3v 2CII83); !// >'*/F/J $ J $ = G [ N,8sF1#FF rr embeddingmatmul transposereshape unsqueezerdivide)r matmul_v2 transpose2reshape2 unsqueeze2rr)create_py_readercreate_double_buffer_readerreadassignfloat32r_gradF)forwarddtype)rE)paddle.cost_model cost_model CostModelstatic_cost_datar)r|rrridrrXrEr@rrrrget_static_op_timerrN)distributed_programsr1cmrGDEFAULT_MULTIPLEOP_NAME_MAPPINGstandalone_cost_data not_enum_opsdistributed_program cost_datar|rr&rEforward_op_namer%op_names r*get_standalone_cost_datarVis91f#J!"!!#OL 4 "//166%22488BGww,&*1"''**,'%%D++A./556  2779%&#foo*>>bgg%&(ggcrlO&*>*>*@@*9*J(;;'e<G"27"E","?"?+#@##&'*:7*M&MGRWWY'(C,??ww/"6"6"88$GG, %77@.wDAG&-bggjjl #I9L ##I.S 4V r,cUR5nUR5nX2R;aURU5 gXBR;aURU5 g[ S5e)Nz6Cannot find the original id in the distributed context)rJ original_id_dist_ops_for_programset_original_idAssertionError) dist_op_descrryop_idop_original_ids r*set_dist_op_desc_original_idr_sg JJLE((*N 222$$U+ == =$$^4 D  r,c^UcU$[U[[45(a [U5$U/$r>)r?r/tuple)rs r*to_listrbs. } %$''E{ 7Nr,c[RRXS[RR 535n[ US5nUR[U55 SSS5 g!,(df  g=f)Nz _program.r) rrrrrropenwriterX)rrr&filenamefs r* debug_programrhs\ww|| i 2 2 ; ; =>?H h  G    s A11 A?cTSSKJn U"5HnURU:XdMUs $ g)Nr)get_all_process_groups)rrjrJ)rrjgs r*ring_id_to_process_grouprls'5 # % 447?H& r,cSS/nURH2nURHnUHnXCR;dM g M! M4 g)N _grad_grad triple_gradTF)r{rr)rhigher_order_op_suffixrrsuffixs r*find_higher_order_backward_oprr sE*M:))B0WW$1 r,c[U[5(deSUR;de[SURS5$)zA input: - var: variable return: number of element in var r4c X-$r>r5r}s r*rget_var_numel..!squr,r)r?r rGr)rs r* get_var_numelrvs= c8 $ $$ $ SYY   $cii 33r,cl[U[RR5(aUR 5$[U[R R5(a;[UR [5(a UR $UR 5$[S[U5S35e)Nzg'optimizer' must be object of class `paddle.optimizer.Optimizer` or `paddle.static.Optimizer`, but got r) r?rr Optimizerget_lrr_learning_rater rrrs r*ryry$s)V--7788!! Iv}}66 7 7 i.. 6 6++ +++- - 66:9o5Fa I  r,cSSKnSSKJn /nSnU"5nURR S5upx[ U5U-n Sn Sn URUR UR5n U RXy45 U RS5 0n UGH<n XR;aM[U R5S:XGaU RRU5nUS:XaS OS nU(Ga U RS nURUR S5unn[ U5U-nURUR UR5nURUU45 UR[!U5R#S 55 UR%U 5R'S 5n[ U5nUU:wa[)S USUS35e[+SU RS35 UR-5 OU RSnUU;aRU R/5unn[ UR%U 5R'55nUU U'UR1U5 OYU UR[!U5R#S 55 U UR-5 [+SU RS35 OMU R35 GM? U R-5 g)Nr)_get_global_envi:i r TFrzutf-8z0Please check comm pair, the recv rank should be z but got rzIt is able to instantiate z as sender now.z as receiver now.)socket collectiver~current_endpointrwr@AF_INET SOCK_STREAMbindlistenrr.r0trainer_endpointsconnectsendrXencoderecvdecoderrcloseacceptrN instantiate)all_process_groupscur_rankrr~has_recv_by_socket magic_numgenv cur_rank_ip cur_rank_portcur_rank_recv_port server_socket buff_sizeclient_socketsrr0is_send recv_rank recv_rank_iprecv_rank_port connect_port client_socketr send_rank recv_addrs r*initialize_pg_in_full_moder3s-I  D!%!6!6!2Y> & NNF$6$6! %%|\&BC""3x=#7#7#@A$)))4;;GD4y9$$J9+U^_c^ddef4]5H5H4IY##%)//2  (::3@3G3G3I0 y"=#5#5i#@#G#G#IJ/<t,*11$7&y166M009'y177989L9L8MM^_ !!#[,\r,cURS5=(a/ SURS5;=(a SURS5;$)Nrz/auto_parallel/rc exclude_rcr*rrs r*is_recompute_oprws? N# 8 277>#: : 8  7 7r,cZURS5=(a SURS5;$)Nrrrrs r*is_recompute_exclude_oprs- ;;~ & <277<,r,c :SSKJn U(dgURnUR(dg/n[ U[ R R5(a[US5(akURRS;aQ[URS5(a6URRn[U5S:aUR5 OURnO URnU(dgUR5nU"XwR 5nUR#5 UR%U5n /n Sn Sn U S-[U 5:aU S:XaeXS-n XR&;aU S- n M5UR&U S nU(a.[)U5S :aU R+S [)U5S-/5 OoUR-X/XS-/5unnnU(a)UR/UU 5nU R+UUS-/5 O [0R2"S US US-S 35 U S- n U S-[U 5:aM[5U 5HJunn[7US US5H.nUR UR9SS[;U5-5 M0 ML g)Nr})RecomputeStategpt)GPTForPretrainingGPTForPretrainingAuto checkpointsr r4rvar_as_output_opsrzCould not recompute op range [z] - [z] rz/auto_parallel/rc_)passes.auto_parallel_recomputer recomputeenabler?rnnLayerhasattr __class__r rrr.r9r)r build_statssort_checkpoints var_op_depsmaxrN is_subgraph_update_segment_startrdebugr"rF _set_attrrX)rlossesstrategyrrrckptsrrc_statersegments start_idxpre_segment_end_idx ckpt_name op_idx_listflagmin_idxmax_idxrKsegmentjs r*set_recompute_segmentsrsk@ ""I    E%)) E5 ! !((  =11II))E5zA~ ))E%%   "EeYY/H ++E2KHI a-#k* * ?#M2I 4 44Q "..y9:MNKs;/!3C $4q$8 9:%-%9%9'(;1}+E*F& "D'7"880'A+ 67 4WIU7Q;-rR Q - a-#k* *0 ) 7wqz71:.A IIaL " " 4s1v = /*r,cJURU5nURnURnXR;a [ X$U5nOUnUSnUS:aQUR US:a>[ URUR UU5n[U5URU54$g)Nrr4r)rr) rrJrIrrrGrr.r0) rrryrrJrIrbatch_size_axis group_rankss r*get_input_split_infors#DDSI#00L#00L///),hO"1oO 2 2? Ca G%  $ $       ;!2!27!;;; r,cUbcSUlSUlUR(aD[UR[R R 5(aSURlU$)NT)_parameter_list _param_groups _grad_clipr?rrClipGradByGlobalNorm _async_add_nr{s r* validate_optrsX$( !"&    J  &))"@"@% % 15I - r,cSSKJnJn SSKJn U"5R nU"US/5n[ U5S:aSOS/[[ UR5S- 5Vs/sHnSPM sn-nU"XU5$s snf)Nr )r shard_tensorr)get_world_process_groupdp) interfacer rrrrr.rFrG)r~r rr world_ranksrJrrPs r*set_data_parallelrs}56)+11K{TF3Lk*Q.$D9CL1,-=--=J  44 =s A:cUR(dgURR5RVs/sHnURPM nn[ U5[ [ 5-(dUR(aggs snfrk) data_parallel_original_serial_main_programr)rrr6__not_naive_data_parallel_op__)ryrops_types r*is_naive_data_parallelrsq  % %<<IIKOOOB O  MC >? ?  $ $ sBc DURnUbj[R"URUR[ [ UR55Vs/sHnS[U5-PM sn5UlURUlURUl gs snfNrB) rJrr rGrrFr.rXrI annotated) cpp_dist_attr py_dist_attrpy_process_meshrKs r*_copy_tensor_dist_attr_to_cpprs"//O"%)%5%5  ! !  ' '#(_-B-B)C#D E#DaS3q6\#D E& " ".!:!:M*44M FsB cSSKJn URnUb U"URURS9UlURUlUR UlgNr )r )rGr)rJr rGrrIr)rrr cpp_process_meshs r*_copy_tensor_dist_attr_from_cpprsT*$11#$/"(((44% !!. : :L*44Lr,c URnUbj[R"URUR[ [ UR55Vs/sHnS[U5-PM sn5UlURUlURUl URUl URUl URR5H!upEURU5n[Xe5 M# UR R5H!upEUR#U5n[Xe5 M# gs snfr)rJrr rGrrFr.rX impl_typeimpl_idxrrinputs_dist_attrsrget_input_dist_attrroutputs_dist_attrsget_output_dist_attr)rrrrKr&py_tensor_dist_attrcpp_tensor_dist_attrs r*_copy_op_dist_attr_to_cppr)s"//O"%)%5%5  ! !  ' '#(_-B-B)C#D E#DaS3q6\#D E& " +44M)22M!-!:!:M*44M%1%C%C%I%I%K!,@@F%&:P&L&2%D%D%J%J%L!,AA$G%&:P&M FsD= cSSKJn URnUb U"URURS9UlURUlUR UlUR UlURUlURR5H!upEURU5n[XV5 M# URR5H!upEURU5n[XV5 M# gr)rJr rGrrrrrrrrrrr)rrr rr&rrs r*_copy_op_dist_attr_from_cppr=s*$11#$/"(((44% !+44L)22L - : :L*44L&3&E&E&K&K&M"*>>tD'  'N '4&F&F&L&L&N"*??E'  'Or,c0URR5H-n[URRUR5 M/ UR R5H-n[ URRUR5 M/ gr>)_dist_tensors_for_programr}rrrrYrrryrrs r*_copy_dist_attr_to_cpprVsu#==DDF %  % % / /1F1F G  55<<>!    ' '):): ?r,c0URR5H-n[URRUR5 M/ UR R5H-n[ URRUR5 M/ gr>)rr}rrrrYrrrs r*_copy_dist_attr_from_cpprbsu#==DDF '  % % / /1F1F G  55<<>#    ' '):): ?r,cURHnUR5(aGUR5b6URU5nUR5Rn[ X25 UR 5(dMvUR5cMURU5nUR5Rn[X25 M gr>) serial_ordered_nodesis_varrget_tensor_dist_attr_for_graphrris_oprget_op_dist_attr_for_graphrrynoderrs r* _copy_dist_attr_to_cpp_for_graphr ns11 ;;==TXXZ3'FFtLL HHJ00M )- F ::<) rrrrrrr rr rr s r*"_copy_dist_attr_from_cpp_for_graphrzs11 ;;==TXXZ3'FFtLL HHJ00M +M H ::<._select_best_depend_var..sAaDr,keyr4)r r.rvsort)r|rvars_with_numelss r*_select_best_depend_var@insert_dependencies_for_two_ops.._select_best_depend_varsr#QRS 1!==l">>l  * ? gVdUeefg *'(%-%>%>?%>T4%>?I)%1%A%AB%AT4%ABJ (  "! !   @ Cs 5D(&D-c U (a[5(ag[U[5(aU/n[U[5(aU/nUH%n URU R5(aM%e UH%n URU R5(aM%e UR US5nUc UR nUceSn U (aURUSSU0SU0S9nOURUSUUS .SU0S9nUR[U5 U(dUS /:wGa[5nSUl S Ul UUlUUl URUlURR!5HAnUR#U5nUR U5R$nUR'UU5 MC URR)5HAnUR#U5nUR U5R$nUR+UU5 MC UR-UU5 U bURS S U 35 U(aUR/5 U$)z[ dependency: op that generates prior_vars should be run before op that generates post_vars NrTnopXOut)rinputsoutputsdepend)r(Depr4defaultr/)rr?r has_varr&rrJ_insert_op_without_syncrrrrrrrrrrrIrrrr_sync_with_cpp)rr prior_vars post_varsryoprolerJrrrrskip_insert_when_sequential_run prior_varpost_varpost_dist_attr depend_opdepend_op_dist_attrrrr7rs r*rrsj$'+<+>+>*h'' \ )X&&K  }}Y^^,,,, }}X]]++++"BB9Q<PN%22  ## #G11 ZI& 2 11 !I&2  V,|t+.0'($(1%+7(+7('5'>'>$&^^;;=M))M*C"CCl  6 6}g N > (nn==?N))N+C"CCl  7 7 P @ 11 * Na ~,>?   r,c&SUR;agg)Nc_TF)rrs r*is_dep_skip_opr> s rww r,c^U4SjnU$)Nc>[RR5(aT"U0UD6$[RRR 5 T"U0UD6sSSS5 $!,(df  g=fr>)r frameworkin_dynamic_moderdygraphguard)argsrfuncs r*__impl__!_dygraph_guard_..__impl__( sW    + + - -(( ($$**,T,V,-,,s A'' A5r5)rFrGs` r*_dygraph_guard_rI' s- Or,cF[[R"S5S5$)N!FLAGS_new_executor_sequential_run)rr get_flagsr5r,r*rr5 s& <= /  r,c[UR5S:aS/4$[UR5n[U5U4$)Nr r)r.rget_sub_process_mesh)rysub_process_meshess r* get_pp_degreerP= sA < & &'!+"u -l.I.IJ ! "$6 66r,c@UR5Rn/n[U5H\up4SUR5;dMUR(dM.URR nXR;dMKUR U5 M^ [U5n[USS9nU$)Npd_opc URS$r)rrs r*r1get_sub_process_mesh_by_program..Q s !--*:r,r) r)rr"r&rrJrNrNr) dist_programall_opsrrrrJrOs r*get_sub_process_mesh_by_programrWE s'')--GNW% bggi BLLL<<44L1%%l3 & .n= : r,c[5n[R"U5nUHnU[UR5-nM /nSn[ U5H`upc[ [UR55[ U5:XaUR U5 MC[UR5U:dM^SnMb U(a#[U5HnURU5 M U$rk) r6rrrr"r.rNrr9)rrrOpm global_pm_idx has_sub_pmrs r*rNrNW s%K~6 s2>>** !MJ/0 s2>>" #s;'7 7   %  ; .J 1 M*C  " "3 '+ r,cnSn[UR5Hup4XR;dMUn U$ U$r>)r"rr)ryrpp_idxrrJs r* get_pp_stager^m s@ F&|'B'BC ++ +F  M D Mr,cT[RR5n[X5$r>)rrrget_pp_stage_by_rank) pp_degreers r*get_pp_stage_by_pp_degreerbv s!!!**,H  44r,cSnURH,n[X15nUbXB:wa gXB:Xd SUSU35eUnM. U$)Nz;Can't get pp_stage by process_mesh with different pp_stage z and )rr`)rJrapp_stage_for_process_meshrpp_stages r*get_pp_stage_by_process_meshrf{ sg $(('8 $ 048 MhZW\]v\wx 8%-!) %$r,cR[RR5nX!-nX-nU$r>)rrr)rra word_size pp_group_sizeres r*r`r` s+""113I*M$H Or,r"rrc/n/n0nURnUHanURRU5n URR U5n U R n [ X5n URU 5 Mc UHanURRU5n URR U5n U R n [ X5n URU 5 Mc UH n URRU 5Xm'M" XEU4$)a Get data used in inferring distributed attributes, including: 1. DistTensorSpec for each input and output tensor of this dist_op. 2. Operator attributes of this dist_op, e.g. transpose_x in matmul op. Args: dist_op: the DistributedOperator input_names: list, name of the dist_op's input tensors output_names: list, name of the dist_op's output tensors attr_names: list, attribute name of the dist_op's corresponding serial op Returns: input_specs: list, DistTensorSpec for each input tensor of the dist_op output_specs: list, DistTensorSpec for each output tensor of the dist_op attrs: dict, attribute map of the dist op Examples: .. code-block:: python >>> # doctest: +SKIP('Depends on other ops.') >>> from paddle.distributed.auto_parallel.static.utils import wrap_data_for_completion >>> op_desc = dist_op.serial_op.desc >>> input_name_list = [] >>> output_name_list = [] >>> input_name_list.append(op_desc.input('X')[0]) # 'X' is the arg name for op >>> input_name_list.append(op_desc.input('Y')[0]) >>> output_name_list.append(op_desc.output('Out')[0]) >>> attr_name_list = ['trans_x', 'trans_y'] >>> input_specs, output_specs, attrs = wrap_data_for_completion( ... dist_op, ... input_name_list, ... output_name_list, ... attr_name_list) ) rrrr_var_recursiverGrrNrrr)rr"rr input_specs output_specsr+rr&rrrY dist_spec attr_names r*wrap_data_for_completionrp sTKL E!!I",,@@Foo,,T2yy "<B 9% ",,AA$Goo,,T2yy "<B I&  $>>..y9  e ++r,cURRRU5RnU(aURR U5nOURR U5n[X45$r>)rrrkrGrrrr)rr&is_inputrYrs r*get_dist_tensor_specrs s`$$**99$?EEL",,@@F",,AA$G , 99r,cURRn[UR55S:XdS5eUSnU$)Nrzinvalid grad_var_to_var)rgrad_var_to_varr.r)rygrad_var_to_var_maprus r*get_grad_var_to_varrw sG&77GG "'') *a /J1JJ /)!,O r,cSSKJnJn Sn[UR 5R 5GHupgUR S5(GaMURS5S:XGa7/SQnURR5U;GaURR5n SU ;d SU ;dS U ;a]SU ;aURRS5O;SU ;aURRS5OURRS 5n URR5n S U ;dS U ;a<S U ;aURRS 5OURRS 5n W SUR5;aW H n X*SX-'M URR (dGMUR U5(dGMURU5[#UR$5-(dGMURU5[#UR&5-(dGMUnGM US:waUR 5R US - nURRS 5SnUR 5R UnURRS 5SnUUR5;aXU'ggg) Nr)rrr4rr)rwr@castc_concatconcatr all_gatherr(Inputr~r)outr)/paddle.distributed.fleet.meta_optimizers.commonrrr"r)rr*rrrr"r$rrramprr@rr)rrrurrfirst_backward_op_idxrrreshard_op_typesr"r*rr+r scale_loss_opscale_loss_var_namefirst_backward_opscale_loss_grad_var_names r*update_grad_var_to_varr sn W113778 KK ' ''+CC  ww||~!11 gg113 ;&+-k)+- c* '+5GGMM'2!#s!3  "ww335 L(E\,A!L0u-WW^^E2 !9 4 4 66")2A)2L/#* LL    K((%FOO(<<<%FKK(888$' !c9h",,.223H13LM +0077>qA#002667LM#4#9#9#@#@#G#J #?+?+?+A A8K4 5 B #r,cURnUH;nURS:XaXlUR5Hn[XA5 M M= grT)rrr{set_all_ops_op_role)rrrVrrns r*rr0 s>iiG ::  JI  3%r,c[Rn[RnU(aU(d SU35eU(d SU35e[U5S:XdS[U535eX#-nU"USUSS9nU"USUSS9nU"US USS9n /n [ U5H8n XX-U S-U-- n U R X5 U R X5 M: U"U SS9$U"USS9$) afuse function for fusing weights (1) fuse_attention_qkv q => [q1,q2,q3,q4] k => [k1,k2,k3,k4] or [k1,k2] for GQA v => [v1,v2,v3,v4] or [v1,v2] for GQA fused weight => [q1,k1,v1,q2,k2,v2,q3,k3,v3,q4,k4,v4] or for GQA [q1,q2,k1,v1,q3,q4,k2,v2] (2) fuse_attention_ffn directly fuse weights to 1 parts [gate_weight], [up_weight] => [gate_weight, up_weight] Args: fuse_params (_type_): to be fused weights is_qkv (bool, optional): for attention qkv weights. Defaults to False. num_heads (_type_, optional): query heads. Defaults to None. num_key_value_heads (_type_, optional): key and value heads. Defaults to None. Returns: _type_: fused weights 3num_heads should be number of heads for Q, but got Mnum_key_value_heads should be number of key_value_heads for K and V, but got r}zKfuse_params length is not equal 3, it should be Q K V list. but got length rr4r^rr )rr{rwr.rFrN) fuse_paramsis_qkv num_headsnum_key_value_heads concat_fnsplit_fnnum_query_groupsq_listk_listv_list qkv_pairsrKs r*fuse_param_funcr9 s70 I||H  A) M y# [\o[p q ";1$ YZ]^iZjYk l $%;+a.)"=+a.*=BG+a.*=BG *+A $A1A'A I   VY '   VY ' , ,,2..r,c[Rn[RnU(aU(d SU35eU(d SU35eX4-n///pnU"XSU--SS9n [U5HYn XXS--U S-US--S- - nU R XS-US--S- 5 U R XS-US--S- 5 M[ U"USS9U"U SS9U"U SS94$U"XSS9$)a)split function for splitting weights (1) fuse_attention_qkv fused weight => [q1,k1,v1,q2,k2,v2,q3,k3,v3,q4,k4,v4] or for GQA [q1,q2,k1,v1,q3,q4,k2,v2] after split q => [q1,q2,q3,q4] k => [k1,k2,k3,k4] or [k1,k2] for GQA v => [v1,v2,v3,v4] or [v1,v2] for GQA (2) fuse_attention_ffn directly split weight to 2 parts [gate_weight, up_weight] => [gate_weight], [up_weight] Args: fused_param (_type_): len(fused_param)=1, only one weight to be split split_nums (int, optional): split_nums. Defaults to 2. is_qkv (bool, optional): for attention qkv weights. Defaults to False. num_heads (_type_, optional): query heads. Defaults to None. num_key_value_heads (_type_, optional): key and value heads. Defaults to None. Returns: _type_: split weights rrr r4r^r)rr{rwrFrN) fused_param split_numsrrrrrrrrr split_headsrKs r*split_param_funcrq sF< I||H  A) M y# [\o[p q "%;!#R Q)<%<<2 *+A )*a!e8H18L-M. F MM+1u1AA1E&F&JK L MM+1u1AA1E&F&JK L , f2 & f2 & f2 &   b99r, global_mesh sub_mesh_dimc|URn[U5nX:d US:aU*U:a[SUSUS35eUS:aX- n[R"UR 5R U5n[R"XBUUS9n/nUH'nUR[XpR55 M) U$)Nrz"The sub_mesh_dim should between (-z, rr^) rGr.rrr#rr6rwrNr rO)rrr mesh_ndimrsplit_process_ids sub_mesh_listsub_process_idss r* split_meshr s""JJI ql]Y60 2i[ J  a! ((;223;;JGK -LM, )>)> ? - r,cUR5nUR5RR5S:wag[R R U5 [RRSURURS9nURUR55 URU5 g)al Update the subblock within a pylayer operation by modifying its output argument. This function optimizes a pylayer operation by removing unnecessary outputs from the 'cf.yield' step. Args: trivale_value (pir::Value): The output argument of the pylayer operation to be modified. Example: (1) Original pylayer operation: (%1, %2) = "pd_op.pylayer" (%0) { () = "cf.tuple_pop" [id:1] (%3, %4) = "dist_op.xxx" [id:2] () = "cf.yield" [id:3] (%3, %4) } (2) After calling `update_pylayer_output(%4)`, the updated pylayer operation removes the unused output: (%1) = "pd_op.pylayer" (%0) { () = "cf.tuple_pop" [id:1] (%3) = "dist_op.xxx" [id:2] () = "cf.yield" [id:3] (%3) } Args: trivale_value(pir::Value): The output argument of the pylayer op to be updated. rN_fake_pylayer_out)r&rGrE)rkget_parent_block parent_opr&rrset_insertion_pointrdatarGrEset_typerreplace_all_uses_with) trivial_value define_op fake_values r*update_pylayer_outputr s4--/I!!#--224G JJ""9-## !!!!$J  **,-'' 3r,) auto_parallelr>)NFN)FFN)NFFNFT)T)FNN)r FNN)rrrrr: functoolsrnumpyrrpaddle.base.frameworkrpaddle.base.libpaddlerpaddle.base.wrapped_decoratorrpaddle.frameworkrpaddle.framework.io_utilsrr paddle.staticr rJr r dist_attributerrrop_proto_and_checker_makerrkOpRoleAttrNamerVarDescVarTypeREADER STEP_SCOPESDENSE_TENSOR_ARRAYFEED_MINIBATCH FETCH_LIST__no_shape_var_type__r_g_gradient_clip_opsrhr+r1r8r;rLrRrVrZr^rerirnrrrurrrrrrrrrrrrrrrrrrrrrrrr7r8rOrgrXrYrWrrrrrrrrrrrrrrrrrrrVr_rbrhrlrrrvryrrrrrrrrrrrrrrr rr%rr>rI dygraph_guardrrPrWrNr^rbrfr`r/rprsrwrrrrr@rrr5r,r*rs   -%"J"<LL ( ( / /--==?  LLLL$$LL++LL''LL## #.     *  , # "-20. "F&0f!H*."Q&4 *. N 7 t)0Z<@6r ' P  A02jSl F0BJ,^$N/d-`  % 2 I# C "" *?&?0M`BJ!:>v r $ 4  AH HV0 5  5 5Q( 2     C E$ =N  $(Wt/ 7$,5 %C,C,.2C,@DC,L:BLJ4DH5/t  <:~Ks4$4r,