x-jUddlmZddlZddlmZddlmZmZddlm Z m Z ddl m Z ddl ZddlZddlmZddlmZdd lmZmZdd lmZdd lmZmZmZe r dd lmZdd l m!Z!iZ"de#d<edGddZ$edGdde$Z%Gdde Z&Gdde Z'dZ(dZ)d(dZ*d(d Z+d!Z,d"Z-Gd#d$Z.d%Z/Gd&d'Z0dS))) annotationsN) defaultdict) dataclassreplace)Enumauto) TYPE_CHECKING)logger)LocalTensorIndexLocalTensorMetadata) ShardedWeight)%compute_local_shape_and_global_offsetget_target_tensor slice_tensor)Group)AbstractCommunicatorzdict[str, tuple[list, list]]PATH_TO_CHECKPOINT_FILEST)frozenceZdZUdZded<ded<ded<ded<d ed <ded <ded <ded <ded<ded<dZded<dS)ReadItema A communication operation for a Tensor between ranks. Attributes: tensor_name (str): Name of the tensor. src_global_offset (tuple[int]): Global offset in the source tensor. dst_global_offset (tuple[int] | None): Global offset in the destination tensor. dst_rank (list[int]): Destination ranks. src_rank (int): Source rank. dst_local_offset (tuple[int]): Local offset in the destination tensor partition. src_local_offset (tuple[int]): Local offset in the source tensor partition. slice_shape (tuple[int]): Shape of the slice to transfer. file_name (str): The name of the file from which the source tensor is read on the source rank. dtype (str): Data type of the tensor. str tensor_namez tuple[int]src_global_offsettuple[int] | Nonedst_global_offsetdst_rankintsrc_rankdst_local_offsetsrc_local_offset slice_shape file_namedtypeNz Group | None comm_group)__name__ __module__ __qualname____doc____annotations__r%p/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/paddle/distributed/flex_checkpoint/dcp/resharder.pyrr/s !!!!((((MMM        NNNJJJ#J######r,rceZdZUdZded<dS)ExtendReadItemNr global_shape)r&r'r(r0r*r+r,r-r/r/Ns#&*L******r,r/c6eZdZeZeZdS) OperationTypeN)r&r'r(rGLOBAL_BROADCASTBROADCAST_ALLGATHERr+r,r-r2r2Ss(tvv$&&r,r2c6eZdZeZeZdS) AllGatherTypeN)r&r'r(r WITH_PADDING NO_PADDINGr+r,r-r6r6Xs$466LJJJr,r6 __internal_padding_tensor_name__c|i}tj}|D],}|jD]\}}||vr||f||<-g} |r"tj| ||n| |i} | D],}|D]\}\} }|| vsJ| |f| |<-| SN)paddle distributedget_rankstorage_metadataitemsall_gather_objectappend) metadata_listlocal_load_files process_groupuse_dist load_infocur_rankmetadatalocal_tensor_indexr#load_info_list load_infosranks r-get_load_infosrN`s(I!**,,H!-5-F-L-L-N-N   )  ,,,1 ,-  N),, I}     i(((J#?? 5>__5F5F ? ? 1  1y%Z7777.2I->J) * * ? r,cur_chunk_metadatar storage_local_tensor_metadatac :g}g}g}t|j|j|j|jD]\}}}}t||} t ||z||z} | |kr.|d|| |z nL| |kr.|| |z |dnt d| d|d||| | z |ddksJd|dd| d| |||fS) NrzInvalid begin_offset:z , cur_offset:z, storage_offset:zInvalid length:z , end_offset:z, begin_offset:)zip local_shape global_offsetmaxminrB ValueError) rOrP cur_offsetsstorage_offsetslengthscur_len cur_offset storage_lenstorage_offset begin_offset end_offsets r-compute_overlaprbysKOGK7 8 8W$77448 5r,ci}|D]L}|j}|D].\}}||dt|z||</Md|D}t|d}|D]R}|j}|D]4\}}||}t|} |||||| z<|| z||<5S|S)Nrc"i|] \}}|dg|z Sr;r+).0kns r- z5build_storage_state_dict_metadata..s$ 7 7 71a$! 7 7 7r,)state_dict_metadatar@getlendictfromkeys) rCcountsmdr@rhlstresultoffsetoris r-!build_storage_state_dict_metadatarvs F44&,,.. 4 4FAs 1a((3s883F1II 48 7  7 7 7F ]]61 % %F&,,..  FAsq ACA#&F1Ia!a%i AF1II   Mr,ci}t|}g}d}|D]\}} d} t| tjr| rWt | jdkr t| j| j | j nd\} } t| j}| | nUt| j} t | jdkr%tdgt | jznd} | }t| j dd} |} nt| tr}t | jdkr| j| jfnd\} } t| jj dd} t|tr|dn|} nt)dt+| t-| | | |}|| D]}t/||rt1||\}}}t3| t|jt|j}||\}}|t7| t|j| tjf|t|t|t|||j  g}g}|r"tj|||n|||D]}|D]}|||S) Nr)r+r+r+.r z&Only support paddle.Tensor., val type:)rT) rrrrrr r!r"r#r$)rvr@ isinstancer<Tensoris_distrmshaper process_mesh placementstuplerr$splitrr0rTrU local_tensorrXtyper rdrbr rBrr=r>rA)rC state_dictrErFrLstorage_state_dict_metadata read_itemsr0 tensor_keyvalrrTrUr$rOrPrYrZr[storage_local_tensor_indexrr#global_read_itemstmpr@items r-get_read_itemsrs#%"C##JL%++--LL C c6= ) )' {{}} +39~~)) : ( "! %SY// &-*?+@$CI.. 36sy>>A3E3EE1#CI.///2 +  NN((--a0E$KK ] + + s'((1,,#"344 'K (.//55c::1=E!+J!>!>N 1 J KDcDD 1 ;|  .I .    )-/LMM 4C"$A55 1K'*:3ABB!"?"KLL*** & #--G"H Hi    +&+5C'''4$099;;=%%*;%7%7%*?%;%; %g'7=        > C,,S*mLLLL :++ + +D  $ $T * * * * + r,c0eZdZ d d dZdZdZd ZdS) StateDictResharderNFT communicatorrch||_||_||_||_||_||_||_dSr;)target_state_dictsource_state_dictrCrrEoffloadrF)selfrrrCrrErrFs r-__init__zStateDictResharder.__init__%s>"3!2*(*   r,cD|jrE|jD]+\}}d|D|j|<,t|j}t |j||j|j}t|j|j |j|j|}d|j D}td|D}td|D}|r |r Jd||fS)Nci|]I\}}|t|tjr'tj|tjn|JS)placerynpndarrayr< to_tensorCPUPlacergrhvs r-rjz1StateDictResharder.preprocess..:b5551!!RZ00v'1B1BCCCC555r,cRi|]$\}}|t|tr|jn|%Sr+)ryrrrs r-rjz1StateDictResharder.preprocess..OsC' ' ' 1 A}!=!=Dq~~1' ' ' r,c3@K|]}t|tVdSr;ryrrgrhs r- z0StateDictResharder.preprocess..S=  %&Jq%       r,c3BK|]}t|t VdSr;rrs r-rz0StateDictResharder.preprocess..V@  )* 1e$$ $      r,=target_state_dict contains a mix of tuple and non-tuple keys.) rrr@listkeysrNrCrErFrrany) rr#rrDrLrprocessed_target_state_dict has_tuple_keyhas_non_tuple_keys r- preprocesszStateDictResharder.preprocess7s < )-)?)E)E)G)G  % :55!+ 0 0 2 2 555&y11   6 ; ; = =>>#      M   $    "   M    ' ' .4466' ' ' #  *E        .I     " &7  K  8+J66r,cx|D]}|j|j|j}t||j|j}t||}t||j|j}|j |j kr| |j }tj ||dSr;) rr#rrr!r" contiguousrr rtor<assign)rrr read_item src_tensorsrc_chunk_tensor dst_tensordst_chunk_tensors r- local_reshardz StateDictResharder.local_reshard^s# > >I/ 0CD%J ,I6 8M  jll ++YJ ,I6 8M    %)9)???#3#6#67G7M#N#N M*,< = = = = > >r,cbtj}|\}}t jdt |d|s|S||jd}|j|d}|j r|j |||n| |||`|S)Nz/ReadItem generation completed, with a total of rx)rMrE)rr) r<r=r>rr informrErrFr communicater)rrHrrcontextstates r-reshardzStateDictResharder.reshardps%..0026//2C2C/#Z Pc*oo P P P    /. .!/   "&!7!<   = H   ) )*eW E E E E   z+F G G G  "**r,)NFT)rr)r&r'r(rrrrr+r,r-rr$sf!!!!!$%7%7%7N>>>$+++++r,rc"|j|jks Jdt|j}gg}}gg}}t|D]}|j|}||j|z} |j|} | |j|z} t || } t | | } | | krdS|| |z || |z || | z || | z tj |j tt|||}tj |j tt|||}tj ||dS)NzGlobal shapes must be the same)axesstartsends) r0rmrangerUrTrVrWrBr<slicerrr)srcdstndimrr dst_startsdst_endsi src_beginsrc_end dst_begindst_end overlap_begin overlap_end src_slice dst_slices r-assign_sharded_weightrs  s/ / / /( 0 / / s DrDFrJ 4[[ 1 1%a( coa00%a( coa00Iy11 '7++ - ' ' FF mi/000 K)+,,,-)3444 i/0000  tE$KK00dI   %++    I  M)Y'''''r,cbeZdZ dddZdZdZdZd Z dd Zd Z d Z dZ dZ dZ dZd S)ThreeDCommGroupStateResharderFmemory_growth_thresholdrc ||_||_t|dks Jd|d|_||_||_|jdf|jdffD])\} } | jdksJd| d| jd| d *||_|j p|jjdk|_||_ ||_ d |_ | dS) Nr zOnly support one metadata now!r horizontalverticalzThe number of ranks in the z@ communication group must be greater than 1, but actually it is z). Please check this communication group: !T) rrrmrIh_groupv_groupnranksp_groupusing_2d_comm_grouprrusing_tuple_keyr) rrrrCrrrrrgroupnames r-rz&ThreeDCommGroupStateResharder.__init__s-"3!2=!!Q&&&(H&&&%a(   \< ( \: &   KE4.rr,c3@K|]}t|tVdSr;rrs r-rz;ThreeDCommGroupStateResharder.preprocess..rr,c3BK|]}t|t VdSr;rrs r-rz;ThreeDCommGroupStateResharder.preprocess..rr,rc3FK|]\}}t|tVdSr;)ryr)rg_rs r-rz;ThreeDCommGroupStateResharder.preprocess..sD  1 q- ( (      r,z/All sharded weights must be ShardedWeight type.r)rrr@r$r<float32cuda pin_memoryrrrDrrallrrgrouped_target_state_dictrBr=r>rH_build_cross_section_topologyrschedule_read_itemsaggregate_global_read_items) rr#rrtensorrrkeysharded_weights r-rz(ThreeDCommGroupStateResharder.preprocesss < @)-)?)E)E)G)G  % :55!+ 0 0 2 2 555&y11*.)?)E)E)G)G @ @% :+5+;+;+=+=@@'K|v~5528++--2J2J2L2L ;//28++-- ;// @ !%T%;%@%@%B%B C C  *.*@        .2.D     " &7  K  8  .4466      = = = = = -)4T):):&#'#9#?#?#A#A K K C# K.s1v6==nMMMM.s3::>JJJJ*3355  **,,,    """ ((*****r,c|j}|j}g}tj|||g}tj|||d|D}|S)Ncg|] }|D]}| Sr+r+)rgsublistxs r- zMThreeDCommGroupStateResharder.all_gather_cross_section_fn..s%FFFwgFFFFFFr,)rrr<r=rA)rrrr h_obj_list v_obj_list gathered_infos r-all_gather_cross_section_fnz9ThreeDCommGroupStateResharder.all_gather_cross_section_fnsi,, ,,ZwGGG ,,ZWMMMFF*FFF r,cg}g|_tj||j|jtj|j||j|js.g}tj||j|jn|jg}dt|D|_ ||_ |j |j|_ dt|jD|_dt|jD|_dt|jD|_|j|j|_|jj|_|jj|_dS)Nci|]\}}|| Sr+r+)rgrrMs r-rjzOThreeDCommGroupStateResharder._build_cross_section_topology..#sIII71dtQIIIr,c,g|]}t|Sr+)set)rgcols r-rzOThreeDCommGroupStateResharder._build_cross_section_topology..'sGGGCs3xxGGGr,c$i|] \}}|D]}||Sr+r+)rgrranksrMs r-rjzOThreeDCommGroupStateResharder._build_cross_section_topology..(sE! ! ! 5! !  !! ! ! ! r,c$i|] \}}|D]}||Sr+r+)rgrrowrMs r-rjzOThreeDCommGroupStateResharder._build_cross_section_topology..-s?   3s  ?CD!    r,)topologyr<r=rArHrrrr enumerateparallel_indexp_rankscur_parallel_indexrSvertical_rankshorizontal_indexvertical_indexcur_horizontal_indexr h_group_size v_group_size)rh_ranksr s r-rz;ThreeDCommGroupStateResharder._build_cross_section_topologys ,, T]DL    ,, M7DL   ' &G   0 0     }oGIIi6H6HIII "&"5dm"DGG3 3FGGG! ! %d&9::! ! !    "+DM":":   %)$9$-$H! L/ L/r,ctt}|D]1}|j|j|jf}|||2g}|D],\}}t|d}||-|S)Nc|jSr;rrs r-z@ThreeDCommGroupStateResharder.dedup_read_items..=s r,r)rrrrr"rBr@rW)rrrrrrsr@min_items r-dedup_read_itemsz.ThreeDCommGroupStateResharder.dedup_read_items6sD!!% $ $D#T%;T=MNC #J  d # # # #++-- $ $JC5&:&:;;;H MM( # # # # r,Nc`tj}|jj}|jj}i}|D]\\}}|j}||} t| dksJd|d|| dj } ||f} | |j |j f} | || <]g} |j D],\}}|D]\}}|||f\} }}t|jdd}t!|jt!|ks(Jd|d|d|dt!|j||t!|t!|d t|zd t|zt!|t!| ||ddd }| t'd i|.|| }d |D}|||_dS) Nrz'No metadata found for tensor with name z in file rxr zShape mismatch in tensor name z, expected shape z , but got r) rrrrr!r r"r0r#r$rr%cg|] }|D]}| Sr+r+)rgplanrs r-rz@ThreeDCommGroupStateResharder.get_read_items..ws9( ( ( ( ( 9=D( ( ( ( r,r+)r<r=r>rIrkr?r@rrmr0rTrUrrr$rrr|rBr/rrr)rall_gather_args current_rankrkr? shard_infosrJr#rlocal_tensor_metadatar0r shard_infolocal_read_plan read_filerrrrTrUr$ common_attrsgathered_plans_per_rankglobal_read_plan_per_sections r-rz,ThreeDCommGroupStateResharder.get_read_itemsAs)2244 "m?=9 -=-C-C-E-E * * )  +6J$7 $C !,--222Z*ZZyZZ32213@Ly)C"."0J *K  %)%;%A%A%C%C G G !Iz'1'7'7'9'9 G G# V;F ),<8 k=FL))//44Q7V\**eK.@.@@@@UkUUIUUhsUU@EFLFR@S@SUUA@@$/ ,).})=)=).})=)=(,s;/?/?(?(,s;/?/?(?#(#5#5$),$7$7!*" $"&    &&~'E'E 'E'EFFFF- G0#'"B"B # # ( ( 4( ( ( $//0LMMr,c 4 |j}g}tt}|jD]}|j}|j}t jtt |}|tj |zt|z}||j kr||||f||t|d} t#| }d} |D]\} } t| | || <g} t|D]?\\}}}t||jkr"t||jkrdgt|z}d}t't)|D]Z\}}|vr |j}|j|}||4|||<||dz }|t|krn[t1d|DrD| |t2jffdt)|D}||||f<nnt||jkAt|dkrdgt|z}d}tt&d}|D] \\}}}|}|D]|\\}}}||krt)|D]]\}}|j}|j|}||<|||<||f||dz }|t|krn^}g}t|D]O\}fdt)|D}t|dkr|J||<P|D]|=t)|D]B\}}|;t7||}t8|d d d d d d d |dd }t;dd di|}|||<C| |t2jft|dk||_| |_ dS)Nc:|dd|ddfS)Nrr r+rs r-rzCThreeDCommGroupStateResharder.schedule_read_items..s!Q!Qr,rc |j|j|j|j|j|j|jnd|j|j|j|j f S)Nr+) rrrrr rr!r"r#r$)rs r-rzCThreeDCommGroupStateResharder.schedule_read_items..sU  !    '    &*6  ' '  &  !   O ) r,rr c3K|]}|duV dSr;r+)rgrs r-rzDThreeDCommGroupStateResharder.schedule_read_items..s&CCq}CCCCCCr,c"g|] \}}|v | Sr+r+)rgrr used_indicess r-rzEThreeDCommGroupStateResharder.schedule_read_items..s2#AtL00000r,c.g|]\}}|v|Sr+r+)rgrrrr1s r-rzEThreeDCommGroupStateResharder.schedule_read_items..s7###4 S 111111r,r)r padding_vfile) rrrrr!r r"r0r#r$r%rr+)!rrrrr$r"r<core size_of_dtypegetattrmathprodrmrrBsortedr@rnrrr rraddrr6r8rWINTERNAL_PADDING_TENSOR_NAMEr/r7global_broadcast_read_itemsbatch_read_items)rrr<bucket_read_itemsr cur_dtype cur_shape element_size memory_growthbucket_read_items_t order_rulesrhrr=r@cur_batch_read_itemscntrrh_indexcur_batch_dtype need_removeremaining_itemsr)padding_read_itemrr1s @@r-rz1ThreeDCommGroupStateResharder.schedule_read_items}s,&(#'--O 4 4D I(I!;44WVY5O5OPPLty333c.6I6II t;;;+224888 ),!#&--d3333$  # # % %   !!455   &++-- > >DAq#)!#=#=#= a -12C2I2I2K2K-L-L   ) "Y E5zzD---e** 111(,vN0C0C'C$"uu (// " "GAtL(( #}H"3H=G+G4<8<,W5$((+++q#n"5"555!ECC.BCCCCC $++-}/GH'0'7'7E AF%y)&<==9e** 111<#$$))$(6C,?,?#? "O&s++LC1B1H1H1J1J  -&I"+1B1H1H1J1J " "-&I//(//""GAt#}H"3H=G+G4<8<,W5$i%;<@@CCCq#n"5"555!EK"#4#:#:#<#<== = = U######,U#3#3### ''1,,&&s++++-<%c**" + +%c**$%9:: @ @4<">!#455H'C$,-1-1,0,0'+(,%4!0&* $ $L)7))!%))5))%/@(+  # #%}'AB   w#$$))~,G( 0r,cR|jr$|j|_d|jD|_dSg}g}t j||j|jt j||j|jd|D|_g|_td|D}t|D]v}g}|D]U}t|dkr)| | d>| gdfV|j |wdS)Ncg|]}|gSr+r+)rg batch_itemss r-rzMThreeDCommGroupStateResharder.aggregate_global_read_items..s'000"- 000r,cg|] }|D]}| Sr+r+)rgrrs r-rzMThreeDCommGroupStateResharder.aggregate_global_read_items..&s?7 7 7 7 7  7 7 7 7 r,c,g|]}t|Sr+)rm)rgrs r-rzMThreeDCommGroupStateResharder.aggregate_global_read_items..-s E E EgS\\ E E Er,r) rr<&aggregated_global_broadcast_read_itemsr=aggregated_batch_read_itemsdistrArrVrrmrBpop)rrQrR max_tasksr task_batchesr=s r-rz9ThreeDCommGroupStateResharder.aggregate_global_read_itemss  # 0  700151F000D , F13.&(#  2  , L    '  ! L   7 7 A7 7 7 3 ,.( E E)D E E E  y!! B BAL$? 4 4 '((A-- ''(8(<(.Ps/OODDId&677OOOOOOr,rRN)r)"rmrrHrrr;r<emptyr"r$rr#ryr CUDAPlacerr= broadcastrr6r7rVr7r8r|reshapenumelzeros_clear all_gatherrr clonerB)rrNrallgather_typerbuffer tensor_list max_numelr_ padded_bufferunpadded_tensor_listidx padded_tensorunpadded_tensors r-'_process_one_batch_broadcast_in_sectionzEThreeDCommGroupStateResharder._process_one_batch_broadcast_in_section8s%0" N z??a  It89 =I. . .(,HHH Y2IODDD+I,?@) flF,<== '\)"7yOOOF$$  *$, %    ]7 7 7OOJOOOOOIy&&)33!   & i[ M M M (.vt(D(D fuf% &   ) )V4< *   $& &/ &<&< ' '"]&sO  )"788!&5&)//1199):OPP %++O<<<$$&&&&.KK   ) )V4< *     r,c||j\}}t}t|D]*\}}|jtkr||+t |dD]}||=g}t|D]P\}\}}|j|} |D]8}|jtkr&t|| } | | 9Qd} t|D]n\}}|j sk|j |j kr|| } | dz } n tj|j|j} tj| |j |jn || } | dz } t)|j| |j|j|j} |j|jD]}|js3tj|j} | |j| j}|jj}|j|kr||}|| _t=| | | ~ pdS) NT)reverserrr rXrYrrrTr0rUrr) rrr rr;r:r9r rrBrrrHr<r[r"r$r=r]rrr0rrr_is_initialized zeros_like_share_buffer_torrrra)rrfrVr=rdneed_remove_indicesrjrfiltered_read_itemsrreplcaed_read_itemrFrereceived_sharded_weighttarget_sharded_weightr tgt_places r-"broadcast_cross_p_group_and_assignz@ThreeDCommGroupStateResharder.broadcast_cross_p_group_and_assignts+78O+P(.!ee'(899 - -NC$(DDD#'',,,-t<<< ! !CC   7@7N7N C C 3C3"N|C(H- C C (,HHH)0X)N)N)N&'../ABBB C '(;<<0 (0 (NC+ %66(-F1HCC#\!-Y_F",, 2$,-%S)q&3)#%1&3'9 ''' #*.)G%*  %-9IIKKP#.-:F++,A,NOOO4A 1>D #y00!+y!9!9J7A'4%/- MMOOO''a0 (0 (r,c |j}t|}t|dD]B\}}|dzdks||kr"tjd|d|d|j|j|jkrQ|j|j |j}t|j tj s|}n tj|j|j}tj||jd t)|j||j|j|j }|j|jD]}t1|| |~DdS) Nr start rzBroadcasting item /z: rXrYrprq)rQrmr r rrrHrrr#ryrr<r\rr[r"r$r=r]rr0rrrra)rr< total_itemsrjrrerxrys r-'broadcast_cross_global_group_and_assignzEThreeDCommGroupStateResharder.broadcast_cross_global_group_and_assigns  7 $566 '(C1MMM$ ($ (NCRx1}}{ 2 2 UUU{UUi>SUU} 222/ 0CD)"&,0@AA+#[[]]F)   ( (I.d )   '4)#%1&3'9 ''' #*.)G%*  %&/- MMOOO''I$ ($ (r,c (t|j}tjdt |jdD]n\}}|||j}||||dzdks||kr$tjd|d|d|d z|zd otjd tjd |tjd tjddS)NzL[ThreeDCommGroupStateResharder] Begin resharding using batch broadcasting...r r}rrzResharding batches: rz (dz%)zJ[ThreeDCommGroupStateResharder] End resharding using batch broadcasting...zM[ThreeDCommGroupStateResharder] Begin resharding using global broadcasting...zK[ThreeDCommGroupStateResharder] End resharding using global broadcasting...z4[ThreeDCommGroupStateResharder] Resharding finished.) rmrRr rr rmrr{r)rtotalrjrVrfs r-rz%ThreeDCommGroupStateResharder.reshardsWD455 Z   "+  ,A" " "   CFFT45K  3 3K N N NRx1}}u  P3PPPP#)u:LPPP  X     [    44666 Y     JKKKKKr,)rF)rrr;)r&r'r(rrrrrrrrrmr{rrr+r,r-rrs(3     D/+/+/+b   "0"0"0J   :N:N:N:NxQ1Q1Q1f&B&B&BP:::xC(C(C(J)()()(VLLLLLr,r)rOr rPr )1 __future__rr7 collectionsr dataclassesrrenumrrtypingr numpyrr<paddle.distributedr=rS'paddle.distributed.fleet.utils.log_utilr rIr r rrutilsrrrpaddle.distributed.collectiver reshard_commrrr*rr/r2r6r;rNrbrdrvrrrrr+r,r-rs#"""""" ######********  !!!!!!::::::;;;;;;;; 33333332222229;;;;; $$$$$$$$$< $+++++X+++!!!!!D!!! D B21111@$*aaaHf+f+f+f+f+f+f+f+R ( ( (FN LN LN LN LN LN LN LN LN LN Lr,