a,j& UddlZddlZddlZddlZddlmZmZmZddlm Z m Z m Z ddl m Z ddlmZmZmZddlZddlmZddlmZddlmZddlmZmZmZmZmZm Z ddl!m"Z"dd l#m$Z$m%Z%m&Z'm(Z(m)Z)m*Z*m+Z+m,Z,dd l-m.Z.m/Z/dd l0m1Z1dd l2m3Z3dd l4m5Z6ddl7m8Z8gdZ9dZ:dZ;dZe?Z@e1ezejAzeBzeCze?zZDeDeEeDzeFeDzeGe?dfzZHeGe?eHfZIeEeIZJeGe?eIeJzfZKe>ZLe>eeMd<ejNdZOe GddZPe GddePZQ dKdejRde?de?d eSd!eSd"e@f d#ZTGd$d%ZUdLd&ZVddd'dejRd(eFejWjXd)fd*eSd+e>ejRdzd,ePdzd"eQf d-ZYd.eGe?eHfd/eKd0eQd"dfd1ZZd2ejRejWjXzd3e?d"efd4Z[d5eGe?efd0eQd"eGe?effd6Z\ej]dejRd0eQd"eGe?eHffd7Z^ej]dejRd5eGe?eHfd0eQd"e3fd8Z_d9ejWjXd"dfd:Z`d5eKd"eGe?eHffd;Zad9ejWjXd5eGe?eHfd0eQd"eKfd<Zbej]dejRd=eFejWjXd)fd0eQd"eKfd>ZcdejRd9ejWjXd/eKd0eQd"eKf d?Zdej]dejRd=eFejWjXd)fd5eKd0eQd"df d@Zeddd'dejRd+e>ejRdzd,ePdzd"eGe?eHffdAZfddd'dejRd=ejWjXeejWjXzd+e>ejRdzd,ePdzd"eKf dBZgddd'dejRd=ejWjXeejWjXzd+e>ejRdzd,ePdzd"eFeGe?eHfeKff dCZhdejRd5eGejReGe?eHffeGe?eHfzd"eGe?eHffdDZiddEdejRd.eGe?eHfd,ePdzd"e3fdFZjddEdejRd=ejWjXeejWjXzd/eKd,ePdzd"df dGZkddEdejRd=ejWjXeejWjXzd.eGe?eHfd/eKd,ePdzd"e3f dHZleddEdejRd,ePdzd"dfdIZmeddEdejRd=eFejWjXd)fd,ePdzd"dfdJZndS)MN)Callable GeneratorIterable)asdict dataclassfield)chain)Anycast no_type_check) ShardedTensor)_broadcast_state_dict_distribute_state_dict_flatten_state_dict_gather_state_dict_offload_state_dict_to_cpu_unflatten_state_dict)_CHECKPOINT_PREFIX)FullOptimStateDictConfigFullStateDictConfigFullyShardedDataParallelOptimStateDictConfigShardedOptimStateDictConfigShardedStateDictConfigStateDictConfig StateDictType)._get_module_fsdp_state_if_fully_sharded_moduleFSDP_WRAPPED_MODULE)DTensor)_IncompatibleKeys)DistributedDataParallel) tree_map_only) FQNS_T PrimitiveType ValueType DictValueTypeListDictValueTypeOptimizerStateTypeStateDictOptionsget_model_state_dictget_optimizer_state_dictget_state_dictset_model_state_dictset_optimizer_state_dictset_state_dict _flat_param param_groupsparamsstater%_patched_state_dictc#Ktj}tj dV|rtjdSdS#|rtjwwxYwN)gc isenableddisableenable) is_enableds g/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/torch/distributed/checkpoint/state_dict.py _gc_contextr=QshJJLLL    IKKKKK  :  IKKKK s AAceZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed<dZ eed <dZ eed <d Z eed <d S)r)ap This dataclass specifies how get_state_dict/set_state_dict will work. - ``full_state_dict``: if this is set to True, all the tensors in the returned state_dict will be gathered. No ShardedTensor and DTensor will be in the returned state_dict. - ``cpu_offload``: offload all the tensors to cpu. To prevent CPU OOM, if ``full_state_dict`` is also true, then only the rank0 will get the state_dict and all other ranks will get empty state_dict. - ``ignore_frozen_params``: if the value is True, the returned state_dict won't contain any frozen parameters -- the ``requires_grad`` is False. The default value is False. - ``keep_submodule_prefixes`` (deprecated): when ``submodules`` is not None, this option indicates whether to keep the submodule prefixes from the state_dict keys. or example, if the submodule is ``module.pretrain`` and the full FQN of the parameter is ``pretrain.layer1.weight`` of the param. When this option is True, the parameter's key in the returned state_dict will be ``pretrain.layer1.weight``. If the options is False, the key will be ``layer1.weight``. Note that if ``keep_submodule_prefixes`` is False, there may be conflicted FQNs, hence there should be only one submodule in ``submodules``. - ``strict``: the ``strict`` option when ``set_state_dict`` calls model.load_state_dict(). - ``broadcast_from_rank0``: when the option is True, rank0 should receive a full state_dict and will broadcast the tensors in the state_dict/ optim_state_dict one by one to other ranks. Other ranks will receive the tensors and shard according to the local shards in the model and optimizer. ``full_state_dict`` must be set to True when using this option. This option currently only supports DTensor, not the legacy ShardedTensor. Ffull_state_dict cpu_offloadignore_frozen_paramsTkeep_submodule_prefixesstrictbroadcast_from_rank0flatten_optimizer_state_dict_fqn_modifiersdsd_fqn_modifiersN)__name__ __module__ __qualname____doc__r?bool__annotations__r@rArBrCrDrErGstrr<r)r)\s""H"OT!!!K!&$&&&$(T(((FD!&$&&&). $...-s-----rPr)cNeZdZUeeZeeejze ejzfe d<eeZ eeejze ejzfe d<ee Z e ee d<dZee d<dZee d<ejZee d<eeZeeje d <d S) _StateDictInfo)default_factoryfqn_param_mappingshared_params_mappingsubmodule_prefixesT handle_model handle_optim fsdp_context fsdp_modulesN)rHrIrJrdictrTrNtorchTensorr#rMrUsetrVrWrLrX contextlib nullcontextrYrlistrZnnModulerOrPr<rRrRs d###t el $$$ d###4 el $$$$)5#=#=#=C===L$L$'3L(333$)E$$?$?$?L$ry/?????rPrRrFTmodelnamerGskip_ddp_prefixskip_compiler_prefixreturncx |td}d|vr|hS|d}g}|}t|D]?\}} t |t r8| dkrt d| d|j}|s|| St |tr|t|dz krZ||dztkrFd | t|t} r d fd| jDcSt|t}| tkr%|| t|| }t |t jjjr9| dkrt d | d|j}|s|| |t+||rMt||| x} r t+|| rt|| }|| | t.jjjkr'|t|dz krt5d /t|| }Ad |tdhS) a This API is used to convert the name of a parameter to the FQNs. For FSDP without `use_orig_params`, the name of FlatParameter can be mapped to multiple original parameters. As a result, the return type of this function is `set[str]`. Args: module (nn.Module): the root model. name (str): the name skip_ddp_prefix (bool): whether to skip DDP's `module` prefix Returns: The canonical FQNs based on the model traversal. .modulezExpected 'module', got ''ch|]}| SrOrO).0fqnprefixs r< z_get_fqns..s$EEES6(3((EEErP _orig_modzExpected '_orig_mod', got 'z-Expect `_extra_state` to be the last obj name)replacersplit enumerate isinstanceDDPAssertionErrorrlappendFSDPlen _FLAT_PARAMjoingetattr_fqnsrr\_dynamo eval_frameOptimizedModulerthasattrgetrbmodules_EXTRA_STATE_KEY_SUFFIX RuntimeError) rdrerGrfrg obj_names fqn_obj_namescurr_obji curr_obj_name flat_param removed_fqnrrs @r< _get_fqnsrs. <<*B / /D $v  3IMH%i00&<&<= h $ $% <(($%P %P%P%PQQQH" 4$$]333 $ ' ' <3y>>A%%%)AE*:k*I*I-00$X{;; * &\\\FEEEEJ4DEEEEEEx)<==H 333$$]333"8];; %-":"J K K < ++$%S=%S%S%STTT)H' 4$$]333x!233 B"F'(4E"F"F"H"H"L"L!##;Bx55B#*8[#A#A   / / / 1 IIII***&'VWWW+#8];; HH] # # + +,> C C DDrPceZdZdS) _EXTRA_STATEN)rHrIrJrOrPr<rrsDrPrc#Ktdtjdtdtffd |dEd{VdS)Nrlcurr_fqnrhc3K||r|dnd}|D]i\}}|vr t|r7|t|vr |dd}n||}||Ed{Vjt |d|dD]\}}||jvr||}||fVt|j dtj j tj j kr.|tj jj}|t!fVdSdS)NrkrjF)recurseget_extra_state)addnamed_childrenrrvaluesr named_buffersnamed_parameters_non_persistent_buffers_set __class__rbrcrrrlrr) rlrre submodulenew_fqnobjrGrvisited_moduless r<rz+_iterate_valid_model_state..recursesF###%-5h>>>>2%4466 3 3OD)O++ 122 .>GF,=>>@@GGIIII#3B3-%-t--wy'22 2 2 2 2 2 2 2 2   / /1H1HQV1H1W1W    ID#v999!)4))G3,     F$&79R S Sy( ) )"N2:#4#LNNG<>>) ) ) ) ) )  ) )rPrj)r^rbrcrNr)rdrGrrs `@@r<_iterate_valid_model_staters~&)eeO * *S *Y * * * * * * * *Dwub!!!!!!!!!!!rP) submodulesoptionsoptims. optim_onlyrrc p|rtjdtd|r|std|p t }i}i}t |D]\}}t |trt||} | |} | Ettt|| | ||||<n| ||<| D]} t |ts||| <t|D]'\} } | D]} tt"j| || < (t} |rt|}|D][\}}||vr t||} t)| dkrt+d| d| D\|jr|jst1d t3j|}|r|jrJt7|j|j }t;|j|jp|j }t<j}n6tA|j }tC|j }t<j"}tFj$d }tKj&||||| }n tFj'}tQditS|||| |tttTj+|| t)|dkdS)zW Verify the model and options passed by the user and generates _StateDictInfo. zGetting submodules only model/optim state_dict is deprecated and will be removed in 2.5. This feature can be achieved by manually filtering out the state_dict returned from get_state_dict. stacklevelz;Optimizers are not passed in but optim_only is set to True.Nrnz)Submodule FQN should only have 1 instancec3 K|] }|dV dS)rkNrO)rprqs r< z"_verify_options..Ks(%@%@Ciii%@%@%@%@%@%@rPz?full_state_dict must be True when broadcast_from_rank0 is True.)offload_to_cpu rank0_only)rc3Ktj5tjddtt j||||5dVdddn #1swxYwYddddS#1swxYwYdS)NignorezFSDP.state_dict_type)messagecategoryrlstate_dict_typestate_dict_configoptim_state_dict_config)warningscatch_warningsfilterwarnings FutureWarningr|rrs r<$fsdp_state_dict_type_without_warningz=_verify_options..fsdp_state_dict_type_without_warningis#(**  '&<})!$3&7,C  EEE                   s55A4 A A4A A4#A $A44A8;A8rr)rTrUrVrYrZrWrXrO),rwarnrrr)rrxrrrr r^rNupdatecopyraitemsr\r] named_modulesr}rzrDr? ValueErrorr|rZrr@rrFULL_STATE_DICTrrSHARDED_STATE_DICTr_contextmanager functoolspartialr`rRrrbrc)rdrrrrrTrUreparamfqnsrqparam_fqns_rVrlrZrrrrrYs r<_verify_optionsrs   I       &  I   +)++GKMOQ1%88// e e\ * *  %%##E** ? S,U3 4 4 ; ;D A A A+O  "   # " $!( 0+/$;    "-    // +3-!$ry/<88#^&kkAo     rPmodel_state_dictoptim_state_dictinfoc|jD]"}t|}|td#|jrP|sN|jsG|js@|jr|js2|jr+|j s$tdtj d|j r)|s'|jr|js|j std||D]%}t|vrt|dtd&dS)Nz)Expected a fsdp_state with a fsdp module.z}The option indicates that model state_dict is required to save or load, but model state_dict is empty.rank = dist.get_rank()=rkzgThe option indicates that model state_dict is required to save, or load but optim state_dict is empty. z contains z6. This can happen if the model is not the root module.)rZrrzrWrVrAr@r?rCrDrdistget_rankrXr~)rrrrl fsdp_statekeys r<_verify_state_dictrs #NNCFKK   !LMM M       '  )  !  '+&:  K  )   *moo * * *       % *.*> .  M:JMM    #  **+***  rPrapict||}|tvr)tjt|j||}|S)N)self)rr4rrr)rrcalls r<_state_dict_fnrsC 3  D """ !.verifyss88s3xx'' 5IIcNN IIcNN )29)=)=%%%GX9W#5551 "c)nn44#*c)nnq.@#@@@@5!%<<< $uutrPzAn unexpected key, z, exists. FQN is )rWrYrrakeysrr}rznextiterrLrpoprV startswithrBrAr requires_gradr) rdrrrrrqrnew_state_dictrrrrs r<_get_model_state_dictrs      ;;8^E<88:: ;;;;;;;;;;;;;;;JOO%%&&22$$ t99>> J3JJs4yyJJDJJ 4:: #:: D    "6#s## V"#T#T#Ts#T#TUUU(nnS11JsO  $/1 > >C1 > >~~f--/>*4S/N3''!#f++--0G.8oN7++ >$   $0022 $ $JC" UC((D $ $s#### $ )T : ::sAA Ac |jr |s|jstiiSi}t||jD]\}}t |||j}t |||jdd}t ||D]f\}} |jrtjdkr>|| kr8| |d} | |j rtd|dn| || <||| <gd} |js|j rt} |D]H}tj|r2|dkr| |jItjd| vr)| tjdd} t+| dkr2| tjn"t+| dkrt1d |jr1t3||| |j |j n+|j r$t7||| |||5t=tt?|d ||j | cdddS#1swxYwYdS)NF)rfrgrz Missing key: rkmetaTrnzMultiple devices found)devicerCr@rload_state_dict)rrCassign) rWrDr rrGrziprrrrCrr?r^rr\ is_tensordimrrremover}distributed_c10d_get_pg_default_devicerrr@rrrYr r) rdrrlocal_state_dictrvaluerfqns_with_prefixrqfqn_with_prefix load_valuerdevicess r<_load_model_state_dictrss  )Z)8Q) R(((08NOO66 UT%;<<$    "!!&    %(.>$?$? 6 6 C- =15A1E1E(('^^C66 %{C*+A3+A+A+ABBBC3=J/05 _ - - 6F  ,D$8,%%%,,.. * *Eu%% *%))++// EL))) <  7 * * NN5<// 0 0 0F w<<1   KK-DDFF G G G G \\A  566 6  $ W ! {{}}{ ,       ! W ":/? V V V V*+++        4N5"3 4 4%dk&                       s4KKKoptimcB|jrdS|jD]}|tD] }|jdS|jD]2}|tD]"}|jrt j||_#3g}|jD]Z}d|vrT||dt|dt j rt j dnd|d<[| d|jD]}d|vr| d|d<| ddS)zH Initialize optim states by calling the step() with zero grads. Nlrg)closurerT) set_to_none)r3r1_PARAMSgradrr\ zeros_liker{rxr]tensorstepr zero_grad)r param_grouprlrss r<_init_optim_staterasx { )  )  Ez%& )55  ) 5 5E" 5"-e44  5 C) ;   JJ{4( ) ) )k$/>> S!!!    JJtJ)++ ;   # K  OOO%%%%%rPc  dtttfdtdtttff fd d i}t t |t D]U\}}t d|}| t tttf||Vt t|tD]k}| t}t tt|D].}|D]\}}||td|d|</l|S)a3 This API flattens the optimizer state_dict to support optimizer resharding for MPMD, e.g., pipeline parallelism. Without the API, the original optimizer state_dict looks like: { "state": { "layer1.weight": { "step": 10, "exp_avg": SomeTensor, "exp_avg_sq": SomeTensor }, "layer2.weight": { "step": 10, "exp_avg": SomeTensor, "exp_avg_sq": SomeTensor }, }, "param_groups": [ { "lr": 0.0, "betas": (0.9, 0.95), ..., "params": ["layer1.weight", "layer2.weight"] } ] } With this API, the optimizer state_dict looks like: { "state.layer1.weight.step": 10, "state.layer2.weight.step": 10, "state.layer1.weight.exp_avg": SomeTensor, "state.layer2.weight.exp_avg": SomeTensor, "state.layer1.weight.exp_avg_sq": SomeTensor, "state.layer2.weight.exp_avg_sq": SomeTensor, "param_groups.layer1.weight.lr": 0.1, "param_groups.layer2.weight.lr": 0.1, "param_groups.layer1.weight.betas": (0.9, 0.95), "param_groups.layer2.weight.betas": (0.9, 0.95), } The "state" section supports arbitrary levels of nesting for optimizers like Shampoo. nested_dictrrrhci}|D]d\}}t|}|r|d|n|}t|tr |||T||||<e|S)a Recursively flatten a nested dictionary with dot-separated keys. Args: nested_dict: The dictionary to flatten prefix: The prefix to prepend to all keys Returns: Flattened dictionary with dot-separated keys rk)rrNrxr[r) rrr flattenedrrstr_keyfull_key_flatten_state_nested_dict_raise_if_type_not_supporteds r<rz=_flatten_optim_state_dict.._flatten_state_nested_dicts+- %++-- , ,JC#hhG06C&,,7,,,GH%&& ,  !;!;E8!L!LMMMM-,U333&+ (##rPct|tjttt fs t dt|ddS)Nz[Flattening optimizer state_dict only supports tensor, int, float, dict states now. Type is rk)rxr\r]intfloatr[NotImplementedErrortype)vs r<rz?_flatten_optim_state_dict.._raise_if_type_not_supportedsX!elC=>> %&77&&&   rPrk)r[rNr r%r r&_STATErrr'_PGrr ra) rretrqr3 state_prefixrrkr#rrs @@r<_flatten_optim_state_dictr)sR#s(^-0 c9n <!#C=*V*<==CCEE  U ((3((  & &tDcNE'B'BL Q Q    -z#??,, w''S 4(( , ,C#))++ , ,1*+s&&S&&1&&'' , , JrPc dtdtttfdtttffd}i}g}t|t|i}|jD]}|tgi|tD]'}|j|D]} | |j vr)d} |D]#} | tkrtd| d| } | |vrd} nd} | s:|dt} t| tstd t| | | |jsi|| <|j|D]d}td| d|}||vr+|||}|t!t"|| |<@||t!t"|| |<e)t!tt|dtd }|D]s} | tkr|td|d| }| |dvr ||d| <9|d| |kr(t%d |d| d |d |d| d t|S)z This API unflattens the state_dict generated by _flatten_optim_state_dict(). Supports arbitrary levels of nesting in the state section through recursive reconstruction. See the docstring of _flatten_optim_state_dict() for more detail. flattened_keyflattened_dictrhcf|d}i}|D]\}}||s|t|d}|d}|}|ddD]5} | |vri|| <t || t st || }6|||d<|S)z Reconstructs a potentially nested value from flattened keys. For non-nested values, returns the value directly. For nested values, reconstructs the nested structure with string keys. rkNr)rrr}rvrxr[rz) r+r,rrrrr remaining_keypartscurrentparts r<_reconstruct_nested_dictz=_unflatten_optim_state_dict.._reconstruct_nested_dicts"$$$&( )..00 ' 'JC>>&))  F .M"'',,E!Gcrc  ( (w&&$&GDM!'$-66)((!$-"'GE"I  rPFrkTrExpected list, got rzaAll the parameters in the same parameter group should have the same saved param_group value. But z is z while other(s) is )rNr[r%r$r%r1r{r rTrUrxrarzr"rr3r r&r)rrrr2r3pg_state return_osdrrrq in_paramsr( flatten_keyr2 state_nameflattened_state_keyreconstructed_valuefirst_param_fqnrs r<_unflatten_optim_state_dictr<s500,0i,@0 c9n 0000dE"$H&,eS(%CJ)AA " &&& )1 1 E-e40 0 $444 %I(<<$),&8&8s&8&8Q&8&8 &*44(,I $I !"g.!&$//O()MtF||)M)MNNN c"""* c "'+e"4J-3*H*Hc*H*HJ*H*H'**<<.F.F///+ 0]E#J77 CC GQ/G]E#J77 CC?0 dtCy(2,w*?@@C  AG||#====!==>E $$"' Q"aE))"G=LGGOPGGGG4z)_get_optim_state_dict..s&JJJ!!))L"55JJJrPc30K|]}|tVdSr6)r )rpgs r<rz(_get_optim_state_dict..s&-U-UQaj-U-U-U-U-U-UrPrnrrc g|] }| SrOrO)rppidfqn_pid_mappings r<rCz)_get_optim_state_dict..s!Q!Q!Q3/#"6!Q!Q!QrP)#rXr$r%rrrZrYr|rrarrrur r from_iterabler1r[rranger}rrrzrrr r&rr'extendrEr(r)r)rdr=rrrosdr(rEr2param_pid_mappingrrrrqrGgrouprHs @r<_get_optim_state_dictrOus   ,2BR+@4H4H%   1nUL1133  + R""$$ ? ?+E5#>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? #f+**,,-- R R!##?B6{q?Q?QCK , ; ;<X $ $JJqzJJJ#'  $%--U-U%BT-U-U-UUUVVF $Ss6{{1C1C%D%D E E  O#4466 + + U ,,t99>>(J3JJs4yyJJ4::&& 111'.'*$'*$$CK,,..// 8 8%c*$'v;??3#7#7F C  S R R!Q!Q!Q!Q%.!Q!Q!Qg   ],V455<.s, U U!:a   U U U U U UrPFTrr3z'Missing optimizer state for parameter 'z' in checkpoint. The parameter requires gradients but has no saved optimizer state. To load anyway, use StateDictOptions(strict=False).rnrzThere are param groups that have zero parameters. In such a case, DSD only support exactly one param group with zero parameters.But the loaded state_dict has zero or more than one param groups that have zero parameters.z`When there is a parameter group that has zero parameters, multiple optimizers are not supported.)r$r%allr r&r1r{r rTrUr'rarNrxrzr"rrCrr}idrrr)rdrrrr3r4r5 pg_mappingrrrqr6loaded_param_groupr2r&pg_idxrrs r<_split_optim_state_dictrWs *E"$H&,eS(%CJ!#J U UtM;KF;S'T'T U U UUU )7J7J " &&& ) V VE-e4 V V$444 %I.2)+;C+@//""*$tCy2DW2M"N"NNN(,I!EO!%I !"g.!&$//O()MtF||)M)MNNN c"""&d=2B62JKKKK%)-9I&9Q%R%RSV%Wc *RcRRR +/%'7'<++VV&d49.@.IJJJJ=@C=Q=QTU=U 2&8#9#9: V7 VB {7# $ $ ) )C&*+<>Ns>S&T&T 3 3"tDI'9''BCCDDIIJJ12223xx1}} 1#C())S1C-D-DDD =25Z_1E1E1IJr,-- .-/?/DEE * * ;44 R<< %++-- * *JCg~~$)HV S ! !  * rPc |jsdS|D]y}t||rSt|vrt||||}n9t |t t ttf||}ni}|j r| D][\}}t||}t||d} || kr/t|dkr"td|dt||| |tD]M} t t tt f| } fd| t"D} | | t"<Nt t$|t} t'| D]2}|vr,| || |<3]|5t/j|||}dddn #1swxYwYn9|jr1d|_t5||f|}d|_dfd}t7t8j||}td t=|\}}t=|\}}|jrtA|| ntC|| |D]3}||vr-||vrtd |d ||||<||||<4tE||}|tD]:}t"|vr/gt t ttf|t"<;tG|d |{dS)NF)rgrnzExpected 1 FQN for 'rc<g|]}|SrOrA)rprrqfqn_with_compilers r<rCz*_load_optim_state_dict..Ks5@C C):;;rPTc|dkr$|jn|jkrtd|S)NrzDevice mismatch)rrr)trs r<_devicez'_load_optim_state_dict.._device^sC5577Q;;~!"18++():;;;rPzExpected device to be setrzExpected key 'z' in osd_mappingr)r)$rXrr$rWr<r r[rNr%rZrrr}rzrr%r r r&rarrurYr|optim_state_dict_to_loadr?rOr"r\r]rrDrrrr)rdr=rrrr original_fqn_rfqns_with_compilerrEvalr2 osd_stater(rr] flatten_osd osd_mappingflatten_local_osdlocal_osd_mapping optim_keypgrrqrZs @@@r<_load_optim_state_dictrj!s  [N[N%    "###:5*d$$  $?4S)^ 4jAA4$$   "   I A$)#9#9#;#; X X a  55%.>(O|OOCIIOOhhjj$6$:$:$<$<!)#.**AtCH~q11CGJ7|F$*CLL 0@0HII inn..//XXAaxxGP}}UVGWGW !))C1B"C"CDX""$$  #'#@5"2$$                 !) A#(D 4UUHdKK #'D F     elG5EFFA~$%@AAA':;K'L'L $K3FGW3X3X 0 0( V%k3DVTTTTT&{4EfUUUU ) J J $555 33,HYHHH4?y3I%i03>y3I%i04!#4   's+ A A"$$>@Dc9n-r227; 1u/00>> # xdoctest: +SKIP >>> import torch >>> from torch.distributed.fsdp import FullyShardedDataParallel as FSDP >>> from torch.nn.parallel import DistributedDataParallel as DDP >>> from torch.distributed.checkpoint.state_dict import get_state_dict >>> fsdp_model = FSDP(copy.deepcopy(model)) >>> fsdp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_model = DDP(copy.deepcopy(model)) >>> ddp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_state_dict, ddp_optim_state_dict = get_state_dict(ddp_model, ddp_optim) >>> fsdp_state_dict, fsdp_optim_state_dict = get_state_dict( ... fsdp_model, fsdp_optim ... ) >>> # if we simply call ddp_model.state_dict() and fsdp_model.state_dict(), >>> # the asserts will fail. >>> assert ddp_state_dict == fsdp_state_dict >>> assert ddp_optim_state == fsdp_optim_state_dict Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules (deprecated): Optional[set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: ``Tuple`` that contain model state_dict and optimizer state_dict. :rtype: typing.Tuple[typing.Dict[str, ValueType], OptimizerStateType] FrlN) r=rxr\rrnrorrrOr)rdr=rrrrrs r<r,r,sP 22*ek&;<< #ZMMz""    !    1==0 DII+-=tDDD!11!222222222222222222sA;BBBc" |siSttt|tjrt jdtdtttjtttff|}i}| D]\}}|D]\}}||kr t||}t!|dkrt#dtt|d | fd| D|Sttttf|S)NzPassing model_state_dict as a ``Dict[nn.Module, Dict[str, Any]]``is deprecated and will be removed in 2.5. If you need this feature, please preprocessing the model_state_dict to achieve the same functionality.rrrnz/FQNs for a submodule should only have 1 elementrkc"i|] \}}|z| SrOrO)rpsubfqnrrrs r< z/_unflatten_model_state_dict..Ts#XXX Vf_eXXXrP)rxrrrrbrcrrrr r[rNr%rrrr}rzr) rdrcast_state_dictrrsub_state_dictremrrrs @r<_unflatten_model_state_dictrx6s  $tJOO--..//;;6  "      tBItCN/C$CDjQQ/1)8)>)>)@)@   %I~ ..00  a >> --t99>>(I!d,,///%%XXXXAUAUAWAWXXX Di(*555rP)rct||}t5t|dd|}t|i|t |||cdddS#1swxYwYdS)a=Load the model state_dict. The counterpart of ``get_model_state_dict`` to set the state_dict to the model. See ``set_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. model_state_dict: (Dict[str, ValueType]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys * **unexpected_keys** is a list of str containing the unexpected keys :type model_state_dict: typing.Dict[str, ValueType] rOFrrN)rxr=rrr)rdrrrs r<r-r-[s:.I .. EEubUGLLL+R666%e-=tDD EEEEEEEEEEEEEEEEEEs5A!!A%(A%c"t5t|tjjr|fnt |}t ||d|}ti||t||||ddddS#1swxYwYdS)aLoad the optimizers state_dict. The counterpart of ``get_optimizer_state_dict`` to set the state_dict to the optimizers. See ``set_state_dict`` for the detail usage. WARN: ``set_optimizer_state_dict`` can only be called before ``backward()`` or after ``step()`` is called on the optimizers. Otherwise, the optimizer states won't be initialized correctly. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None :type optim_state_dict: typing.OptimizerStateType TrzN) r=rxr\rrnrorrrj)rdr=rrrs r<r.r.s>  J J*ek&;<< #ZMMz""  ujT7SSS2/666uj2BDIII J J J J J J J J J J J J J J J J J JsA(BB Bcdt||}t5t|tjjr|fnt |}t||| |}t|||t||||t|||cdddS#1swxYwYdS)aLoad the model state_dict and optimizers state_dict. The counterpart of ``get_state_dict`` to set the state_dict to the model and optimizers. The given ``model_state_dict`` and ``optim_state_dict`` do not have to be returned by ``get_state_dict`` but must meet the following requirements: 1) all FQNs are canonical FQNs as defined in ``get_state_dict``, 2) if a tensor is sharded, it must be either a ShardedTensor or DTensor, 3) optimizer state_dict cannot contain the parameter IDs; the keys should be the canonical FQNs. WARN: ``set_state_dict`` can only be called before ``backward()`` or after ``step()`` is called on the optimizers. Otherwise, the optimizer states won't be initialized correctly. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. model_state_dict: (Union[Dict[nn.Module, Dict[str, ValueType]], Dict[str, ValueType]]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys of the model state_dict. * **unexpected_keys** is a list of str containing the unexpected keys of the model state_dict. :type model_state_dict: typing.Dict[str, ValueType] :type optim_state_dict: typing.OptimizerStateType rzN) rxr=rxr\rrnrorrrjr)rdr=rrrrs r<r/r/s-\.I ..  E E*ek&;<< #ZMMz""   :.>*>    +-=tDDDuj2BDIII%e-=tDD E E E E E E E E E E E E E E E E E EsA9B%%B),B)cFtjt||fd}||_tjt||dt t tfffd }||_t |t |dS)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``model``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``model`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rdrcSr6rO_state_dict_callsr<state_dict_callz0_patch_model_state_dict..state_dict_call!!!rPrc |dS)N)rrOr_load_state_dict_calls r<load_state_dict_callz5_patch_model_state_dict..load_state_dict_callz::::::rPN) rrr*rr-r[rNr rr4r)rdrrrrrs @@r<_patch_model_state_dictrs6!( """""'E%- ;c3h;;;;;;1EO,,,011111rPctjt|||fd}tjt|||dtt t fffd }t|t|t|tj j r|fnt|}|D]}||_||_dS)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Note that if there are multiple optimizers, all of the optimizers will be patched. So users only need to call one of the state_dict() to get the full result. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rdr=rcSr6rOrsr<rz4_patch_optimizer_state_dict..state_dict_callLrrPrc |dS)N)rrOrs r<rz9_patch_optimizer_state_dict..load_state_dict_callVrrPN)rrr+r.r[rNr r4rrxr\rrnrorr)rdr=rrrrrrs @@r<_patch_optimizer_state_dictr&s>!(  """""&-  ;c3h;;;;;;O,,,0111 j%+"7 8 8  :   55* 455rP)rFTT)rF)or_rr7rcollections.abcrrr dataclassesrrr itertoolsr typingr r r r\torch.distributedrrtorch.nnrb'torch.distributed._shard.sharded_tensorr #torch.distributed._state_dict_utilsrrrrrr;torch.distributed.algorithms._checkpoint.checkpoint_wrapperrtorch.distributed.fsdprrrr|rrrrr$torch.distributed.fsdp._common_utilsrrtorch.distributed.tensorrtorch.nn.modules.moduler torch.nn.parallelr!rytorch.utils._pytreer"__all__r~r%r r$r^rNr#r]rr r$raror[r%r&r'r(r4rMrr=r)rRrcrLrrrrrnrrrrno_gradrrrr)r<rOrWrjr*r+r,rxr-r.r/rrrOrPr<rs  99999999990000000000++++++++++ AAAAAA                    -,,,,,555555<<<<<<------   "    S-'%,6>c;FVAWW S)^$ '#}/@@@A&)SUUS]***  ,.,.,.,.,.,.,. ,.^  @ @ @ @ @% @ @  @&. !% FEFE 9FE FEFE FE  FE  FEFEFEFER        %"%"%"%"Z)-'+ }}} 9} %+', -}} BI% }  $ }}}}}@+3 >*+(+ + ++++\ EK$99S#X&4 #s(^$?; 9?;*?; #y.?;?;?;?;DA 9A S)^$A  A  A A A A H'&U[2'&t'&'&'&'&T_*<_c9nAU____DE ; ES)^$E E EEEEPDA 9DAek+S01DA DA DADADADANa 9a ; a)a  a  aaaaHdN 9dNek+S01dN#dN  dN  dNdNdNdNT)-'+ " " " 9" BI%"  $ "  #y. " " " " R)-'+ * * * 9*  %1F(GG* BI% *  $ *  * * * * b)-'+ X2X2X2 9X2 %1F(GGX2BI% X2  $ X2  4Y !3 34 X2X2X2X2v"6 9"6RYS)^ 445S)^8LL"6 #y."6"6"6"6R(, $E$E$E 9$E3 >*$E $ $E  $E$E$E$EX(, (J(J(J 9(J %1F(GG(J)(J  $ (J  (J(J(J(Jb(, =E=E=E 9=E %1F(GG=E3 >* =E ) =E  $ =E=E=E=E=ED(,121212 912 $12 12121212l (, ;5;5;5 9;5ek+S01;5 $ ;5  ;5;5;5;5;5;5rP