IЦi7tSSKrSSKrSSKJrJrJr SSKrSSKJr SSK J r J r SSK J r SSKJr SSKJr SS KJrJr SS KJr SS KJr S S KJr S SKJrJr \(aSSKJr "SS\5r Sr!Sr""SS\ 5r#"SS\ 5r$"SS\ 5r%"SS\ 5r&"SS\ 5r'"SS\5r(g) N)DictList TYPE_CHECKING) BackwardState)compiled_autograd variables) trace_wrapped) unimplemented)%call_module_hooks_from_backward_state) GuardBuilder install_guard) AttrSource)istype)VariableTracker)ConstantVariable EnumVariable)InstructionTranslatorcF^\rSrSrSrSU4SjjrSr\S5rSr U=r $)DistributedVariablea_ The base distributed variable that encapsulates common methods for the distributed objects (i.e. ProcessGroup, DeviceMesh, etc.). Concrete distributed objects could inherit this class and add object specific logic. i.e. It provides the check on the distributed package existance and hold the tracking value for the corresponding distributed object. c z>[TU]"S0UD6 [R5(d [ S5 Xlg)Nz+torch.distributed package is not available!)super__init__r is_availabler value)selfrkwargs __class__s b/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/torch/_dynamo/variables/distributed.pyrDistributedVariable.__init__#s0 "6""//11 G H c,[UR5$N)typerrs r" python_typeDistributedVariable.python_type)sDJJr$c>[RR5$r&)torch distributedrrr$r"r DistributedVariable.is_available,s  --//r$r)returnN) __name__ __module__ __qualname____firstlineno____doc__rr) staticmethodr__static_attributes__ __classcell__r!s@r"rrs&  00r$rc[R5(dgSSKJn [R "U5=(a XR L$)NFr)DTensor)rrtorch.distributed.tensorr;inspect isfunction from_local)rr;s r" is_from_localr@2s6  + + - -0   e $ D2D2D)DDr$c[R5(dgSSKJnJnJnJnJn UUUUU/n[R"U5=(a X;$)NFr)_get_group_size_by_name_get_group_tag_rank_not_in_group$_resolve_group_name_by_ranks_and_tagget_process_group_ranks) rr"torch.distributed.distributed_c10drBrCrDrErFr=r>)rrBrCrDrErFconstant_processgroup_functionss r"is_constant_pg_functionsrI:sR  + + - - , '#   e $ Q)QQr$cL^\rSrSrSr\S5rSSS\S\4U4Sjjr S r U=r $) WorldMetaClassVariableQzt Tracks torch.distributed.GroupMember and torch.distributed.group, which are instances of the metaclass _WorldMeta. cTUR5(dgSSKJn [U5UL$)NFr) _WorldMeta)rrGrNr')clsrrNs r"is_group_member_type+WorldMetaClassVariable.is_group_member_typeWs&!!AE{j((r$txrnamer0c>US:Xa[[URSS9n[UR[R 55 [ URR5$US:Xa[[URSS9n[UR[R 55 [URR5$[TU]1X5$)NWORLD)basememberNON_GROUP_MEMBER) rsourcer make_guardr ID_MATCHProcessGroupVariablerrUrrXr var_getattr)rrRrSrYr!s r"r]"WorldMetaClassVariable.var_getattr`s 7?T[[AF &++L,A,AB C' (8(89 9 ' 'T[[9KLF &++L,A,AB C ; ;< <w"2,,r$r) r1r2r3r4r5 classmethodrPstrrr]r7r8r9s@r"rKrKQs< )) -5 -S -_ - -r$rKcR^\rSrSr\S5rSrSU4SjjrSrU=r $)PlacementClassVariablelc[R5(dgSSKJn [ U5[L=(a [ X5$NFr) Placement)rr(torch.distributed.tensor.placement_typesrfr' issubclassrrfs r"is_placement_type(PlacementClassVariable.is_placement_typems2#//11FE{d"Cz%'CCr$cUR$r&r/r(s r"as_python_constant)PlacementClassVariable.as_python_constantw zzr$c|>[R"URSS5[R4;awUR (af[R UR5n[ U5n[R"URSS5(aURUSX#5 U$[TU]%XU5$)N__new__r) r=getattr_staticrobjectrqrYPlacementVariable call_methodr call_function)rrRargsr new_objvarr!s r"rv$PlacementClassVariable.call_functionzs  " "4::y$ ?FNNCT T nnTZZ0G#G,C%%djj*dCCJ= w$Rv66r$r)rRrrwList[VariableTracker]r Dict[str, VariableTracker]r0r) r1r2r3r4r6rjrmrvr7r8r9s@r"rbrblsLDD7 #7&7- 7  77r$rbcj^\rSrSr\S5rSrSSS\S\4U4Sjjr S U4S jjr S r U=r $) rtcX[R5(dgSSKJn [ X5$re)rrrgrf isinstanceris r" is_placementPlacementVariable.is_placements$#//11F%++r$cUR$r&r/r(s r"rm$PlacementVariable.as_python_constantror$rRrrSr0c>US:Xa*[R"URR5$[TU]X5$)Ndim)rcreaterrrr]rrRrSr!s r"r]PlacementVariable.var_getattrs3 5=#**4::>>: :w"2,,r$c>SSKJn /SQnX&;Ga[UR5n[R "USS5bS5e[R "Xr5nU[RLaUR"S5$UV s/sHoR5PM nn UR5V V s0sHupXR5_M nn n US:XaU"UR/UQ70UD6 U$U"UR/UQ70UD6n UR"U 5$[T U]5XX45$![ a SnNf=fs sn fs sn n f)Nr)r)r __setattr__is_shard is_partial is_replicate __getattr__zno custom getattr allowed!r)rr'rr=rrAttributeErrorrsrrrmitemsrru)rrRrSrwr rconstant_fold_functions value_typemethodxkv constant_valr!s r"ruPlacementVariable.call_methods= '#   * !$**- **:}dKS0/0S // A('..t44489Dq((*DD9t>v>L#**<8 8w"2T::"   :KsA D*D<5E* D98D9rrwr{r r|r0r) r1r2r3r4r6rrmr`rr]rur7r8r9s@r"rtrts],,-5-S-_- *;& *; - *;  *;*;r$rtcj^\rSrSr\S5rSrSSS\S\4U4Sjjr S U4S jjr S r U=r $) DeviceMeshVariablecX[R5(dgSSKJn [ X5$)NFr) DeviceMesh)rrtorch.distributed.device_meshrr)rrs r"is_device_mesh!DeviceMeshVariable.is_device_meshs$#//11<e((r$cUR$r&r/r(s r"rm%DeviceMeshVariable.as_python_constantror$rRrrSr0c>US:Xa*[R"URR5$US:Xa*[R"URR5$[ TU]X5$)Nndim device_type)rrrrrrr]rs r"r]DeviceMeshVariable.var_getattrsV 6>#**4::??; ; = #**4::+A+AB Bw"2,,r$cT>US:XaUVs/sHoUR5PM nnUR5VVs0sHupxXxR5_M n nn[R"URR "U0U D65$US:Xa.[R"URR 55$US:Xa#[URR55$US:Xa#[URR55$[T U]-XX45$s snfs snnf)Nsizeget_coordinate get_group_get_or_create_default_group) rmrrrrrrr\rrrru) rrRrSrwr r const_argsrr const_kwargsr!s r"ruDeviceMeshVariable.call_methods 6>:>?$Q..0$J?BH,,.Q.$!A3355.LQ#**4::??J+W,+WX X # ##**4::+D+D+FG G ; ' (<(<(>? ? 1 1' (O(O(QR Rw"2T::@Qs DD$rr) r1r2r3r4r6rrmr`rr]rur7r8r9s@r"rrs]))-5-S-_-;& ; - ;  ;;r$rcb^\rSrSrSrSrSU4SjjrS U4Sjjr\S5r Sr U=r $) r\ao We don't want a ProcessGroup object to end up in our output graph. But it's common for dynamo to intercept a PG that is then used to get info like rank() or world_size(), as well as passed to utility functions in distributed_c10d which desugar it into plain types like a ranklist and tag. For convenience and proper guarding, we construct a variable type. TODO: make it possible to use ProcessGroupVariable as input to simple functions like _expand_group without dynamo complaining about making a proxy for it. It is not a tensor-like type, and we don't want a proxy- but dynamo assumes torch library functions are dealing with tensor-like types and would have proxies for their args. TODO: should we make this inherit VT instead of UDOV? Do we want any of the default behaviors or just graph-break whenever one of our special cases is not hit? cUR$r&r/r(s r"rm'ProcessGroupVariable.as_python_constant ror$c>US:Xa7[RRURR 55$US:Xa7[RRURR 55$US:Xa7[RRURR 55$[TU]!XX45$)Nrankr_get_backend_name) r rrrrrrrrurrRrSrwr r!s r"ru ProcessGroupVariable.call_methods 6>--44TZZ__5FG G 6>--44TZZ__5FG G & &--44TZZ5Q5Q5ST Tw"2T::r$c>^^^TS:Xa3[RRTRR5$TS;a[R "UUU4Sj5$[ TT]TT5$)N group_name)rrc*>TRTTX5$r&)ru)rwr rSrrRs r"2ProcessGroupVariable.var_getattr..#s(8(8T4(Pr$)r rrrrLambdaVariablerr]rs```r"r] ProcessGroupVariable.var_getattrs_ < --44TZZ5J5JK K # #++P w"2t,,r$ch[R5(dgSSKJn SSKJn [ XU45$)NFr) ProcessGroup)FakeProcessGroup)rrtorch._C._distributed_c10dr+torch.testing._internal.distributed.fake_pgrr)rrrs r"is_process_group%ProcessGroupVariable.is_process_group(s-#//11;Pe,<=>>r$rrrRr) r1r2r3r4r5rmrur]r6rr7r8r9s@r"r\r\sI$;& ; - ;  ; -??r$r\c ^\rSrSrSr\S\S\S\4Sj5rS\RRS\S\S\SS 4 U4S jjr S r S \ \S \\\4S\4U4SjjrSSjrSrU=r$)BackwardHookVariablei3zI Handles torch.utils.hooks.BackwardHook for module-level backward hooks. module user_hooksuser_pre_hooksc^^ ^ [R(d [S5 S[4UU U 4SjjnURR US5umnURR U5um nURR U5um nURR SUU405n[RRRSSS5URRS'[XqX#5$)Nz6module-level backwards hooks require compiled autogradbw_statec >[RRRS[R "[ [UTTS94[R "[ [UTTS945$)z Rather than installing the user hooks in the graph (which don't survive AotAutograd), we install hooks that will call trace_wrapped in the backward pass that CompiledAutograd can turn into actual hook calls. N)fnr hooks_name module_name)r,utilshooks BackwardHook functoolspartialr r )rruser_hooks_nameuser_pre_hooks_names r"_in_graph_bw_hooks7BackwardHookVariable.create.._in_graph_bw_hooksCsk;;$$11%%%@!)#2$/ %%%@!)#6$/  r$modrvr example_value)rcompiled_autograd_enabledr routputadd_backward_state_hook create_proxyr,rrrnodemetar) rRrrrrbw_state_proxy_proxyrrrs @@@r"rBackwardHookVariable.create9s!:: R S   :')ii&G&GPU&V# ^!#!B!B>!RQYY>>zJ &&        ,1;;+<+<+I+I$PRTV+W (#E:NNr$rr0Nc V>[TU]"S0UD6 XlX lX0lX@lg)Nr)rrrrrr)rrrrroptionsr!s r"rBackwardHookVariable.__init__ls* #7#  $,r$cUR$r&)rr(s r"as_proxyBackwardHookVariable.as_proxyzror$rwr c\>US;aUR"X/UQ70UD6$[TU] XX45$)N)setup_input_hooksetup_output_hook) _setup_hookrrurs r"ru BackwardHookVariable.call_method}s; < <##B>t>v> >w"2T::r$c SSKJn U"UURRSUUR 5UR 54055$)Nr) wrap_fx_proxyru)builderrrrr)rrRhook_method_namerwrs r"r BackwardHookVariable._setup_hooksD*  II " " $--/2    r$)rrrrr)r1r2r3r4r5r6rrr,fxProxyrrrrr`rurr7r8r9s@r"rr3s 0O0O$0O( 0O0Od -xx~~ -  -$ - ( -  - ;?# ; S/)* ;  ;   r$r))rr=typingrrrr,%torch.fx.experimental._backward_staterrrr _trace_wrapped_higher_order_opr excr external_utilsr guardsr rrYrrrrVrconstantrrtorch._dynamo.symbolic_convertrrr@rIrKrbrtrr\rrr$r"rs,, ?+:B0!4D0/04ER.-0-6 70 7F=;+=;@&;,&;R8?.8?v` ?` r$