<Цi)n 2%SSKrSSKJrJrJr SSKJr SSKJrJ r J r J r J r J r Jr SSKJr SSKJr SSKJr SS KJr SSKrS S /r\R2R4rS r0r\ \ \ 4\S 'SrS5Sjr\"\R@5SS.S\!4Sjj5r"\"\RF5S6S\!4Sjj5r$\"\RJ5S6S\!4Sjj5r&\"\RN5S6S\!4Sjj5r(S5S\\!S\\!S\\!S\)S\!4 Sjjr*\"\RV\RX/5SS.S\!4Sjj5r-\"\R\5S\!4Sj5r/Sr0\"\Rb\Rd\Rf/5SS.S\!4Sjj5r4Sr5SS .S\\ \ \!S!4\ \!S!4\ \!S!4\ \ \!S!444S"jjr6SS .S\\ \ \!S!4\ \!S!4\ \!S!4\ \ \!S!444S#jjr7\"\RpS$S%9SS.S\!4S&jj5r9\"\RtS$S%9S\!4S'j5r;S(r<\"\Rz\R|\R~/5SS.S\!4S)jj5r@\"\RS$S%9S\!4S*j5rB\"\RS$S%9S\!4S+j5rD0\R@\"_\RF\$_\RJ\&_\RN\(_\RV\-_\RX\-_\R\\/_\Rb\4_\Rd\4_\Rf\4_\Rz\@_\R|\@_\R~\@_\Rp\9_\Rt\;_\R\B_\R\D_rS,rE/S-QrFS.rGS/rHS0rIS1rJ"S2S 5rK"S3S4\5rLg)7N)tree_map tree_flattentree_unflatten) ModuleTracker)ListAnyDictOptionalUnionTupleIterator) defaultdict)TorchDispatchModeprodwrapsFlopCounterModeregister_flop_formulac\[U[R5(a UR$U$N) isinstancetorchTensorshape)is W/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/torch/utils/flop_counter.py get_shapers!!U\\""ww H flop_registryc8^[T5SS.U4Sjj5nU$)N)out_valcB>[[XU45upnT"USU0UD6$)N out_shape)rr)r#argskwargsr%fs rnfshape_wrapper..nfs."*9tW6M"Ni$6)6v66r rr(r)s` r shape_wrapperr,s# 1X77 Ir c^^UU4SjnU$)Nc>^T(d [T5mU4Sjn[RRR UT5 T$)Nc>[U[RR5(d[ SUS[ U535eU[ ;a[SU35eT[ U'g)Nzlregister_flop_formula(targets): expected each target to be OpOverloadPacket (i.e. torch.ops.mylib.foo), got z which is of type zduplicate registrations for )rr_opsOpOverloadPacket ValueErrortyper! RuntimeError)target flop_formulas rregister=register_flop_formula..register_fun..register&slfejj&A&ABB Hh0f@AA&"%A&#JKK$0M& !r )r,rutils_pytree tree_map_)r6r7get_rawtargetss` r register_fun+register_flop_formula..register_fun"s7(6L 1  %%h8r )r=r<r>s`` rrr!s& r )r%returnc4UupVUupxXg:XdeXX-S-U-$)zCount flops for matmul.r@) a_shapeb_shaper%r&r'mkk2ns rmm_floprJ7s+ DA EB 7N7 519q=r c [X5$)zCount flops for addmm.)rJ self_shaperDrEr%r's r addmm_floprNBs 7 $$r c PUupEnUupxn XG:XdeXh:XdeXE-U -S-U-n U $)z"Count flops for the bmm operation.rCr@) rDrEr%r'brFrGb2rHrIflops rbmm_floprSGsA GA!IBA 7N7 7N7 519q=1 D Kr c [X5$)z&Count flops for the baddbmm operation.rSrLs r baddbmm_floprVTs G %%r x_shapew_shaper% transposedc|USnU(aUOUSSnUtpgn[U5[U5-U-U-U-S-n U $)aCount flops for convolution. Note only multiplication is counted. Computation for bias are ignored. Flops for a transposed convolution are calculated as flops = (x_shape[2:] * prod(w_shape) * batch_size). Args: x_shape (list(int)): The input shape before convolution. w_shape (list(int)): The filter shape. out_shape (list(int)): The output shape after convolution. transposed (bool): is the convolution transposed Returns: int: the number of flops rrCNr) rWrXr%rY batch_size conv_shapec_outc_in filter_sizerRs rconv_flop_countr`\s[*J''Y;J 'E+  d;/ /* .ts$a%(#d59o55r rrFrb)rr`)grad_out_shaperWrXrcrdrerfrY_output_padding_groups output_maskr%rk flop_countgrad_input_shapegrad_weight_shapes rconv_backward_floprts6JDL1~$Yq\2on?OQ_`` 1~%il3  /!N*;QwZK\I]jop pJ  /!G*a6GK\I]jop pJ r cUup4pVUupxpUuppX7s=:XaU :Xa#O eXHs=:XaU :XaO eXj:Xa X:XaXj:XdeSnU[X4-XV4X4-Xi45- nU[X4-XY4X4-X45- nU$)zR Count flops for self-attention. NB: We can assume that value_shape == key_shape rrU) query_shape key_shape value_shaperPhs_qd_q_b2_h2s_k_d2_b3_h3_s3d_v total_flopss rsdpa_flop_countrs !NA#"Cc$Cc ?s?[ [q3[ [3:#*QTQ[[ [K8QUC-s/@AAK8QUC-s/@AAK r c[XU5$)Count flops for self-attention.r)rvrwrxr%r&r's r sdpa_floprs ;; ??r cSSKJn SSKJn [ XU45(dUR 5R 5$U/URS5S- -$)z If the offsets tensor is fake, then we don't know the actual lengths. In that case, we can just assume the worst case; each batch has max length. r) FakeTensor)FunctionalTensorr)torch._subclasses.fake_tensorr#torch._subclasses.functional_tensorrrdifftolistsize)offsetsmax_lenrrs r_offsets_to_lengthsr sJ 9D g,<= > >||~$$&& 9 Q!+ ,,r )grad_out.c#h# Ub[UR5S:Xde[UR5S:XdeUbURUR:XdeURupn URupn URupnUceUceURUR:Xde[XF5n[XW5n[UU5H'unnSU UU 4nSU UU 4nSU UU4nUbUOSnUUUU4v M) gURURURUb UROS4v g7f)a' Given inputs to a flash_attention_(forward|backward) kernel, this will handle behavior for NestedTensor inputs by effectively unbinding the NestedTensor and yielding the shapes for each batch element. In the case that this isn't a NestedTensor kernel, then it just yields the original shapes. Nrlenrrzip)querykeyvaluer cum_seq_q cum_seq_kmax_qmax_k_h_qr{h_kd_kh_vr seq_q_lengths seq_k_lengths seq_q_len seq_k_lennew_query_shape new_key_shapenew_value_shapenew_grad_out_shapes r%_unpack_flash_attention_nested_shapesrsJ$ 399~"""5;;1$$$8>>U[[#@@@kk ii kk $$$$$$)//111+I= +I= &)-&G "Y  #y#6OY4M #y#6O4<4Hd !=/CUU U 'H  ++syy%++AUx~~[_ __sD0D2c#n# Ub[UR5S:Xde[UR5S:XdeUbURUR:XdeURu pn URu pn URu pnUceUceURUR:Xde[XF5n[XW5n[UU5H'unnSU UU 4nSU UU 4nSU UU4nUbUOSnUUUU4v M) gURURURUb UROS4v g7f)a+ Given inputs to a efficient_attention_(forward|backward) kernel, this will handle behavior for NestedTensor inputs by effectively unbinding the NestedTensor and yielding the shapes for each batch element. In the case that this isn't a NestedTensor kernel, then it just yields the original shapes. Nrr)rrrr cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_krrr{rrrr seqlens_q seqlens_klen_qlen_krrrrs r)_unpack_efficient_attention_nested_shapesrFsR$399~"""5;;1$$$8>>U[[#@@@131313''''''!!\%7%7777' C ' C  95LE5 #uc2OUC0M #uc2O4<4Hd !=/CUU U 6  ++syy%++AUx~~[_ __sD3D5T)r<c D[UUUUUUUS9n [SU 55$)r)rrrrrrrc3@# UHupp4[XU5v M g7frr.0rvrwrxrs r 0_flash_attention_forward_flop..&6; 2KK  <<6;rsum) rrrrrrrr%r&r'sizess r_flash_attention_forward_floprvs?" 2  E 6; r c D[UUUUUUUS9n [SU 55$)r)rrrrrrrc3@# UHupp4[XU5v M g7frrrs rr4_efficient_attention_forward_flop..rrrr) rrrbiasrrrrr&r'rs r!_efficient_attention_forward_floprs?" 6 !!!! E 6; r cSnUupVpxUuppUupnnUunnnnXYs=:Xa U s=:XaU:Xa O eXjs=:Xa Us=:XaU:Xa O eX:XdeUU:Xa X:XaUU:XdeSnU[XV-Xx4XV-X45- nU[XV-UU4XV-UU 45- nU[XV-X4XV-UU45- nU[XV-X{4XV-X45- nU[XV-X4XV-X{45- nU$)NrrU)rmrvrwrxrrPryrzr{r|r}r~rrrrr_b4_h4_s4_d4s rsdpa_backward_flop_countrs2K NA#"Cc$Cc3'Cc3 !s !c !K Ka&<#&<&<K KK K #:#*3 3K8QUC-s/@AAK8QUC-sC/@AAK8QUC-sC/@AAK8QUC-s/@AAK8QUC-s/@AAK r c[XX#5$)z(Count flops for self-attention backward.r)rmrvrwrxr%r&r's rsdpa_backward_floprs $N XXr c F[UUUUUUUU S9n [SU 55$)N)rrrrrrrrc3@# UHupp4[XAX#5v M g7frrrrvrwrxrms rr1_flash_attention_backward_flop..&CI ?KK !iUUCIrr) rrrrout logsumexprrrrr&r'shapess r_flash_attention_backward_floprsB"3  F CI r c F[UUUUUUUU S9n [SU 55$)N)rrrrrrrrc3@# UHupp4[XAX#5v M g7frrrs rr5_efficient_attention_backward_flop..rrr) rrrrrrrrrrr&r'rs r"_efficient_attention_backward_floprsB"7 !!!! F CI r c6[U[5(dU4$U$r)rtuple)xs rnormalize_tupler.s a  t Hr )KMBTc [S[[[5S- [[ U55S- S-55n[U$)NrrrCr)maxminrsuffixesstr)numberindexs rget_suffix_strr7s= 3s8}q(3s6{+;a+?A*EF GE E?r cX[RU5nUSU-- SnU[U-$)Niz.3f)rr)rsuffixrrs rconvert_num_with_suffixr>s2 NN6 "E %c*E 8E? ""r cUS:XagX- S$)Nr0%z.2%r@)numdenoms rconvert_to_percent_strrEs zk# r c0^[T5U4Sj5nU$)Nc>>[U5upT"U6n[X25$r)rr)r& flat_argsspecrr(s rr))_pytreeify_preserve_structure..nfKs#&t, mc((r rr+s` r_pytreeify_preserve_structurerJs  1X)) Ir c ^\rSrSrSrSS\\\RR\ \RR4S\ S\ S\\ \\44U4SjjjrS\ 4S jrS\ \\ \\ 444S jrSS jrS rS rSrSrU=r$)riTa ``FlopCounterMode`` is a context manager that counts the number of flops within its context. It does this using a ``TorchDispatchMode``. It also supports hierarchical output by passing a module (or list of modules) to FlopCounterMode on construction. If you do not need hierarchical output, you do not need to use it with a module. Example usage .. code-block:: python mod = ... with FlopCounterMode(mod) as flop_counter: mod.sum().backward() modsdepthdisplaycustom_mappingc n>[TU]5 [S5UlX lX0lSUlUc0nUb[R"SSS9 0[EUR5VVs0sH%upVU[USS5(aUO [U5_M' snnEUl [5Ulgs snnf)Nc [[5$r)rintr@r r*FlopCounterMode.__init__..os +VYJZr z[T RUR55nT UT:-m SU-n/nURX0-[ UT5[ UT5/5 T RUR 5H<upVURUS-[U5-[ UT5[ UT5/5 M> U$)N z - )rrrappendrrrr) mod_namerrpaddingrrGr global_flops global_suffixis_global_subsumedrs r process_mod.FlopCounterMode.get_table..process_modsd..x8??ABK +"= = EkGF MM"' ]C&{LA  ((288: eOc!f,+A}=*1l;; Mr r.rr))r0r)leftrightr5)headerscolalign) rtabulatePRESERVE_WHITESPACErrsortedrkeyscountextendr) rrr8headerrr0mod mod_depth cur_valuesrr-r.r/s ` @@@r get_tableFlopCounterMode.get_tables# =JJE =E'+$.++- &|4 "  ,$**//12Ch #*I $Sa-8J MM* %3 t'' '0Bq>a !1-6F v;! +,F  B\ ]]r cURR5 URR5 [ U5UlUR R5 U$r)rclearr __enter___FlopCounterModerrs rrFFlopCounterMode.__enter__sG   ""$$T*   r cURceURR"U6nSUlURR5 UR(a$[ UR UR 55 U$r)r__exit__rrprintrBr)rr&rPs rrJFlopCounterMode.__exit__s`yy$$$ II   %  !!# << $.., -r cXR;a[URUnU"U0UDSU0D6n[URR5HnURUU==U- ss'M U$)Nr#)r!setrparentsr)r func_packetrr&r'flop_count_funcrqpars r _count_flopsFlopCounterMode._count_flopsso ,, ,"00=O($F&F#FJ4++334  %k2j@25 r )rrrr!rr)NrCTNr)__name__ __module__ __qualname____firstlineno____doc__r r rnnr&rr boolr r rrrr#rBrFrJrS__static_attributes__ __classcell__)rs@rrrTs*MQ 7; +5$uxx2G!GHI++ + %T#s(^4 ++*88 Ac4S>&9!: A:^zr c,\rSrSrS\4SjrSSjrSrg)rGicountercXlgrr_)rr_s rr_FlopCounterMode.__init__s r NcU(aUO0nU[RRRR[RRRR [RRR R[RRRR[RRRR[RRRR[RRRR[RRRR[RRRR[RRRR[RRRR[RRRR[RRR R[RR"R$R1;a[&$XR(R*;acU[RR"R,RLa2U UR."U0UD6nU[&La UsSSS5 $SSS5 U"U0UD6nUR(R1UR2XcU5$!,(df  N==fr)ropsaten is_contiguousdefault memory_formatis_strides_like_formatis_non_overlapping_and_densersym_sizestride sym_stridestorage_offsetsym_storage_offsetnumel sym_numeldimprimlayoutNotImplementedr_r!device decomposerS_overloadpacket)rfunctypesr&r'rrs r__torch_dispatch__#_FlopCounterMode.__torch_dispatch__s!r EIINN0088IINN00>>IINN99AAIINN??GGIINN''//IINN++33IINN))11IINN--55IINN1199IINN55==IINN((00IINN,,44IINN&&..IINN))11 3 3" ! ||11 1d%))..BWBWB_B_6_NND3F3N** D#F#||(()=)=s&QQs L99 Mra)r@N)rUrVrWrXrrr|r\r@r rrGrGsRr rG)Fr)Mrtorch.utils._pytreerrrmodule_trackerrtypingrr r r r r r collectionsrtorch.utils._python_dispatchrmathr functoolsrr__all__rdrerr!__annotations__r,rmmr rJaddmmrNbmmrSbaddbmmrVr[r` convolution _convolutionrgconvolution_backwardrtr'_scaled_dot_product_efficient_attention#_scaled_dot_product_flash_attention#_scaled_dot_product_cudnn_attentionrrrr_flash_attention_forwardr_efficient_attention_forwardrr0_scaled_dot_product_efficient_attention_backward,_scaled_dot_product_flash_attention_backward,_scaled_dot_product_cudnn_attention_backwardr_flash_attention_backwardr_efficient_attention_backwardrrrrrrrrrGr@r rrs FF)DDD#: 5 6 yy~~ !# tCH~",tww/3# tzz"%#%#%txx  C ! t||$&C&%& % #Y% #Y%Cy% %  %N(($*;*;<=bfOuxO>O t001ee2eN$DD@@@@BCEI@WZ@C@ -" +`eE#s(OU38_eCHoxPUVY[^V^P_G``ab+`f -`eE#s(OU38_eCHoxPUVY[^V^P_G``ab-``t44dC D>t88$G H>6MMIIIIKL^bYpsYLYt55tD E@t994H I@GGWJJ  HHh LL,   i   y  1 00) ,,i ,,i 99;M 557I 557I !!#@ %%'H  ""$B!" &&(J# (  $# LL^"R("Rr