RЦimjSrSSKJr SSKJrJrJrJrJrJ r SSK r SSK J r SSK JrJr SSKJrJrJrJrJrJrJrJr SSKJr SS KJr SS KJrJr SS K J!r!J"r" S /r#"S S\ RH5r%"SS\ RH5r&"SS\ RH5r'"SS\ RH5r("SS \ RH5r)S1Sjr*S2Sjr+\!"\+"SS9\+"SS9\+"SS9\+"SS9\+"SS9\+"SS9\+"SSSSSS9\+"SSSSSS9\+"SSSSSS9\+"SSSSSS9\+"SSS9\+"SSS9S . 5r,S!\)4S"jr-S3S#jr.\"S3S$\)4S%jj5r/\"S3S$\)4S&jj5r0\"S3S$\)4S'jj5r1\"S3S$\)4S(jj5r2\"S3S$\)4S)jj5r3\"S3S$\)4S*jj5r4\"S3S$\)4S+jj5r5\"S3S$\)4S,jj5r6\"S3S$\)4S-jj5r7\"S3S$\)4S.jj5r8\"S3S$\)4S/jj5r9\"S3S$\)4S0jj5r:g)4aZFocalNet As described in `Focal Modulation Networks` - https://arxiv.org/abs/2203.11926 Significant modifications and refactoring from the original impl at https://github.com/microsoft/FocalNet This impl is/has: * fully convolutional, NCHW tensor layout throughout, seemed to have minimal performance impact but more flexible * re-ordered downsample / layer so that striding always at beginning of layer (stage) * no input size constraints or input resolution/H/W tracking through the model * torchscript fixed and a number of quirks cleaned up * feature extraction support via `features_only=True` )partial)CallableListOptionalTupleTypeUnionNIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)MlpDropPath LayerNorm2d LayerScale2d trunc_normal_ClassifierHeadNormMlpClassifierHeadcalculate_drop_path_rates)build_model_with_cfg)feature_take_indices) named_apply checkpoint)generate_default_cfgsregister_modelFocalNetc^\rSrSrSSSSS\SS4S\S\S \S \S \S \S \S\S\\ R4U4Sjjjr Sr Sr U=r$)FocalModulation-TFNdim focal_window focal_level focal_factorbias use_post_normnormalize_modulator proj_drop norm_layerc >XS.n [TU]5 XlX lX0lX@lX`lXplXURS-/Ul[R"USU-URS--4SUS.U D6Ul [R"X4SUS.U D6Ul [R"5Ul[R"X4SS0U D6Ul[R "U5Ul[R$"5Ul/Ul[+UR5Hn UR U -UR-nUR&R-[R."[R"X4XUS-SS.U D6[R"555 UR(R-U5 M UR (aU "U40U D6Ulg[R0"5Ulg)Ndevicedtyperr ) kernel_sizer&r/F)r/groupspaddingr&)super__init__r"r#r$r%r'r( input_splitnnConv2dfhGELUactprojDropoutr) ModuleList focal_layers kernel_sizesrangeappend SequentialIdentitynorm)selfr"r#r$r%r&r'r(r)r*r-r.ddkr/ __class__s S/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/models/focalnet.pyr3FocalModulation.__init__.s / (&(*#6 d&6&6&:;3C4+;+;a+? @aaVZa^`a3DDD779IIcr:meanr$r(r8rDr;r)) rExqctxgatesctx_alll focal_layer ctx_globalx_outs rIforwardFocalModulation.forwardYs  FF1I A'7'7; '(9(9:NAc"CeAqQwJ&7 77G;XXchhvth<= q$2B2B2C/C)DDD  # #!1!1A!56GFF7O# %  % u% rK)r:r"r7r%r>r$r#r8r4r?rDr(r;r)r')__name__ __module__ __qualname____firstlineno__rintboolfloatrr5Moduler3r\__static_attributes__ __classcell__rHs@rIrr-s !""'(-!*5)S)S)S )S  )S  )S )S"&)S)SRYY)S)SVrKrc^\rSrSrSrSSSSSSSSS\R \S S 4 S \S \ S \S \S\ S\ S\ S\ \ S\ S\ S\ \RS\ \R4U4SjjjrSrSrU=r$) FocalNetBlocktz$Focal Modulation Network Block. @rrMF-C6?r!Nr" mlp_ratior$r#r'use_post_norm_in_modulationr(layerscale_valuer) drop_path act_layerr*c >XS.n[TU]5 XlX lX@lX0lXPlU(d U "U40UD6O[R"5Ul [U4UUR UUU U S.UD6Ul U(a U "U40UD6O[R"5Ul Ub [X40UD6O[R"5UlU S:a [U 5O[R"5UlU(d U "U40UD6O[R"5Ul[%SU['X-5U U SS.UD6UlU(a U "U40UD6O[R"5UlUb [X40UD6O[R"5UlU S:a[U 5Ulg[R"5Ulg)a Args: dim: Number of input channels. mlp_ratio: Ratio of mlp hidden dim to embedding dim. focal_level: Number of focal levels. focal_window: Focal window size at first focal level. use_post_norm: Whether to use layer norm after modulation. use_post_norm_in_modulation: Whether to use layer norm in modulation. layerscale_value: Initial layerscale value. proj_drop: Dropout rate. drop_path: Stochastic depth rate. act_layer: Activation layer. norm_layer: Normalization layer. r,)r#r$r'r(r)r*Nr!T) in_featureshidden_featuresrrdropuse_conv)r2r3r"rnr#r$r'r5rCnorm1r modulation norm1_postrls1r drop_path1norm2r rbmlp norm2_postls2 drop_path2)rEr"rnr$r#r'ror(rpr)rqrrr*r-r.rFrHs rIr3FocalNetBlock.__init__xs> / "(&*2?Z*r*R[[] )   %((5 3!    4A*S/B/bkkm@P@\<<UUS.n[TU]5 XlX0lSUlU(a[ SUUSUUS.UD6UlO[R"5Ul[R"[U5Vs/sH4n[SUUUUU U U U U [U[5(aUUOUUS. UD6PM6 sn5Ulgs snf)a Args: dim: Number of input channels. out_dim: Number of output channels. depth: Number of blocks. mlp_ratio: Ratio of mlp hidden dim to embedding dim. downsample: Downsample layer at start of the layer. focal_level: Number of focal levels focal_window: Focal window size at first focal level use_overlap_down: User overlapped convolution in downsample layer. use_post_norm: Whether to use layer norm after modulation. use_post_norm_in_modulation: Whether to use layer norm in modulation. layerscale_value: Initial layerscale value proj_drop: Dropout rate for projections. drop_path: Stochastic depth rate. norm_layer: Normalization layer. r,Fr )in_chsout_chsstrideoverlapr*) r"rnr$r#r'ror(rpr)rqr*Nrx)r2r3r"rgrad_checkpointing Downsamplerr5rCr=r@rj isinstancelistblocks)rEr"rrrnrr$r#rr'ror(rpr)rqr*r-r.rFirHs rIr3FocalNetStage.__init__sJ /  "' ((%   DO!kkmDOmm5\%#"  #')+,G$7!1#*4Y*E*E)A,9%  "%#$ %#s=;CcXlgr)r)rEenables rIset_grad_checkpointing$FocalNetStage.set_grad_checkpointings"(rKcURU5nURHLnUR(a0[RR 5(d [ X!5nMDU"U5nMN U$r)rrrrOjit is_scriptingr)rErSblks rIr\FocalNetStage.forwardsS OOA ;;C&&uyy/E/E/G/Gs&F  rK)rrr"rrT)r^r_r`rarrrbrdrcrr rrr5rer3rOrignorerr\rfrgrhs@rIrrs& "# !%*"'05(-04!35*5%G$G$G$ G$  G$  G$G$G$#G$ G$*.G$"&G$'uoG$G$UDK/0G$ RYY!G$G$R YY))rKrc t^\rSrSrS S\S\S\S\S\\\R4 U4Sjjjr Sr S r U=r $) ri&rrrrr*c>XgS.n[T U]5 X0lSn Un U(aUS;deUS:XaSupO US:XaSup[R"X4XU S.UD6UlUbU"U40UD6Ulg [R "5Ulg ) z Args: in_chs: Number of input image channels. out_chs: Number of linear projection output channels. stride: Downsample stride. overlap: Use overlapping convolutions if True. norm_layer: Normalization layer. r,r)r r)r r )rMr)r/rr1N)r2r3rr5r6r;rCrD) rErrrrr*r-r.rFr1r/rHs rIr3Downsample.__init__(s& /   V# ##{'+$ W1'+$ IIfm;_fmjlm 1;1GJw-"- R[[] rKcJURU5nURU5nU$r)r;rDrErSs rIr\Downsample.forwardIs! IIaL IIaLrK)rDr;r)rFNNN)r^r_r`rarbrcrrr5rer3r\rfrgrhs@rIrr&sm !48[[[ [  [ !bii1 [[BrKrc( ^\rSrSrSrSSSSSSS S S S S S S S S SSS\"\SS9S S 4S\S\S\S\S\ \S4S\ S\ \S4S\ \S4S\ S\ S\ S\ S\ \S \ S!\ \ S"\ S#\ S$\ S%\ \R4&U4S&jjjr\R$R&S'5r\R$R&SS\S\ \4S,jjrS?S-\R2S.\ \\\\4S/\ S0\ S1\S2\ S*\\\R2\ \R2\\R2444S3jjrS@S.\\\\4S4\ S5\ 4S6jjrS7rS)epsin_chans num_classes global_pool embed_dimdepths.rn focal_levels focal_windowsrr'ror(head_hidden_sizehead_init_scalerp drop_rateproj_drop_ratedrop_path_rater*c>[TU]5 UUS.n[U5Ul[ UR5Vs/sH nUSU--PM nnX lXlX@lUS=UlUl /Ul [SUUSU US.UD6Ul USn[U[U55n/n[ UR5H}nUUn[SUUUUUUS:UUUUU U U U UUU[USU5[USUS-5US.UD6nUnUU/- nU=R[!US SU--S U3S 9/- sl M ["R$"U6UlU (aB["R("5UlXl [-URU4U UUUS .UD6UlO7U"UR40UD6Ul[1URU4UUS .UD6Ul[3[5[6US9U5 gs snf)a Args: in_chans: Number of input image channels. num_classes: Number of classes for classification head. embed_dim: Patch embedding dimension. depths: Depth of each Focal Transformer layer. mlp_ratio: Ratio of mlp hidden dim to embedding dim. focal_levels: How many focal levels at all stages. Note that this excludes the finest-grain level. focal_windows: The focal window size at all stages. use_overlap_down: Whether to use convolutional embedding. use_post_norm: Whether to use layernorm after modulation (it helps stabilize training of large models) layerscale_value: Value for layer scale. drop_rate: Dropout rate. drop_path_rate: Stochastic depth rate. norm_layer: Normalization layer. r,r r)rrrr*Nr)r"rrrnrr$r#rr'ror(rpr)rqr*rlayers.)num_chs reductionmodule) hidden_size pool_typerr*)rr)rrx)r2r3len num_layersr@rrr num_featuresr feature_inforstemrsumrdictr5rBlayersrCrDrheadrrr _init_weights)rErrrrrrnrrrr'ror(rrrprrrr*r-r.rFrin_dimdprri_layerrlayerrHs rIr3FocalNet.__init__Ss_P  /f+383IJ3IaY!q&)3I J& "4=bMAD1 aL$!     1'F DT__-G(G!Wo#"Q;(1*73!1+,G$7!1(c&'"23C|! 8L4MN% !E$F ug F   $w!a7lBR]deldm[n"o!p p -.0mmV,  DI$4 !-!!-%#% DI#4#4#4;;DI&!!&#   DI GM?KTRGKsG5cS1$)NrxrEs rIno_weight_decayFocalNet.no_weight_decays t rKc4[SU(aSS/S9$/SQS9$)Nz^stem)z^layers\.(\d+)Nz^norm)i))z^layers\.(\d+).downsample)r)z^layers\.(\d+)\.\w+\.(\d+)Nr)rr)r)rEcoarses rI group_matcherFocalNet.group_matchers3*$    rKcTXlURHnURUS9 M g)N)r)rrr)rErrXs rIrFocalNet.set_grad_checkpointings'"(A $ $F $ 3rKreturnc.URR$r)rfcrs rIget_classifierFocalNet.get_classifiersyy||rKcBXlURRXS9 g)N)r)rrreset)rErrs rIreset_classifierFocalNet.reset_classifiers&  ;rKrSindicesrD stop_early output_fmtintermediates_onlyc4US;dS5e/n[[UR5U5upURU5n[R R 5(dU(d URn OURSU S-n [UR5S- n [U 5HEupU "U5nX;dMU(aX:XaURU5nOUnURU5 MG U(aU$W U :XaURU5nX4$)aForward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence norm: Apply norm layer to compatible intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features Returns: )NCHWzOutput shape must be NCHW.Nr) rrrrrOrrrQrDrA)rErSrrDrrr intermediates take_indices max_indexstageslast_idxfeat_idxstagex_inters rIforward_intermediatesFocalNet.forward_intermediatess*Y&D(DD& "6s4;;7G"Q  IIaL 99 ! ! # #:[[F[[)a-0Ft{{#a'(0OHaA'H0"iilGG$$W- 1  x  ! ArK prune_norm prune_headc[[UR5U5upEURSUS-UlU(a[R"5UlU(aUR SS5 U$)z?Prune layers not required for specified intermediates. Nrrr)rrrr5rCrDr)rErrrrrs rIprune_intermediate_layers"FocalNet.prune_intermediate_layerss[#7s4;;7G"Q kk.9q=1  DI   ! !!R (rKclURU5nURU5nURU5nU$r)rrrDrs rIforward_featuresFocalNet.forward_features%s. IIaL KKN IIaLrK pre_logitscPU(aURXS9$URU5$)N)r)r)rErSrs rI forward_headFocalNet.forward_head+s#6@tyyy2RdiiPQlRrKcJURU5nURU5nU$r)rr rs rIr\FocalNet.forward.s'  ! !! $   a rK) rrrrrrrrDrrrrFrr)NFFrF)rFT)#r^r_r`rarrrrbstrrrdrcrrr5rer3rOrrrrrrrTensorr rrrrr r\rfrgrhs@rIrrOs #$&2!,8-9%*"'05(-.2%(04!$&$'*1+4*H-nSnSnS nS  nS #s(O nSnS S/nS!c?nS#nS nS*.nS"&nS'smnS#nS 'uo!nS"#nS$"%nS&"'nS(RYY)nSnS` YY YY     YY44  YY *   S$SrKrc`[U[R5(aL[URSS9 UR b*[R RUR 5 gg[U[R5(a[URSS9 UR b)[R RUR 5 U(aRSU;aKURRRU5 UR RRU5 gggg)Ng{Gz?)stdhead.fc) rr5r6rweightr&initzeros_Lineardatamul_)rnamers rIrr4s&"))$$fmm- ;; " GGNN6;; ' # FBII & &fmm- ;; " GGNN6;; ' I% MM   # #O 4 KK   ! !/ 2&4 'rKc 2USSSSS[[SSSS . UE$) Nr)rMr)rrg?bicubicz stem.projrmit) urlr input_size pool_sizecrop_pct interpolationrRr first_conv classifierlicenser )rkwargss rI_cfgr(Bs3=v%.B!   #  rKztimm/) hf_hub_id)rMr*) r+iRU)r)r r!r"r)r)r) zfocalnet_tiny_srf.ms_in1kzfocalnet_small_srf.ms_in1kzfocalnet_base_srf.ms_in1kzfocalnet_tiny_lrf.ms_in1kzfocalnet_small_lrf.ms_in1kzfocalnet_base_lrf.ms_in1kzfocalnet_large_fl3.ms_in22kzfocalnet_large_fl4.ms_in22kzfocalnet_xlarge_fl3.ms_in22kzfocalnet_xlarge_fl4.ms_in22kzfocalnet_huge_fl3.ms_in22kzfocalnet_huge_fl4.ms_in22kmodelcZURSU5nSU;aU$SSKn0nUR5nUR5HupVUR SSU5nUR SS5nUR SS U5nS U;aXT;aUR S S U5nUR S S5nUR SS5nXT;a]XER 5UR 5:Xa9XERUR:waURXER5nXcU'M U$)Nr,zstem.proj.weightrz gamma_([0-9])z ls\1.gamma patch_embedrzlayers.(\d+).downsamplecDS[URS55S-S3$)Nrrz .downsample)rbgroup)rSs rI&checkpoint_filter_fn..zs!73qwwqz?UVCVBWWb9crKrDz norm([0-9])z norm\1_postzln.znorm.rr) getre state_dictitemssubreplacenumelshapereshape)r5r,r4out_dict dest_dictrGvs rIcheckpoint_filter_fnr?ps4JZ' H  "I  " FF#]A 6 IImV , FF-/cef g Q;1-~~q9A IIeW % IIfi ( >il002aggi?ILDVDVZ[ZaZaDa ),,,-A # OrKc [S[URSS5555nURSU5n[ [ X4[ [SUS9S.UD6nU$)Nc3*# UH upUv M g7frrx).0r_s rI #_create_focalnet..s\.[da.[sr)rrrMr out_indicesT)flatten_sequentialrF)pretrained_filter_fn feature_cfg)tuplerQr3poprrr?r)variant pretrainedr'default_out_indicesrFr,s rI_create_focalnetrOsf\i 8\8Z.[\\**],?@K ' 1DkJ    E LrKrc <[S/SQSS.UD6n[SSU0UD6$)NrrrrrMrx)focalnet_tiny_srfrrOrMr' model_kwargss rIrRrRs)D|rDVDL  WJ W, WWrKc <[S/SQSS.UD6n[SSU0UD6$)Nr r r rrQrMrx)focalnet_small_srfrSrTs rIrYrYs)E}EfEL  XZ X< XXrKc <[S/SQSS.UD6n[SSU0UD6$)NrWrQrMrx)focalnet_base_srfrSrTs rIr\r\s)F}FvFL  WJ W, WWrKc B[S/SQS/SQS.UD6n[SSU0UD6$)NrrrrrrrMrx)focalnet_tiny_lrfrSrTs rIr_r_s,_|r _X^_L  WJ W, WWrKc B[S/SQS/SQS.UD6n[SSU0UD6$)NrWrrr^rMrx)focalnet_small_lrfrSrTs rIraras,`}`Y_`L  XZ X< XXrKc B[S/SQS/SQS.UD6n[SSU0UD6$)NrWr[rr^rMrx)focalnet_base_lrfrSrTs rIrcrcs,a},aZ`aL  WJ W, WWrKc R[S /SQS/SQS/S-SSSS.UD6n[S S U0UD6$) NrWrrTrmrrrrr'rrprMrx)focalnet_large_fl3rSrTs rIrhrhsKT,WXVY\]V]TDTLRTL  XZ X< XXrKc H[S/SQS/SQSSSS.UD6n[S SU0UD6$) NrWrerrrrTrmrrrr'rrprMrx)focalnet_large_fl4rSrTs rIrlrls?T,TDTLRTL  XZ X< XXrKc R[S /SQS/SQS/S-SSSS.UD6n[S S U0UD6$) NrWrrfrTrmrgrMrx)focalnet_xlarge_fl3rSrTs rIrorosKT,WXVY\]V]TDTLRTL  Yj YL YYrKc H[S/SQS/SQSSSS.UD6n[S SU0UD6$) NrWrnrjTrmrkrMrx)focalnet_xlarge_fl4rSrTs rIrqrqs?T,TDTLRTL  Yj YL YYrKc T[S /SQS/SQS/S-SSSSS.UD6n[S S U0UD6$) NrW`rrMrTrm)rrrrr'rorrprMrx)focalnet_huge_fl3rSrTs rIrtrtsNv,WXVY\]V]tfjvntvL  WJ W, WWrKc J[S/SQS/SQSSSSS.UD6n[S SU0UD6$) NrWrsrjTrm)rrrr'rorrprMrx)focalnet_huge_fl4rSrTs rIrvrvsBv,tfjvntvL  WJ W, WWrK)Nr)rr);r functoolsrtypingrrrrrr rOtorch.nnr5 timm.datar r timm.layersr rrrrrrr_builderr _featuresr _manipulaterr _registryrr__all__rerrjrrrrr( default_cfgsr?rOrRrYr\r_rarcrhrlrorqrtrvrxrKrIrs &?? A   ++0< ,DbiiDNTBIITnXBIIXv&&RbryybJ 3%!%""&#!%"!%""&#!%"$( HsPU$W$( HsPU$W%) HsPU%W%) HsPU%W#'##'#; &  FH* XXXX YhYY XXXX XXXX YhYY XXXX YhYYYhYYZxZZZxZZXXXXXXXXrK