RЦi& SrSSKrSSKrSSKJrJrJrJrJrJ r J r J r J r SSK r SSKJr SSKJrJr SSKJrJrJrJrJrJrJrJrJrJrJrJr SSK J!r! SSK"J#r# SS K$J%r% SS K&J'r'J(r( SS K)J*r*J+r+J,r, SS K-J.r. S /r/\R`"\15r2\ \3\ \3\344r4S\ RjS\ \3\34S\ Rj4Sjr6\%S\ RjS\ \3\34S\3S\3S\ Rj4 Sj5r7SbS\3S\3S\ Rj4Sjjr8"SS\Rr5r:"SS\Rr5r;"SS\Rr5r<"SS \Rr5r="S!S \Rr5r>S"\?S#\RrS\\@\ Rj44S$jrAScS%\@S&\BS\>4S'jjrCSdS(\@S\\@\44S)jjrD\*"0S*\D"S+S,S-9_S.\D"S+S/S-9_S0\D"S+S1S2S3S4S59_S6\D"S+S7S-9_S8\D"S+S9S2S3S4S59_S:\D"S+S;S-9_S<\D"S+S=S-9_S>\D"S+S?S-9_S@\D"S+SAS2S3S4S59_SB\D"S+SCS-9_SD\D"S+SESFSG9_SH\D"S+SISFSG9_SJ\D"S+SKSFSG9_SL\D"S+SMS2S3S4SFSN9_SO\D"S+SPSFSG9_SQ\D"S+SRS2S3S4SFSN9_SS\D"S+STS-9_\D"S+SUS-9\D"S+SVS-9SW.E5rE\+ScS\>4SXjj5rF\+ScS\>4SYjj5rG\+ScS\>4SZjj5rH\+ScS\>4S[jj5rI\+ScS\>4S\jj5rJ\+ScS\>4S]jj5rK\+ScS\>4S^jj5rL\+ScS\>4S_jj5rM\+ScS\>4S`jj5rN\,"\1SJSLSOSQSa.5 g)eaSwin Transformer A PyTorch impl of : `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows` - https://arxiv.org/pdf/2103.14030 Code/weights from https://github.com/microsoft/Swin-Transformer, original copyright/license info below S3 (AutoFormerV2, https://arxiv.org/abs/2111.14725) Swin weights from - https://github.com/microsoft/Cream/tree/main/AutoFormerV2 Modifications and additions for timm hacked together by / Copyright 2021, Ross Wightman N) AnyDictCallableListOptionalSetTupleUnionTypeIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) PatchEmbedMlpDropPathcalculate_drop_path_ratesClassifierHead to_2tuple to_ntuple trunc_normal_use_fused_attnresize_rel_pos_bias_tableresample_patch_embedndgrid)build_model_with_cfg)feature_take_indices)register_notrace_function)checkpoint_seq named_apply)generate_default_cfgsregister_modelregister_model_deprecations)get_init_weights_vitSwinTransformerx window_sizereturncURup#pEURX#US-USXAS-USU5nURSSSSSS5R5RSUSUSU5nU$)zPartition into non-overlapping windows. Args: x: Input tokens with shape [B, H, W, C]. window_size: Window size. Returns: Windows after partition with shape [B * num_windows, window_size, window_size, C]. rrshapeviewpermute contiguous)r&r'BHWCwindowss [/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/models/swin_transformer.pywindow_partitionr:*sJA! q{1~%{1~qN7JKXYN\]^Aii1aAq)446;;B AP[\]P^`abG Nr8r5r6cURSnURSX!S-X1S-USUSU5nURSSSSSS5R5RSX#U5nU$)zReverse window partition. Args: windows: Windows with shape (num_windows*B, window_size, window_size, C). window_size: Window size. H: Height of image. W: Width of image. Returns: Tensor with shape (B, H, W, C). r.rrr*r+r,r-r/)r8r'r5r6r7r&s r9window_reverser==sy  bA R!n,aq>.A;q>S^_`SacdeA !Q1a#..055b!BA Hr;win_hwin_wc [R"[[R"X[RS9[R"X[RS955n[R "US5nUSS2SS2S4USS2SSS24- nUR SSS5R5nUSS2SS2S4==US- - ss'USS2SS2S4==US- - ss'USS2SS2S4==SU-S- -ss'URS5$)zGet pair-wise relative position index for each token inside the window. Args: win_h: Window height. win_w: Window width. Returns: Relative position index tensor. devicedtyperNr+rr.) torchstackrarangelongflattenr2r3sum)r>r?rBcoordscoords_flattenrelative_coordss r9get_relative_position_indexrMPs[[ U< U<F]]61-N$Q4Z0>!T1*3MMO%--aA6AACOAq!G )Aq!G )Aq!GE A -   r ""r;cV^\rSrSr%Sr\R R\\ S'SS\ S\ S\ \ S\ S \S \ S \ 4U4S jjjrSSjrSSjrS\\ \ 4S S4SjrS \R&4SjrSS\R&S\ \R&S \R&4SjjrSSjrSrU=r$)WindowAttentionhzWindow based multi-head self attention (W-MSA) module with relative position bias. Supports both shifted and non-shifted windows. fused_attnNdim num_headshead_dimr'qkv_bias attn_drop proj_dropc  >XS.n [TU]5 Xl[U5UlURupX-UlX lU=(d X-nX2-n US-Ul[SS9Ul [R"[R"SU -S- SU -S- -U40U D65UlURS[R"X-X-U[R S9SS 9 [R""XS -4S U0U D6Ul[R&"U5Ul[R""X40U D6Ul[R&"U5Ul[R."S S 9UlUR35 g)an Args: dim: Number of input channels. num_heads: Number of attention heads. head_dim: Number of channels per head (dim // num_heads if not set) window_size: The height and width of the window. qkv_bias: If True, add a learnable bias to query, key, value. attn_drop: Dropout ratio of attention weight. proj_drop: Dropout ratio of output. rAgT) experimentalr+rrelative_position_indexF persistentr*biasr.)rRN)super__init__rRrr' window_arearSscalerrQnn ParameterrDemptyrelative_position_bias_tableregister_bufferrGLinearqkvDropoutrVprojrWSoftmaxsoftmaxreset_parameters)selfrRrSrTr'rUrVrWrBrCddr>r?attn_dim __class__s r9r_WindowAttention.__init__osW, / $[1''  ="/s/'% (d;-/LL KKUQ1u9q=99 K K-M)  % KK u}V5:: V  99SQ,DXDDI.IIh2r2 I.zzb)  r;r(cL[URSS9 UR5 g)"Initialize parameters and buffers.g{Gz?)stdN)rre _init_buffersrns r9rm WindowAttention.reset_parameterssd77SA r;cURupURR[XURRS95 g).Compute and fill non-persistent buffer values.rBN)r'rZcopy_rMrB)rnr>r?s r9rvWindowAttention._init_bufferss9''  $$** 'T=Y=Y=`=` a r;c [U5nXR:XagXlURup#X#-Ul[R"5 SU-S- SU-S- -UR 4n[ R"[URURUS95Ul URS[X#URRS9SS9 SSS5 g!,(df  g=f) zbUpdate window size & interpolate position embeddings Args: window_size (int): New window size Nr+rnew_window_sizenew_bias_shaperZr{Fr[) rr'r`rDno_gradrSrbrcrrerfrMrB)rnr'r>r?rs r9set_window_sizeWindowAttention.set_window_sizes  , ** * &''  = ]]_%i!mE A >NN02 )55$($4$4#11D -  )+EAbAbAiAij  ! __s BC C(cURURRS5RURURS5nUR SSS5R 5nUR S5$)Nr.r+rr)rerZr1r`r2r3 unsqueeze)rnrelative_position_biass r9_get_rel_pos_bias!WindowAttention._get_rel_pos_biasss!%!B!B  ( ( - -b 1"33748H8H$JZJZ\^3_ !7!?!?1a!H!S!S!U%//22r;r&maskcURup4nURU5RX4SURS5R SSSSS5nUR S5upxn UR (aUR5n UbaURSn URSU SXD5RX;-SURSS5nXRSURXD5-n [RRRXxU U UR(aURR OSS9nOXpR"-nXxR%S S5-n XR5-n UbjURSn U RSXRXD5UR'S5R'S5-n U RSURXD5n UR)U 5n URU 5n X-nUR%SS5RX4S5nUR+U5nUR-U5nU$) zForward pass. Args: x: Input features with shape of (num_windows*B, N, C). mask: (0/-inf) mask with shape of (num_windows, Wh*Ww, Wh*Ww) or None. Returns: Output features with shape of (num_windows*B, N, C). r*r.r+rrr,) attn_mask dropout_p)r0rhreshaperSr2unbindrQrr1expandrDrb functionalscaled_dot_product_attentiontrainingrVpra transposerrlrjrW) rnr&rB_Nr7rhqkvrnum_winattns r9forwardWindowAttention.forwards77qhhqk!!"DNNB?GG1aQRTUV**Q-a ??..0I**Q-yyGQ5<>$'DA KK1  % %bR 0 IIaL NN1 r;c$UR5 gz"Initialize non-persistent buffers.Nrvrws r9init_non_persistent_buffers+WindowAttention.init_non_persistent_buffers r;) rVrRrQrSrjrWrhrerarlr`r')NTrrNNr(NN)__name__ __module__ __qualname____firstlineno____doc__rDjitFinalbool__annotations__intr_int_or_tuple_2_tfloatr_rmrvr rTensorrrr__static_attributes__ __classcell__rqs@r9rOrOhs %% '+-.!!!4 4 4 sm 4 + 4  4 4 4 4 l  5c?t235<<3 ((Xell-C(u||(Tr;rOc x^\rSrSrSrSSSSSSSS S S S \R \RSS4S \S \ S \S\ \S\ S\S\ S\ S\ S\ S\ S\ S\ S\ \RS\ \R4U4SjjjrS+SjrS+SjrS,S\ \R&S\ \R(S \ \R*S\ \R&4S!jjrS-S"\\\\\44S#\ \\\\\44S\\\\4\\\444S$jjrS-S%\\\4S\\\4S\ \ 4S&jjrS'rS\R&S\R&4S(jrS+S)jrS*rU=r$).SwinTransformerBlockzcSwin Transformer Block. A transformer block with window-based self-attention and shifted windows. r,NrrF@TrrRinput_resolutionrSrTr' shift_sizealways_partition dynamic_mask mlp_ratiorUrWrV drop_path act_layer norm_layerc >UUS.n[TU]5 XlX l[ U5UlXplXlURXV5uUl Ul URSURS-Ul Xl U"U40UD6Ul [U4UUURU U U S.UD6UlU S:a [!U 5O["R$"5UlU"U40UD6Ul[+S U[-X-5UU S.UD6UlU S:a [!U 5O["R$"5UlUR3SSS S 9 UR55 g) ae Args: dim: Number of input channels. input_resolution: Input resolution. window_size: Window size. num_heads: Number of attention heads. head_dim: Enforce the number of channels per head shift_size: Shift size for SW-MSA. always_partition: Always partition into full windows and shift mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. proj_drop: Dropout rate. attn_drop: Attention dropout rate. drop_path: Stochastic depth rate. act_layer: Activation layer. norm_layer: Normalization layer. rArr)rSrTr'rUrVrWr) in_featureshidden_featuresrdroprNFr[)r^r_rRrrtarget_shift_sizerr_calc_window_shiftr'rr`rnorm1rOrrrbIdentity drop_path1norm2rrmlp drop_path2rfrm)rnrRrrSrTr'rrrrrUrWrVrrrrBrCrorqs r9r_SwinTransformerBlock.__init__sdJ /  0!*:!6 0(,0,C,CK,\)$/++A.1A1A!1DD"*r* #   ((     2;R(9-R[[]*r*  0    2;R(9-R[[] [$5A r;r(c$UR5 g)rtNrrws r9rm%SwinTransformerBlock.reset_parametersRrr;cUR(daURRRnURRRnUR XS9nUR SUSS9 gg)rzrArFr[N)rrweightrBrC get_attn_maskrf)rnrBrCrs r9rv"SwinTransformerBlock._init_buffersVsb  ZZ&&--FJJ%%++E**&*FI  iE J !r;r&rBrCc h[UR5(GaUb6URSURSpTURnURnOUR upEUnUn[ R"X@RS- 5URS-n[ R"XPRS- 5URS-n[R"SXES4X2S9nSnSURS*4URS*URS*4URS*S44HnnSURS*4URS*URS*4URS*S44H$n XvSS2USUS2U SU S2SS24'US- nM& Mp [X`R5n U RSUR5n U RS5U RS5- n U RU S:g[!S55RU S:H[!S55n U $Sn U $)Nrr+r)rCrBr.gYr)anyrr0rBrCrmathceilr'rDzerosr:r1r`r masked_fillr) rnr&rBrCr5r6img_maskcnthw mask_windowsrs r9r"SwinTransformerBlock.get_attn_mask^sE t  }wwqz1771:1,, !..q112T5E5Ea5HHA !..q112T5E5Ea5HHA{{AqQ+>?ooa(($/ T--a001**1--0B/BC//!,,d3A '>>$BRBRSTBUU!!!$q+;+;A+>'>>$BRBRSTBUUHH''++I1a57QR   r%Y0@0@A NN2t'7'7;  4 / /**95IIyyy; $((T-=-=a-@$BRBRSTBUWXY "<1A1A2J a!RaRl+668   9__6JAAr;c2URup#pEXRURURU555-nUR USU5nXR UR URU555-nUR X#XE5nU$)z{Forward pass. Args: x: Input features with shape (B, H, W, C). Returns: Output features with shape (B, H, W, C). r.)r0rrrrrrr)rnr&r4r5r6r7s r9rSwinTransformerBlock.forwards|WW a  4::a= 9: : IIaQ  A 78 8 IIaA !r;c$UR5 grrrws r9r0SwinTransformerBlock.init_non_persistent_buffersrr;)rrrRrrrrrrrrrrr`r'r)NNNr) rrrrrrbGELU LayerNormrrrrrr Moduler_rmrvrDrrBrCrr r rrrrrrrrs@r9rrs9&*-.%*!&!!!!!)+*,,,%K K 0K  K sm K + K K #K K K K K K K BIIK RYY!K K ZK)--1+/ & %&U\\*&EKK( & %,,  &VHL5 %c5c?&: ;5 (c5c?.B(CD5 uS#Xc3h/ 0 5204  S#X sCx 'tn  4%N%,, r;rc^\rSrSrSrS\R SS4S\S\\S\ \R4U4Sjjjr S\ RS \ R4S jrS rU=r$) PatchMergingizNPatch Merging Layer. Downsample features by merging 2x2 neighboring patches. NrRout_dimrc>XES.n[TU]5 XlU=(d SU-UlU"SU-40UD6Ul[ R "SU-UR4SS0UD6Ulg)z Args: dim: Number of input channels. out_dim: Number of output channels (or 2 * dim if None) norm_layer: Normalization layer. rAr+r,r]FN)r^r_rRr normrbrg reduction)rnrRr rrBrCrorqs r9r_PatchMerging.__init__sf / )!c' q3w-"- 1s7DLLKuKKr;r&r(c\URup#pESSSUS-SUS-4n[RRX5nURupspGUR X#S-SUS-SU5R SSSSSS5R S5nURU5nURU5nU$)zForward pass. Args: x: Input features with shape (B, H, W, C). Returns: Output features with shape (B, H//2, W//2, out_dim). rr+rr*r,r-) r0rbrrrr2rHr r)rnr&r4r5r6r7 pad_valuesrs r9rPatchMerging.forward sWW aAq1uaQ/ MM  a ,WW a IIaaAFAq 1 9 9!Q1a K S STU V IIaL NN1 r;)rRr r r)rrrrrrbrrrr rr_rDrrrrrs@r9r r sq&**,,, LLc]LRYY LL*%,,r;r c"D^\rSrSrSrSSSSSSSSS S S \R SS4S \S \S \\\4S \S\ S\S\ \S\ S\ S\ S\ S\ S\ S\ S\ \\ \ 4S\\R 4 U4SjjjrS!S\\\4S\S\ \ 4SjjrS\R(S\R(4SjrS rU=r$)"SwinTransformerStagei!ztA basic Swin Transformer layer for one stage. Contains multiple Swin Transformer blocks and optional downsampling. Tr,NrFrrrRr rdepth downsamplerSrTr'rrrrUrWrVrrcr>UUS.n[TU]5 XlX0lU(a[ SU55OUUlX@lSUl[U5n[ UVs/sHnUS-PM sn5nU(a[S UUUS.UD6Ul O!X:Xde[R"5Ul [R"[U5Vs/sHLn[S UUR UUUUS-S:XaSOUU U U U U U[!U["5(aUUOUUS.UD6PMN sn6Ulgs snfs snf) ar Args: dim: Number of input channels. out_dim: Number of output channels. input_resolution: Input resolution. depth: Number of blocks. downsample: Downsample layer at the end of the layer. num_heads: Number of attention heads. head_dim: Channels per head (dim // num_heads if not set) window_size: Local window size. mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. proj_drop: Projection dropout rate. attn_drop: Attention dropout rate. drop_path: Stochastic depth rate. norm_layer: Normalization layer. rAc3*# UH oS-v M g7fr+Nr.0is r9 0SwinTransformerStage.__init__..Qs&H7G!Av7GFr+)rRr rr)rRrrSrTr'rrrrrUrWrVrrNr)r^r_rRrroutput_resolutionrgrad_checkpointingrr rrbr Sequentialranger isinstancelistblocks)rnrRr rrrrSrTr'rrrrUrWrVrrrBrCrorrrrqs r9r_SwinTransformerStage.__init__'sXL /  0LV&H7G&H!H\l "' , K8KqAFK89  *% DO> !> kkmDOmm$5\%&#$"# ! !%!7!7#!'!"Q!1*!1)#!##*4Y*E*E)A,9%  ""%&#$ 9&#s "D/AD4rcXl[UR[R5(aXlO[ SU55UlURHnURUR UUS9 M g)zUpdates the resolution, window size and so the pair-wise relative positions. Args: feat_size: New input (feature) resolution window_size: New window size always_partition: Always partition / shift the window c3*# UH oS-v M g7frrrs r9r6SwinTransformerStage.set_input_size..s*E9a69rrr'rN) rr$rrbrr rr&r)rnrr'rblocks r9r#SwinTransformerStage.set_input_sizexsh!* door{{ 3 3%. "%**E9*E%ED "[[E  00'!1 ! !r;r&r(cURU5nUR(a;[RR 5(d[ UR U5nU$UR U5nU$)zKForward pass. Args: x: Input features. Returns: Output features. )rr!rDr is_scriptingrr&rnr&s r9rSwinTransformerStage.forwardsX OOA   " "599+A+A+C+Ct{{A.A AAr;)r&rrRrr!rr r)rrrrrrbrrr rrrrr rr rr_rrDrrrrrs@r9rr!sg $&*-.%*!&!!!!35*,,,'O$O$O$$CHo O$  O$  O$O$smO$+O$#O$O$O$O$O$O$ T%[%/0!O$"RYY#O$O$j04 S#X'tn 2%,,r;rc,^\rSrSrSrSSSSSSS S S S S SSSSSSS\\RSS S 4S\S\ S\ S\ S\ S\ S\ \ S4S\ \ S4S\ \ S\S\ S\ S \S!\ S"\S#\S$\S%\S&\\R S'\\ \\R 4S(\ 4*U4S)jjjr\R(R*SFS*\ S+\ S,S 4S-jj5r\R(R*S,\\ 4S.j5rSGS\ \ \ \ 4S\ \ \ \ 4S\ \ \ \ 4S/\ S\ \ S,S 4 S0jjr\R(R*SHS1\ S,\\ \44S2jj5r\R(R*SIS3\ S,S 4S4jj5r\R(R*S,\R 4S5j5rSJS\ S\ \ S,S 4S6jjrSKS7\R@S8\ \\ \!\ 4S9\ S:\ S;\ S<\ S,\\!\R@\ \R@\!\R@444S=jjr"SLS8\\ \!\ 4S>\ S?\ S,\!\ 4S@jjr#S7\R@S,\R@4SAjr$SHS7\R@SB\ S,\R@4SCjjr%S7\R@S,\R@4SDjr&SEr'U=r($)Mr%izSwin Transformer. A PyTorch impl of : `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows` - https://arxiv.org/pdf/2103.14030 r,r*avg`r+r+r+r*r8 NrFTrrg?img_size patch_sizein_chans num_classes global_pool embed_dimdepths.rSrTr'rstrict_img_sizerrU drop_rateproj_drop_rateattn_drop_ratedrop_path_rate embed_layerr weight_initc >[T!U]5 UUS.nUS;deX@lX0lXPlSUl[ U5UlX`l[USURS- --5=Ul Ul /Ul [U[[45(d4[!UR5Vs/sHn[USU--5PM nnU"S"UUUUSUU SS.UD6UlUR"R$n['UR5"U 5n [U [[45(d['UR5"U 5n O[ U 5S:XaU 4UR-n [ U 5UR:Xde['UR5"U 5n [)UUSS 9n/nUSnSn[!UR5HnUUn U[+S"0S U_S U _S USU-USU-4_S UU_SUS:_SUU_SU U_SU U_SU _SU (+_SU U_SU_SU_SU_SUU_SU_UD6/- nU nUS:aUS-nU=R[-U UU-SU3S9/- sl M [.R0"U6UlU"UR40UD6Ul[7URU4UUUR S.UD6UlUS:XaSOUUlUS:waUR=SS 9 g!g!s snf)#a Args: img_size: Input image size. patch_size: Patch size. in_chans: Number of input image channels. num_classes: Number of classes for classification head. embed_dim: Patch embedding dimension. depths: Depth of each Swin Transformer layer. num_heads: Number of attention heads in different layers. head_dim: Dimension of self-attention heads. window_size: Window size. mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. drop_rate: Dropout rate. attn_drop_rate (float): Attention dropout rate. drop_path_rate (float): Stochastic depth rate. embed_layer: Patch embedding layer. norm_layer (nn.Module): Normalization layer. rA)r<r5NHWCr+rr)r=r>r?rBrrD output_fmtT) stagewiserRr rrrrSrTr'rrrrUrWrVrrlayers.)num_chsrmodule) pool_typerE input_fmtskipresetF) needs_resetNr)r^r_r@r?rArMlen num_layersrBr num_featureshead_hidden_size feature_infor$rr%r# patch_embed grid_sizerrrdictrbr"layersr rheadweight_init_mode init_weights)"rnr=r>r?r@rArBrCrSrTr'rrDrrUrErFrGrHrIrrJrBrCkwargsror patch_griddprr_in_dimrar rqs" r9r_SwinTransformer.__init__s\  /k)))& & f+"47 A$//\]J]D^8^4__D1)eT]33:?:PQ:PQYa/0:PIQ'  !l!+    %%// T__-h7+e}55#DOO4[AK   "&.4??:K;4??222doo.y9 '$O1t'AlG +qMU*qMU*"Qiq5$A,"!(N"2"10$A," )!")#$a&%&&) F,F1u    $w*uBT_fghfi]j"k!l l 7(8mmV, t007B7 "     "oo     ,7&+@k &   %  0 !KRs/K/moderVr(cU=(d URnUS;deSU;a![R"UR5*OSn[ [ XUS9U5 g)aInitialize model weights. Args: mode: Weight initialization mode ('jax', 'jax_nlhb', 'moco', or ''). needs_reset: If True, call reset_parameters() on modules that have it. Set to False when modules have already self-initialized in __init__. )jaxjax_nlhbmocorUr<nlhbr) head_biasrVN)rarlogr@r r$)rnrhrVrns r9rbSwinTransformer.init_weights.sV,t,,????39T>TXXd..//r (P[\^bcr;c~[5nUR5Hup#SU;dM URU5 M U$)z,Parameters that should not use weight decay.re)setnamed_parametersadd)rnnwdnrs r9no_weight_decaySwinTransformer.no_weight_decay<s:e))+DA-2 , r; window_ratiocXUcUb/URRXS9 URRnUc[WVs/sHowU-PM sn5n[ UR 5H6upS[ US- S5-n U RWSU -USU -4UUS9 M8 gs snf)aUpdate the image resolution and window size. Args: img_size: New input resolution, if None current resolution is used. patch_size: New patch size, if None use current patch size. window_size: New window size, if None based on new_img_size // window_div. window_ratio: Divisor for calculating window size from grid size. always_partition: Always partition into windows and shift (even if window size < feat size). N)r=r>r+rrr+)r\rr]r enumerater_max) rnr=r>r'ryrrdpgindexstage stage_scales r9rSwinTransformer.set_input_sizeEs"  :#9    + +X + U))33J  j Ij|!3j IJK%dkk2LEs519a00K  %a=K7A+9UV'!1 ! 3!JsB'coarsec0[SU(aSS9$/SQS9$)z"Group parameters for optimization.z ^patch_embedz^layers\.(\d+)))z^layers\.(\d+).downsample)r)z^layers\.(\d+)\.\w+\.(\d+)N)z^norm)i)stemr&)r^)rnrs r9 group_matcherSwinTransformer.group_matcheres) (.$  5  r;enablec6URH nXlM g)z)Enable or disable gradient checkpointing.N)r_r!)rnrls r9set_grad_checkpointing&SwinTransformer.set_grad_checkpointingqsA#) r;c.URR$)zGet the classifier head.)r`fcrws r9get_classifierSwinTransformer.get_classifierwsyy||r;cBXlURRXS9 g)zReset the classifier head. Args: num_classes: Number of classes for new classifier. global_pool: Global pooling type. )rRN)r@r`rU)rnr@rAs r9reset_classifier SwinTransformer.reset_classifier|s'  ;r;r&indicesr  stop_earlyrMintermediates_onlyclUS;dS5e/n[[UR5U5upURU5n[UR5n [R R 5(dU(d URn OURSU S-n [U 5HjupU "U5nX;dMU(aXS- :XaURU5nOUnURSSSS5R5nURU5 Ml U(aU$URU5nX4$)aForward features that returns intermediates. Args: x: Input image tensor. indices: Take last n blocks if int, all if None, select matching indices if sequence. norm: Apply norm layer to compatible intermediates. stop_early: Stop iterating over blocks when last desired intermediate hit. output_fmt: Shape of intermediate feature outputs. intermediates_only: Only return intermediate features. Returns: List of intermediate features or tuple of (final features, intermediates). )NCHWzOutput shape must be NCHW.Nrrr*r+) rrWr_r\rDrr/r{r r2r3append)rnr&rr rrMr intermediates take_indices max_index num_stagesstagesrrx_inters r9forward_intermediates%SwinTransformer.forward_intermediatess,Y&D(DD& "6s4;;7G"Q    Q % 99 ! ! # #:[[F[[)a-0F!&)HAaA Aa/"iilGG!//!Q15@@B$$W-*  IIaLr; prune_norm prune_headc[[UR5U5upEURSUS-UlU(a[R"5UlU(aUR SS5 U$)a Prune layers not required for specified intermediates. Args: indices: Indices of intermediate layers to keep. prune_norm: Whether to prune normalization layer. prune_head: Whether to prune the classifier head. Returns: List of indices that were kept. Nrrr<)rrWr_rbrr r)rnrrrrrs r9prune_intermediate_layers)SwinTransformer.prune_intermediate_layerss[ #7s4;;7G"Q kk.9q=1  DI   ! !!R (r;clURU5nURU5nURU5nU$)z/Forward pass through feature extraction layers.)r\r_r r0s r9forward_features SwinTransformer.forward_featuress1   Q  KKN IIaLr; pre_logitscRU(aURUSS9$URU5$)zForward pass through classifier head. Args: x: Feature tensor. pre_logits: Return features before final classifier. Returns: Output tensor. T)r)r`)rnr&rs r9 forward_headSwinTransformer.forward_heads&1;tyyty,L ! Lr;cJURU5nURU5nU$)zGForward pass. Args: x: Input tensor. Returns: Output logits. )rrr0s r9rSwinTransformer.forwards)  ! !! $   a r;)rBr[rAr`rZr?r_r r@rYrXrMr\ra)r<T)NNNNF)Tr)NFFrF)rFT))rrrrrrrbrrrstrr rrrr rr r_rDrignorerbrrwrrrrrrrrrrrrrrrrrs@r9r%r%s;+.#$&2)7&*-.%*$(!!!$&$&$'+568ll!1B1'B1B1 B1  B1  B1B1#s(OB1S#XB1smB1+B1#B1"B1B1B1 !B1""#B1$"%B1&"'B1(bii)B1*c4 ?23+B1,-B1B1H YY d d d d d YYS374859 !/3 uS#X/!sCx1"%S/2    'tn  @ YY  D  T#s(^     YY*T*T**  YY *  c 0%,,5<< Mell M M M  %,,  r; state_dictmodelc XSnSU;aSnSSKn0nURSU5nURSU5nUR5GH[upV[SVs/sHowU;PM sn5(aM+S U;abURR R Ru pn URS U :wdURS U :wa[UX4S SSS 9nURS5(aURUSS5n URU RR:wd U RSU RS:wa)[UU RU RRS9nU(a&URSSU5nURSS5nXdU'GM^ U$s snf)zConvert patch embedding weight from manual patchify + linear proj to conv. Args: state_dict: State dictionary from checkpoint. model: Model instance. Returns: Filtered state dictionary. Tzhead.fc.weightFrNrr)rZrzpatch_embed.proj.weightrr.bicubic) interpolation antialiasverbosereirrzlayers.(\d+).downsamplecDS[URS55S-S3$)NrOrz .downsample)rgroup)r&s r9&checkpoint_filter_fn..s"ws177ST:YZGZF[[f=gr;zhead.zhead.fc.)regetitemsrr\rjrr0rendswith get_submodulerer'rsubreplace) rr old_weightsrout_dictrrrvrr5r6ms r9checkpoint_filter_fnrsK:%  H4J j9J  "  HI H1Q HI J J  $ )**//66<>>!--PQBRVWVcVcdeVfBf-$%MM#$#A#A#G#G 13gijkA ':.A 9#: O9JsF' variant pretrainedc [S[URSS5555nURSU5n[ [ X4[ [SUS9S.UD6nU$)zCreate a Swin Transformer model. Args: variant: Model variant name. pretrained: Load pretrained weights. **kwargs: Additional model arguments. Returns: SwinTransformer model instance. c3*# UH upUv M g7frr)rrrs r9r+_create_swin_transformer...s\.[da.[rrC)rrr*r out_indicesT)flatten_sequentialr)pretrained_filter_fn feature_cfg)rr{rpoprr%rr^)rrrcdefault_out_indicesrrs r9_create_swin_transformerr#sh \i 8\8Z.[\\**],?@K  1DkJ    E Lr;urlc 4USSSSSS[[SSS S . UE$) z9Create default configuration for Swin Transformer models.r4)r*r3r3)rrg?rTzpatch_embed.projzhead.fcmit) rr@ input_size pool_sizecrop_pctrfixed_input_sizemeanru first_conv classifierlicenser )rrcs r9_cfgr:s7=v%.B(    #  r;z.swin_small_patch4_window7_224.ms_in22k_ft_in1kztimm/zvhttps://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_small_patch4_window7_224_22kto1k_finetune.pth) hf_hub_idrz-swin_base_patch4_window7_224.ms_in22k_ft_in1kzlhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22kto1k.pthz.swin_base_patch4_window12_384.ms_in22k_ft_in1kzmhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384_22kto1k.pth)r*r)r:r:g?)rrrrrz.swin_large_patch4_window7_224.ms_in22k_ft_in1kzmhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22kto1k.pthz/swin_large_patch4_window12_384.ms_in22k_ft_in1kznhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window12_384_22kto1k.pthz$swin_tiny_patch4_window7_224.ms_in1kzdhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_tiny_patch4_window7_224.pthz%swin_small_patch4_window7_224.ms_in1kzehttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_small_patch4_window7_224.pthz$swin_base_patch4_window7_224.ms_in1kzdhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224.pthz%swin_base_patch4_window12_384.ms_in1kzehttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384.pthz-swin_tiny_patch4_window7_224.ms_in22k_ft_in1kzuhttps://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22kto1k_finetune.pthz%swin_tiny_patch4_window7_224.ms_in22kzhhttps://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22k.pthiQU)rrr@z&swin_small_patch4_window7_224.ms_in22kzihttps://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_small_patch4_window7_224_22k.pthz%swin_base_patch4_window7_224.ms_in22kzhhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22k.pthz&swin_base_patch4_window12_384.ms_in22kzihttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384_22k.pth)rrrrrr@z&swin_large_patch4_window7_224.ms_in22kzihttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22k.pthz'swin_large_patch4_window12_384.ms_in22kzjhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window12_384_22k.pthzswin_s3_tiny_224.ms_in1kzbhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/s3_t-1d53f6a8.pthzbhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/s3_s-3bb4c69d.pthzbhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/s3_b-a1e95db4.pth)zswin_s3_small_224.ms_in1kzswin_s3_base_224.ms_in1kc L[SSSSSS9n[SSU0[U40UD6D6$) z*Swin-T @ 224x224, trained ImageNet-1k r,rr6r7r9r>r'rBrCrSr)swin_tiny_patch4_window7_224r^rrrc model_argss r9rrsFR `noJ #& ]3= ]AEjA[TZA[ ]]r;c L[SSSSSS9n[SSU0[U40UD6D6$) zSwin-S @ 224x224 r,rr6r+r+r+r9rr)swin_small_patch4_window7_224rrs r9rrsFR aopJ #' ^4> ^BFzB\U[B\ ^^r;c L[SSSSSS9n[SSU0[U40UD6D6$) zSwin-B @ 224x224 r,rrr,r rr)swin_base_patch4_window7_224rrs r9rrsFSbpqJ #& ]3= ]AEjA[TZA[ ]]r;c L[SSSSSS9n[SSU0[U40UD6D6$) zSwin-B @ 384x384 r,r:rrrrr)swin_base_patch4_window12_384rrs r9rrsFc-cqrJ #' ^4> ^BFzB\U[B\ ^^r;c L[SSSSSS9n[SSU0[U40UD6D6$) zSwin-L @ 224x224 r,rrr8r:r;0rr)swin_large_patch4_window7_224rrs r9rrsFSbqrJ #' ^4> ^BFzB\U[B\ ^^r;c L[SSSSSS9n[SSU0[U40UD6D6$) zSwin-L @ 384x384 r,r:rrrrr)swin_large_patch4_window12_384rrs r9rrsFc-crsJ #( _5? _CG C]V\C] __r;c J[SSSSSS9n[SSU0[U40UD6D6$) z:Swin-S3-T @ 224x224, https://arxiv.org/abs/2111.14725 r,rrrr6r7r9rr)swin_s3_tiny_224rrs r9rrs;-2l^lnJ # l: lQUV`QkdjQk llr;c J[SSSSSS9n[SSU0[U40UD6D6$) z:Swin-S3-S @ 224x224, https://arxiv.org/abs/2111.14725 r,)rrrrr6rr9rr)swin_s3_small_224rrs r9rrs;/R aoqJ # mJ mRVWaRlekRl mmr;c J[SSSSSS9n[SSU0[U40UD6D6$) z:Swin-S3-B @ 224x224, https://arxiv.org/abs/2111.14725 r,rr6)r+r+r+r9rr)swin_s3_base_224rrs r9rrs;-2m_moJ # l: lQUV`QkdjQk llr;)"swin_base_patch4_window7_224_in22k#swin_base_patch4_window12_384_in22k#swin_large_patch4_window7_224_in22k$swin_large_patch4_window12_384_in22krr)r<)Orloggingrtypingrrrrrrr r r rDtorch.nnrb timm.datar r timm.layersrrrrrrrrrrrr_builderr _featuresr _features_fxr _manipulaterr _registryr!r"r#vision_transformerr$__all__ getLoggerr_loggerrrrr:r=rMrrOrr rr%r^rrrrr default_cfgsrrrrrrrrrrr;r9rs0 " OOO ALLLL*+34YY4     H %#uS#X./ <<38_ \\& ELL uS#X 3 SV [`[g[g  $#s#3# #0TbiiTno299od-299-`299DLbiiL^ .T.")).S%,,EV@W.bctRa. c T#s(^ %H&4d E7HH&4T z6} H&5d { Hs7DH&5d {7~H&6t | Hs8DH&&+D r-u'H&,,T s.v-H&2+D r-u3H&8,T s Hs.D9H&D4T D6FEH&L,T v.MH&T-d w/UH&\,T v.]H&d-d w HsPU/WeH&l-d w/mH&t.t x HsPU0WuH&~ p!rH&D"& p"r!% p!rKH&H V]]]^^^]]]^^^^^^_/__mOmmn_nnmOmmH*Q+S+S,U 'r;