RЦi/rSrSSKrSSKJr SSKJrJrJrJrJ r J r J r J r J r SSKrSSKJr SSKJs Jr SSKJrJr SSKJrJrJrJrJrJrJrJrJ r J!r!J"r" SSK#J$r$ SS K%J&r& SS K'J(r( SS K)J*r* SS K+J,r,J-r-J.r. S /r/\ \0\ \0\044r1S\RdS\ \0\04S\Rd4Sjr3\(S\RdS\ \0\04S\ \0\04S\Rd4Sj5r4"SS\Rj5r6"SS\Rj5r7"SS\Rj5r8"SS\Rj5r9"SS \Rj5r:S\\;\Rd4S\RjS\\;\Rd44S jrSPS$jr?\,"\?"S%S&S'9\?"S%S(S)S*S+S,9\?"S%S-S'9\?"S%S.S)S*S+S,9\?"S%S/S'9\?"S%S0S'9\?"S%S1S'9\?"S%S2S'9\?"S%S3S'9\?"S%S4S'9\?"S%S5S6S7S8S99\?"S%S:S6S7S8S99S;. 5r@\-SOS"\=S\:4S<jj5rA\-SOS"\=S\:4S=jj5rB\-SOS"\=S\:4S>jj5rC\-SOS"\=S\:4S?jj5rD\-SOS"\=S\:4S@jj5rE\-SOS"\=S\:4SAjj5rF\-SOS"\=S\:4SBjj5rG\-SOS"\=S\:4SCjj5rH\-SOS"\=S\:4SDjj5rI\-SOS"\=S\:4SEjj5rJ\-SOS"\=S\:4SFjj5rK\-SOS"\=S\:4SGjj5rL\."\MSHSISJSKSLSMSN.5 g)QaJSwin Transformer V2 A PyTorch impl of : `Swin Transformer V2: Scaling Up Capacity and Resolution` - https://arxiv.org/abs/2111.09883 Code/weights from https://github.com/microsoft/Swin-Transformer, original copyright/license info below Modifications and additions for timm hacked together by / Copyright 2022, Ross Wightman N)partial) AnyCallableDictListOptionalSetTupleTypeUnionIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) PatchEmbedMlpDropPathcalculate_drop_path_rates to_2tuple trunc_normal_ClassifierHeadresample_patch_embedndgrid get_act_layer LayerType)build_model_with_cfg)feature_take_indices)register_notrace_function) checkpoint)generate_default_cfgsregister_modelregister_model_deprecationsSwinTransformerV2x window_sizereturncURup#pEURX#US-USXAS-USU5nURSSSSSS5R5RSUSUSU5nU$)zPartition into non-overlapping windows. Args: x: Input tensor of shape (B, H, W, C). window_size: Window size (height, width). Returns: Windows tensor of shape (num_windows*B, window_size[0], window_size[1], C). rrshapeviewpermute contiguous)r$r%BHWCwindowss ^/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/models/swin_transformer_v2.pywindow_partitionr8%sJA! q{1~%{1~qN7JKXYN\]^Aii1aAq)446;;B AP[\]P^`abG Nr6img_sizecUup4URSnURSX1S-XAS-USUSU5nURSSSSSS5R5RSX4U5nU$)aMerge windows back to feature map. Args: windows: Windows tensor of shape (num_windows * B, window_size[0], window_size[1], C). window_size: Window size (height, width). img_size: Image size (height, width). Returns: Feature map tensor of shape (B, H, W, C). r,rrr(r)r*r+r-)r6r%r:r3r4r5r$s r7window_reverser<8s DA bA R!n,aq>.A;q>S^_`SacdeA !Q1a#..055b!BA Hr9cX^\rSrSrSrSS\S\\\4S\S\S\S \S \S \\\4S S4U4S jjjr SSjr SSjr SS \\ R\ R44SjjrS\\\4S S4SjrSSjrSS\ RS\\ RS \ R4SjjrSrU=r$)WindowAttentionOzWindow based multi-head self attention (W-MSA) module with relative position bias. Supports both shifted and non-shifted window attention with continuous relative position bias and cosine attention. Ndimr% num_headsqkv_biasqkv_bias_separate attn_drop proj_droppretrained_window_sizer&c >XS.n [TU]5 XlX l[ U5UlX0lXPl[R"[R"USS440U D65Ul [R"[R"SSS0U D6[R"SS9[R"SU4SS0U D65Ul[R"XS -4SS0U D6UlU(a[R"[R"U40U D65UlUR'S [R"U40U D6SS 9 [R"[R"U40U D65UlOS UlS UlS Ul[R,"U5Ul[R"X40U D6Ul[R,"U5Ul[R4"S S9UlURupUR'S[R"SSU -S- SU -S- S40U D6SS 9 UR'S[R"X-X-U [R8S9SS 9 UR;5 g )aInitialize window attention module. Args: dim: Number of input channels. window_size: The height and width of the window. num_heads: Number of attention heads. qkv_bias: If True, add a learnable bias to query, key, value. qkv_bias_separate: If True, use separate bias for q, k, v projections. attn_drop: Dropout ratio of attention weight. proj_drop: Dropout ratio of output. pretrained_window_size: The height and width of the window in pre-training. devicedtyperr)biasT)inplaceFr(k_bias persistentNr,r@relative_coords_tablerelative_position_index)r)rK)super__init__r@r%rrFrArCnn Parametertorchempty logit_scale SequentialLinearReLUcpb_mlpqkvq_biasregister_bufferv_biasrNDropoutrDprojrESoftmaxsoftmaxlongreset_parameters)selfr@r%rArBrCrDrErFrIrJddwin_hwin_w __class__s r7rUWindowAttention.__init__Vs2 / &&/0F&G#"!2<< Y14E(L(LM}} II .4 .2 . GGD ! IIc9 75 7B 7  99S'<<< ,,u{{3'="'=>DK  5;;s+Ab+Ae T,,u{{3'="'=>DKDKDKDKI.IIc-"- I.zzb) ''   # KK1u9q=!e)a- Ab A   % KK u}V5:: V   r9c^[RRUR[R "S55 UR bR[RRUR 5 [RRUR5 UR5 g)z"Initialize parameters and buffers. N) rVinit constant_rZmathlogr`zeros_rb _init_buffersris r7rh WindowAttention.reset_parameterssb $**DHHRL9 ;; " GGNN4;; ' GGNN4;; ' r9cZURbURR5 URURRR URRR S9upURRU5 URRU5 g)z.Compute and fill non-persistent buffer values.NrH) rNzero_"_make_pair_wise_relative_positionsrdweightrIrJrRcopy_rS)rirRrSs r7rvWindowAttention._init_bufferss ;; " KK   9=9`9`99##**$))2B2B2H2H:a: 6 ""(()>? $$**+BCr9c[R"URSS- *URSU[RS9n[R"URSS- *URSU[RS9n[R"[ X455nUR SSS5R5RS5nURSS:aOUSS2SS2SS2S4==URSS- -ss'USS2SS2SS2S4==URSS- -ss'ONUSS2SS2SS2S4==URSS- -ss'USS2SS2SS2S4==URSS- -ss'US-n[R"U5[R"[R"U5S-5-[R"S5- nURUS9n[R"URSU[RS9n[R"URSU[RS9n[R"[ Xg55n[R "US5n U SS2SS2S4U SS2SSS24- n U R SSS5R5n U SS2SS2S4==URSS- - ss'U SS2SS2S4==URSS- - ss'U SS2SS2S4==SURS-S- -ss'U R#S 5n X[4$) zCompute pair-wise relative position index and coordinates table. Returns: Tuple of (relative_coords_table, relative_position_index) rrrHr)N?)rJr,)rXaranger%float32stackrr0r1 unsqueezerFsignlog2absrstorgflattensum) rirIrJrelative_coords_hrelative_coords_wrRcoords_hcoords_wcoordscoords_flattenrelative_coordsrSs r7r{2WindowAttention._make_pair_wise_relative_positionss "LLq!A% &(8(8(;FRWR_R_a!LLq!A% &(8(8(;FRWR_R_a % F3D,X Y 5 = =aA F Q Q S ] ]^_ `  & &q )A - !!Q1* -$2M2Ma2PST2T U - !!Q1* -$2M2Ma2PST2T U - !!Q1* -$2B2B12E2I J - !!Q1* -$2B2B12E2I J -" % +@ AEJJ II+ ,s 2E4!46:iil!C 5 8 8u 8 E<< 0 0 3F%**U<< 0 0 3F%**UVH78vq1(At4~aqj7QQ)11!Q:EEG1a D$4$4Q$7!$;; 1a D$4$4Q$7!$;; 1a A(8(8(;$;a$?? "1"5"5b"9$==r9c"[U5nXR:wauURceURRnURRnXlUR X#S9upEUR SUSS9 UR SUSS9 gg)zuUpdate window size and regenerate relative position tables. Args: window_size: New window size (height, width). NrHrRFrOrS)rr%rRrIrJr{ra)rir%rIrJrRrSs r7set_window_sizeWindowAttention.set_window_sizes  , ** *--9 99//66F..44E* 77v7S ; !  !8:O\a b  !:Z>Z>_>_`b>c!d!i!i   Q $"2"21"5 5t7G7G7JTM]M]^_M`7`bd"f!7!?!?1a!H!S!S!U!#emm4J&K!K66q99  jjmG99R..!?$..QRBSB]B]^_B``D99R6D<<%D<<%D~~d# X A & . .ra 8 IIaL NN1 r9)rDr^r@rNrZrArFrdrEr`r_rCrfrbr%)TFr)rrNNr&N)NNN)__name__ __module__ __qualname____firstlineno____doc__intr boolfloatrUrhrvrXTensorr{rrrr__static_attributes__ __classcell__rms@r7r>r>Os%"&+!!6<F F sCxF  F  F $ F F F %*#s(OF  F F PD(> u||U\\) * (>Tg5c?gtg"11Xell-C1u||11r9r>c4^\rSrSrSrSSSSSSSSSS \R SS S 4S \S \S \S\S\S\ S\ S\ S\ S\ S\ S\ S\ S\ \RS\4U4SjjjrS(S\\R"S\\R$S\\R&S\\R"4SjjrS)S \S!\\S\\\\4\\\444S"jjrS)S#\\\4S\\\4S\\ SS 4S$jjrS\R"S\R"4S%jrS\R"S\R"4S&jrS'rU=r$)*SwinTransformerV2Blocki#zSwin Transformer V2 Block. A standard transformer block with window attention and shifted window attention for modeling long-range dependencies efficiently. rF@TrgeluNr@input_resolutionrAr% shift_sizealways_partition dynamic_mask mlp_ratiorBrErD drop_path act_layer norm_layerrFc 6>UUS.n[TU]5 Xl[U5UlX0l[U5UlX`lXplURXE5uUl Ul URSURS-Ul Xl [U 5n [U4[UR5UU U U [U5S.UD6UlU"U40UD6UlU S:a [%U 5O[&R("5Ul[-S U[/X-5U U S.UD6UlU"U40UD6UlU S:a [%U 5O[&R("5UlUR7SUR(aSOUR8"S 0UD6S S 9 g) ag Args: dim: Number of input channels. input_resolution: Input resolution. num_heads: Number of attention heads. window_size: Window size. shift_size: Shift size for SW-MSA. always_partition: Always partition into full windows and shift mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. proj_drop: Dropout rate. attn_drop: Attention dropout rate. drop_path: Stochastic depth rate. act_layer: Activation layer. norm_layer: Normalization layer. pretrained_window_size: Window size in pretraining. rHrr)r%rArBrDrErFr) in_featureshidden_featuresrdrop attn_maskNFrO)rTrUr@rrrAtarget_shift_sizerr_calc_window_shiftr%r window_arearrr>rnorm1rrVIdentity drop_path1rrmlpnorm2 drop_path2ra get_attn_mask)rir@rrAr%rrrrrBrErDrrrrFrIrJrjrms r7rUSwinTransformerV2Block.__init__*sJ /  )*: ;"!*:!6 0(,0,C,CK,\)$/++A.1A1A!1DD"!), #   !$"2"23#,-C#D      *r* 1:R(9-R[[] 0     *r* 1:R(9-R[[]  %%D4+=+=+C+C  r9r$rIrJr&c [UR5(GaUc&[R"S/URQSP7X#S9nOH[R"SUR SUR SS4UR URS9nSnSURS*4URS*URS*4URS*S44HnnSURS*4URS*URS*4URS*S44H$nXTSS2USUS2USUS2SS24'US- nM& Mp [X@R5nURSUR5nURS5URS5- n U RU S:g[S55RU S:H[S55n U $Sn U $) zGenerate attention mask for shifted window attention. Args: x: Input tensor for dynamic shape calculation. Returns: Attention mask or None if no shift. NrrHr)rr,gYr)anyrrXzerosrr.rIrJr%r8r/rr masked_fillr) rir$rIrJimg_maskcnthw mask_windowsrs r7r$SwinTransformerV2Block.get_attn_maskys t  y ;;'ED,A,A'E1'Efb ;;1771:qwwqz1'Eahh^_^e^efC))!,,-&&q))DOOA,>+>?ooa(($/ T--a001**1--0B/BC//!,,d3A '>>$BRBRSTBUU!!!$q+;+;A+>'>>$BRBRSTBUUHH''++I1a57QR   r%Y0@0@A NN2t'7'7;  4 / /**95IIyyy; $((T-=-=a-@$BRBRSTBUWXY "<1A1AB8L a!RaRl+668   9__6JAAr9c2URup#pEXRURURU555-nUR USU5nXR UR URU555-nUR X#XE5nU$)Nr,)r.rrrrrrr)rir$r2r3r4r5s r7rSwinTransformerV2Block.forward szWW a  4::a= 9: : IIaQ   488A; 78 8 IIaA !r9)rrr@rrrrrrrrrArrrr%)NNNr)rrrrrrV LayerNormr_int_or_tuple_2_trrrr ModulerUrrXrrIrJrr rrrrrrrs@r7rr#s./,-%*!&!!!!!#)*,,,89%M M 0M  M + M * M #M M M M M M M !M RYYM %6!M M b)--1+/ ' %'U\\*'EKK( ' %,,  'X>B5 15 ((9:5 uS#Xc3h/ 0 5J04  S#X sCx 'tn    8,u||, ,\%,,r9rc^\rSrSrSrS\R SS4S\S\\S\ \R4U4Sjjjr S\ RS \ R4S jrS rU=r$) PatchMergingizPatch Merging Layer. Merges 2x2 neighboring patches and projects to higher dimension, effectively downsampling the feature maps. Nr@out_dimrc>XES.n[TU]5 XlU=(d SU-Ul[R "SU-UR4SS0UD6UlU"UR40UD6Ulg)z Args: dim (int): Number of input channels. out_dim (int): Number of output channels (or 2 * dim if None) norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm rHr)r*rLFN)rTrUr@rrVr\ reductionnorm)rir@rrrIrJrjrms r7rUPatchMerging.__init__sf / )!c' 1s7DLLKuKKt||2r2 r9r$r&c\URup#pESSSUS-SUS-4n[RRX5nURupspGUR X#S-SUS-SU5R SSSSSS5R S5nURU5nURU5nU$)Nrr)rr(r*r+) r.rVrrrr0rrr)rir$r2r3r4r5 pad_valuesrs r7rPatchMerging.forward2sWW aAq1uaQ/ MM  a ,WW a IIaaAFAq 1 9 9!Q1a K S STU V NN1  IIaLr9)r@rrr)rrrrrrVr rrr r rUrXrrrrrs@r7rrsl&**,,, 33c]3RYY 33*  %,,  r9rc'b^\rSrSrSrSSSSSSSSS\R SSS S 4S \S \S \S \S\S\S\ S\ S\ S\ S\ S\ S\ S\ S\ \ \ \R4S\ \RS\S\ SS 4&U4SjjjrS$S\\\4S\S\\ SS 4SjjrS \R(S\R(4S!jrS%S"jrS#rU=r$)&SwinTransformerV2Stagei?zA Swin Transformer V2 Stage. A single stage consisting of multiple Swin Transformer blocks with optional downsampling at the beginning. FrTrrrNr@rrdepthrAr%rr downsamplerrBrErDrrrrF output_nchwr&c>UUS.n[TU]5 XlX0lU (a[ SU55OUUlX@lUUlSUl[U5n[ UVs/sHnUS-PM sn5nU (a[S XUS.UD6Ul O!X:Xde[R"5Ul [R"[U5Vs/sHMn[!S UUR UUUS-S:XaSOUUUU U U U [#U[$5(aUUOUUUUS.UD6PMO sn5Ulgs snfs snf) ak Args: dim: Number of input channels. out_dim: Number of output channels. input_resolution: Input resolution. depth: Number of blocks. num_heads: Number of attention heads. window_size: Local window size. always_partition: Always partition into full windows and shift dynamic_mask: Create attention mask in forward based on current input size downsample: Use downsample layer at start of the block. mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. proj_drop: Projection dropout rate attn_drop: Attention dropout rate. drop_path: Stochastic depth rate. act_layer: Activation layer type. norm_layer: Normalization layer. pretrained_window_size: Local window size in pretraining. output_nchw: Output tensors on NCHW format instead of NHWC. rHc3*# UH oS-v M g7fr)Nr.0is r7 2SwinTransformerV2Stage.__init__..vs&H7G!Av7GFr))r@rrr)r@rrAr%rrrrrBrErDrrrrFNr)rTrUr@rroutput_resolutionrrgrad_checkpointingrrrrVr ModuleListranger isinstancelistblocks)rir@rrrrAr%rrrrrBrErDrrrrFrrIrJrjrrr!rms r7rUSwinTransformerV2Stage.__init__FsUX /  0LV&H7G&H!H\l &"' , K8KqAFK89  *asPZa^`aDO> !> kkmDOmm&5\'%#&"% # !%!7!7#'!"Q!1*!1)#!##*4Y*E*E)A,9#%'=  ! $"'%#$ 9%#s )D9AD>rc<Xl[UR[R5(aXlO8[UR[ 5(de[SU55UlURHnURUR UUS9 M g)zUpdate resolution, window size and relative positions. Args: feat_size: New input (feature) resolution. window_size: New window size. always_partition: Always partition / shift the window. c3*# UH oS-v M g7frrrs r7r"8SwinTransformerV2Stage.set_input_size..s*E9a69r$rr%rN) rr)rrVrr%rrr+r)rirr%rblocks r7r%SwinTransformerV2Stage.set_input_sizes!* door{{ 3 3%. "doo|<< <<%**E9*E%ED "[[E  00'!1 ! !r9r$cURU5nURHLnUR(a0[RR 5(d [ X!5nMDU"U5nMN U$)zForward pass through the stage. Args: x: Input tensor of shape (B, H, W, C). Returns: Output tensor of shape (B, H', W', C'). )rr+r&rXjit is_scriptingr)rir$blks r7rSwinTransformerV2Stage.forwardsU OOA ;;C&&uyy/E/E/G/Gs&F  r9cURHn[RRURR S5 [RRURR S5 [RRURR S5 [RRURR S5 M g)z/Initialize residual post-normalization weights.rN)r+rVrqrrrrLr|r)rir6s r7_init_respostnorm(SwinTransformerV2Stage._init_respostnorms;;C GG  ciinna 0 GG  cii.. 2 GG  ciinna 0 GG  cii.. 2 r9)r+rr@rr&rrr%rr)rrrrrrVr rr rrr strr r rUr rrrXrrr9rrrs@r7rr?s&+!&$!!!!!5;*,,,89 %+R$R$R$0 R$  R$  R$+R$#R$R$R$R$R$R$R$R$ S$ryy/12!R$"RYY#R$$%6%R$&'R$, -R$R$p04 S#X'tn   4%,,$33r9rc+^\rSrSrSrSSSSSSS S S S S SS SSSSS\R SSS4S\S\S\S\S\ S\S\ \S4S\ \S4S\S\ S\ S \ S!\ S"\ S#\ S$\ S%\ S&\ \ \4S'\\R S(\ \S44(U4S)jjjrSGS*\ S+S4S,jjrSGS-\R S*\ S+S4S.jjrSHS\\ \\4S\\ \\4S\\ \\4S/\\S\\ 4 S0jjr\R.R0S+\\ 4S1j5r\R.R0SIS2\ S+\\ \44S3jj5r\R.R0SGS4\ S+S4S5jj5r\R.R0S+\R 4S6j5rSJS\S\\ S+S4S7jjr SKS8\RBS9\\ \\"\4S:\ S;\ S<\ S=\ S+\ \"\RB\ \RB\"\RB444S>jjr#SLS9\ \\"\4S?\ S@\ 4SAjjr$S8\RBS+\RB4SBjr%SIS8\RBSC\ S+\RB4SDjjr&S8\RBS+\RB4SEjr'SFr(U=r)$)Mr#ia Swin Transformer V2. A hierarchical vision transformer using shifted windows for efficient self-attention computation with continuous position bias. A PyTorch impl of : `Swin Transformer V2: Scaling Up Capacity and Resolution` - https://arxiv.org/abs/2111.09883 r*r(avg`r)r)r)r(rB rFTrrg?r)rrrrNr: patch_sizein_chans num_classes global_pool embed_dimdepths.rAr%rstrict_img_sizerrB drop_rateproj_drop_rateattn_drop_ratedrop_path_raterrpretrained_window_sizesc n>[T U]5 UUS.nX@lX0lUS;deXPlSUl[ U5UlX`l[USURS- --5=Ul Ul /Ul [U[[45(d4[!UR5Vs/sHn[USU--5PM nn[#S"UUUUSUU SS.UD6UlUR$R&n[)UUSS 9n/nUSnSn[!UR5HnUUnU[+S"0S U_S U_S USU-USU-4_S UU_SUS:_SUU_SU _SU _SU (+_SU _SU _SU_SU_SUU_SU_SU_SUU_UD6/- nUnUS:aUS-nU=R[-USU-SU3S9/- sl M [.R0"U6UlU"UR40UD6Ul[7URU4UUUR S.UD6UlUR;SS 9 g!s snf)#a Args: img_size: Input image size. patch_size: Patch size. in_chans: Number of input image channels. num_classes: Number of classes for classification head. embed_dim: Patch embedding dimension. depths: Depth of each Swin Transformer stage (layer). num_heads: Number of attention heads in different layers. window_size: Window size. mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. drop_rate: Head dropout rate. proj_drop_rate: Projection dropout rate. attn_drop_rate: Attention dropout rate. drop_path_rate: Stochastic depth rate. norm_layer: Normalization layer. act_layer: Activation layer type. patch_norm: If True, add normalization after patch embedding. pretrained_window_sizes: Pretrained window sizes of each layer. output_fmt: Output tensor format if not None, otherwise output 'NHWC' by default. rH)r?NHWCr)rr)r:rFrGrJrrL output_fmtT) stagewiser@rrrrrAr%rrrrBrErDrrrrFr*layers.)num_chsrmodule) pool_typerM input_fmtF needs_resetNr)rTrUrHrGrIrUlen num_layersrJr num_featureshead_hidden_size feature_infor)rr*r(r patch_embed grid_sizerrdictrVr[layersrrhead init_weights)!rir:rFrGrHrIrJrKrAr%rrLrrBrMrNrOrPrrrQrIrJkwargsrjr!rddprrfin_dimscalerrms! r7rUSwinTransformerV2.__init__s`  /& k)))& f+"47 A$//\]J]D^8^4__D1)eT]33:?:PQ:PQYa/0:PIQ&  !l!+    $$.. '$O1t'AlG -#,A,%"719N!OQi  q5  $A, ("2"10$"))a&$ &!"(?q'A% F(F1u    $w!e)V]^_]`Ta"b!c c 3(6mmV, t007B7 "     "oo      e,uRs/H2r]r&cUR[URUS95 URHnUR 5 M g)zInitialize model weights. Args: needs_reset: If True, call reset_parameters() on modules (default for after to_empty()). If False, skip reset_parameters() (for __init__ where modules already self-initialized). r\N)applyr _init_weightsrfr9)rir]blys r7rhSwinTransformerV2.init_weightsSs7 74--;GH;;C  ! ! #r9mc0[U[R5(aM[URSS9 UR b+[R RUR S5 ggU(a#[US5(aUR5 ggg)zInitialize weights for Linear layers. Args: m: Module to initialize. needs_reset: Whether to call reset_parameters() on modules. g{Gz?)stdNrrh) r)rVr\rr|rLrqrrhasattrrh)rirsr]s r7rpSwinTransformerV2._init_weights^sm a # # !(( ,vv!!!!&&!," WQ(:;;   <[r9 window_ratioc^UcUb/URRXS9 URRnUc!Ub[WVs/sHowU-PM sn5n[ UR 5H6upS[ US- S5-n U RWSU -USU -4UUS9 M8 gs snf)aUpdates the image resolution, window size, and so the pair-wise relative positions. Args: img_size (Optional[Tuple[int, int]]): New input resolution, if None current resolution is used patch_size (Optional[Tuple[int, int]): New patch size, if None use current patch size window_size (Optional[int]): New window size, if None based on new_img_size // window_div window_ratio (int): divisor for calculating window size from patch grid size always_partition: always partition / shift windows even if feat size is < window N)r:rFr)rrr0)rcrrdr enumeraterfr) rir:rFr%rxrrdrindexstage stage_scales r7r SwinTransformerV2.set_input_sizels"  :#9    + +X + U((22I  <#;I FIql!2I FGK%dkk2LEs519a00K  $Q<;6 ! 8ST'!1 ! 3!GsB*c[5nUR5H;up#[SVs/sHoDU;PM sn5(dM*URU5 M= U$s snf)zyGet parameter names that should not use weight decay. Returns: Set of parameter names to exclude from weight decay. )r^rZ)set named_modulesradd)rinodnrskws r7no_weight_decay!SwinTransformerV2.no_weight_decaysWe&&(DA&@A&@!G&@ABB ) BsA coarsec0[SU(aSS9$/SQS9$)zCreate parameter group matcher for optimizer parameter groups. Args: coarse: If True, use coarse grouping. Returns: Dictionary mapping group names to regex patterns. z^absolute_pos_embed|patch_embedz^layers\.(\d+)))z^layers\.(\d+).downsample)r)z^layers\.(\d+)\.\w+\.(\d+)N)z^norm)i)stemr+)re)rirs r7 group_matcherSwinTransformerV2.group_matchers)3(.$  5  r9enablec6URH nXlM g)zeEnable or disable gradient checkpointing. Args: enable: If True, enable gradient checkpointing. N)rfr&)rirls r7set_grad_checkpointing(SwinTransformerV2.set_grad_checkpointingsA#) r9c.URR$)zGGet the classifier head. Returns: The classification head module. )rgfcrws r7get_classifier SwinTransformerV2.get_classifiersyy||r9cFXlURRX5 g)z}Reset the classification head. Args: num_classes: Number of classes for new head. global_pool: Global pooling type. N)rHrgreset)rirHrIs r7reset_classifier"SwinTransformerV2.reset_classifiers'  1r9r$indicesr stop_earlyrUintermediates_onlyclUS;dS5e/n[[UR5U5upURU5n[UR5n [R R 5(dU(d URn OURSU S-n [U 5HjupU "U5nX;dMU(aXS- :XaURU5nOUnURSSSS5R5nURU5 Ml U(aU$URU5nX4$)aForward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence norm: Apply norm layer to compatible intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features Returns: )NCHWzOutput shape must be NCHW.Nrrr(r)) rr^rfrcrXr4r5rzrr0r1append)rir$rrrrUr intermediates take_indices max_index num_stagesstagesr!r|x_inters r7forward_intermediates'SwinTransformerV2.forward_intermediatess*Y&D(DD& "6s4;;7G"Q    Q % 99 ! ! # #:[[F[[)a-0F!&)HAaA Aa/"iilGG!//!Q15@@B$$W-*  IIaLr9 prune_norm prune_headc[[UR5U5upEURSUS-UlU(a[R"5UlU(aUR SS5 U$)z?Prune layers not required for specified intermediates. NrrrS)rr^rfrVrrr)rirrrrrs r7prune_intermediate_layers+SwinTransformerV2.prune_intermediate_layerss[#7s4;;7G"Q kk.9q=1  DI   ! !!R (r9clURU5nURU5nURU5nU$)zForward pass through feature extraction layers. Args: x: Input tensor of shape (B, C, H, W). Returns: Feature tensor of shape (B, H', W', C). )rcrfrrir$s r7forward_features"SwinTransformerV2.forward_features s3   Q  KKN IIaLr9 pre_logitscRU(aURUSS9$URU5$)zForward pass through classification head. Args: x: Feature tensor of shape (B, H, W, C). pre_logits: If True, return features before final linear layer. Returns: Logits tensor of shape (B, num_classes) or pre-logits. T)r)rg)rir$rs r7 forward_headSwinTransformerV2.forward_heads&1;tyyty,L ! Lr9cJURU5nURU5nU$)zForward pass through the model. Args: x: Input tensor of shape (B, C, H, W). Returns: Logits tensor of shape (B, num_classes). )rrrs r7rSwinTransformerV2.forward%s)  ! !! $   a r9) rJrbrIrgrarGrfrrHr`r_rUrc)T)NNNrNFr)NFFrF)rFT)*rrrrrrVr r rr;r rrr rr r rUrhrprrrXr4ignorer rrrrrrrrrrrrrrrrrs@r7r#r#s +.#$&2)7-.%*$(!!!$&$&$'.4*,,,7C/x-'x-x- x-  x-  x-x-#s(Ox-S#Xx-+x-#x-"x-x-x-x- "!x-""#x-$"%x-&S(]+'x-(RYY)x-*&+38_+x-x-t $ $ $ !ryy !t !t ! 374859*+/3 uS#X/!sCx1"%S/2  #3-  'tn @ YY S   YY D T#s(^  $ YY*T*T** YY 2C2hsm2W[28<$$',0 ||0 eCcN340  0  0  0 !%0  tELL!5tELL7I)I#JJ K0 h./$# 3S >*  %,, 5<<  Mell M M M  %,,  r9 state_dictmodelc URSU5nURSU5nSU;n0nSSKnUR5HupV[SVs/sHowU;PM sn5(aM*SU;abURR R Ru pn URSU :wdURS U :wa[UX4S S S S 9nU(d&URS SU5nURSS5nXcU'M U$s snf)a-Filter and process checkpoint state dict for loading. Handles resizing of patch embeddings and relative position tables when model size differs from checkpoint. Args: state_dict: Checkpoint state dictionary. model: Target model to load weights into. Returns: Filtered state dictionary. rrzhead.fc.weightrN)rSrRrzpatch_embed.proj.weightrr,bicubicT) interpolation antialiasverbosezlayers.(\d+).downsamplecDS[URS55S-S3$)NrWrz .downsample)rgroup)r$s r7&checkpoint_filter_fn..Vs"ws177ST:YZGZF[[f=gr9zhead.zhead.fc.) getreitemsrrcrdr|r.rsubreplace) rrnative_checkpointout_dictrrrrrr3r4s r7checkpoint_filter_fnr3s4J j9J(J6H   "  ab a1Q ab c c  $ )**//66<.hs\.[da.[r$rK)rrrr out_indicesT)flatten_sequentialr)pretrained_filter_fn feature_cfg)rrzrpoprr#rre)rrridefault_out_indicesrrs r7_create_swin_transformer_v2r]sh \i 8\8Z.[\\**],?@K 7 1DkJ    E Lr9c 4USSSSSS[[SSS S . UE$) Nr>)r(r)rrg?rTzpatch_embed.projzhead.fcmit) urlrH input_size pool_sizecrop_pctrfixed_input_sizemeanru first_conv classifierlicenser )rris r7_cfgrss5=v%.B(    #  r9ztimm/z{https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12to16_192to256_22kto1k_ft.pth) hf_hub_idrz{https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12to24_192to384_22kto1k_ft.pth)r(r)rDrDr)rrrrrz|https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12to16_192to256_22kto1k_ft.pthz|https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12to24_192to384_22kto1k_ft.pthzfhttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_tiny_patch4_window8_256.pthzghttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_tiny_patch4_window16_256.pthzghttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_small_patch4_window8_256.pthzhhttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_small_patch4_window16_256.pthzfhttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window8_256.pthzghttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window16_256.pthzkhttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12_192_22k.pthiQU)r(r)rBrB)rrrHrrzlhttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12_192_22k.pth) 2swinv2_base_window12to16_192to256.ms_in22k_ft_in1k2swinv2_base_window12to24_192to384.ms_in22k_ft_in1k3swinv2_large_window12to16_192to256.ms_in22k_ft_in1k3swinv2_large_window12to24_192to384.ms_in22k_ft_in1kzswinv2_tiny_window8_256.ms_in1kz swinv2_tiny_window16_256.ms_in1kz swinv2_small_window8_256.ms_in1kz!swinv2_small_window16_256.ms_in1kzswinv2_base_window8_256.ms_in1kz swinv2_base_window16_256.ms_in1k!swinv2_base_window12_192.ms_in22k"swinv2_large_window12_192.ms_in22kc J[SSSSS9n[SSU0[U40UD6D6$)z"Swin-T V2 @ 256x256, window 16x16.rr@rArCr%rJrKrAr)swinv2_tiny_window16_256rerrri model_argss r7rrsD"B:>XQW>X ZZr9c J[SSSSS9n[SSU0[U40UD6D6$)z Swin-S V2 @ 256x256, window 8x8.rr@rrCrr)swinv2_small_window8_256rrs r7rrsD!r-SabJ &" Y/9 Y=A*=WPV=W YYr9c J[SSSSS9n[SSU0[U40UD6D6$)z"Swin-B V2 @ 256x256, window 16x16.rrr*rr rr)swinv2_base_window16_256rrs r7rrD"MUcdJ &" Y/9 Y=A*=WPV=W YYr9c J[SSSSS9n[SSU0[U40UD6D6$)z Swin-B V2 @ 256x256, window 8x8.rrrrrr)swinv2_base_window8_256rrs r7rrsC!s=TbcJ &! X.8 X<@B:>XQW>X ZZr9c L[SSSSSS9n[SSU0[U40UD6D6$) zQSwin-L V2 @ 192x192, trained at window 12x12, fine-tuned to 256x256 window 16x16.rrrrrrr)"swinv2_large_window12to16_192to256rrs r7rr K#m /1J ', c9C cGKJGaZ`Ga ccr9c L[SSSSSS9n[SSU0[U40UD6D6$) zQSwin-L V2 @ 192x192, trained at window 12x12, fine-tuned to 384x384 window 24x24.rErrrrrr)"swinv2_large_window12to24_192to384rrs r7rrrr9rrrrrr)swinv2_base_window12_192_22k)swinv2_base_window12to16_192to256_22kft1k)swinv2_base_window12to24_192to384_22kft1kswinv2_large_window12_192_22k*swinv2_large_window12to16_192to256_22kft1k*swinv2_large_window12to24_192to384_22kft1kr)rS)Nrrs functoolsrtypingrrrrrr r r r rXtorch.nnrVtorch.nn.functionalrr timm.datarr timm.layersrrrrrrrrrrr_builderr _featuresr _features_fxr _manipulater _registryr r!r"__all__rr rr8r<r r>rrrr#r;rrrr default_cfgsrrrrrrrr r rrrrrr9r7r)s OOO A;;;;*+3#YY  #uS#X./ <<38_ \\&  38_ S/  \\   ,QbiiQhpRYYpf&299&RM3RYYM3`a aH 'T#u||*;%<'RYY'SWX[]b]i]iXiSj'T$Uf,%:> J;;? J Hs; <@ K<<@ K Hs< (, t()- u))- u)*. v*(, t()- u) *. ymv* +/ zmv+e7&7 tYYDUYYXXCTXXZ$ZEVZZYYDUYYYYDUYYXXCTXXYYDUYYb$bM^bbb$bM^bbZ$ZEVZZc4cN_ccc4cN_ccH$G1e1e%I2g2g 'r9