RЦiISrSSKrSSKJr SSKJrJrJrJrJ r J r J r J r J r SSKrSSKJr SSKJs Jr SSKJrJr SSKJrJrJrJrJrJrJrJrJ r J!r!J"r"J#r# SSK$J%r% SS K&J'r' SS K(J)r) SS K*J+r+J,r, S /r-S5S \ \.\.4S\R^4Sjjr0"SS\Rb5r2"SS\Rb5r3"SS\Rb5r4"SS \Rb5r5S6S\6S\\6\44Sjjr7\+"\7"SS9\7"SSSS9\7"SSS9\7"SS9\7"SSSS9\7"SS SS9\7"SSS9\7"S\\S!9\7"S\\S!9\7"SS\\S"9\7"SS#\\S$9\7"SS#\\S$9\7"SS\\S"9S%. 5r8S7S&\\6\R^4S'\RbS(\6S)\9S\\6\R^44 S*jjr:S8S+\6S,\9S\54S-jjr;\,S8S,\9S\54S.jj5r<\,S8S,\9S\54S/jj5r=\,S8S,\9S\54S0jj5r>\,S8S,\9S\54S1jj5r?\,S8S,\9S\54S2jj5r@\,S8S,\9S\54S3jj5rA\,S8S,\9S\54S4jj5rBg)9aBEiT: BERT Pre-Training of Image Transformers (https://arxiv.org/abs/2106.08254) Model from official source: https://github.com/microsoft/unilm/tree/master/beit @inproceedings{beit, title={{BEiT}: {BERT} Pre-Training of Image Transformers}, author={Hangbo Bao and Li Dong and Songhao Piao and Furu Wei}, booktitle={International Conference on Learning Representations}, year={2022}, url={https://openreview.net/forum?id=p-BhZSz59o4} } BEiT-v2 from https://github.com/microsoft/unilm/tree/master/beit2 @article{beitv2, title={{BEiT v2}: Masked Image Modeling with Vector-Quantized Visual Tokenizers}, author={Zhiliang Peng and Li Dong and Hangbo Bao and Qixiang Ye and Furu Wei}, year={2022}, eprint={2208.06366}, archivePrefix={arXiv}, primaryClass={cs.CV} } At this point only the 1k fine-tuned classification weights and model configs have been added, see original source above for pre-training models and procedure. Modifications by / Copyright 2021 Ross Wightman, original copyrights below N)partial) AnyCallableDictListOptionalSetTupleTypeUnion)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) PatchEmbedMlpSwiGLU LayerNormDropPathcalculate_drop_path_rates trunc_normal_use_fused_attnresample_patch_embedresample_abs_pos_embedresize_rel_pos_bias_tablendgrid)build_model_with_cfg)feature_take_indices) checkpoint)generate_default_cfgsregister_modelBeit window_sizereturnc (SUS-S- SUS-S- -S-nUSUS-n[R"[[R"USU[RS9[R"USU[RS955n[R "US5nUSS2SS2S4USS2SSS24- nUR SSS5R5nUSS2SS2S4==USS- - ss'USS2SS2S4==USS- - ss'USS2SS2S4==SUS-S- -ss'[R"US-4S-XRS9nURS5USS2SS24'US- USSS24'US- USS2S4'US- US 'U$) aGenerate relative position index for window-based attention. Creates a lookup table for relative position indices between all pairs of positions within a window, including special handling for cls token interactions. Args: window_size: Height and width of the attention window. Returns: Relative position index tensor of shape (window_area+1, window_area+1) where +1 accounts for the cls token. rrdevicedtypeN)sizer(r))rr) torchstackrarangelongflattenpermute contiguouszerosr)sum)r"r(num_relative_distance window_areacoordscoords_flattenrelative_coordsrelative_position_indexs O/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/models/beit.pygen_relative_position_indexr<IsQ/!3KN8JQ8NORSSa.;q>1K [[ [^F%**E [^F%**EF]]61-N$Q4Z0>!T1*3MMO%--aA6AACOAq!G A 22Aq!G A 22Aq!GKN 2Q 66#kk a/AA/Ef\q\qr&5&9&9"&=ABF#%:Q%>AqrE"%:Q%>ABE"$9A$=D! ""cR^\rSrSr%Sr\R R\\ S'SS\ S\ S\S\S\ S \ S \ \ \ \ 4S \ \ 4U4S jjjrS \R4SjrSS\RS\ \RS \R4SjjrSSjrSSjrSSjrSrU=r$) AttentionlzMulti-head attention module with optional relative position bias. Implements multi-head self-attention with support for relative position bias and fused attention operations. Can use either standard or custom head dimensions. fused_attndim num_headsqkv_biasqkv_bias_separate attn_drop proj_dropr" attn_head_dimc t>XS.n [TU]5 X lX-n UbUn XR-n U S-Ul[ 5UlX@l[R"XS-4SS0U D6Ul U(a[R"[R"U 40U D65Ul URS[R"U 40U D6SS9 [R"[R"U 40U D65UlOSUl SUlSUlU(aXplS US -S - S US -S - -S-UlUS US -n[R"[R"UR$U40U D65UlURS [R"US -US -4U [R(S9SS9 OSUlSUlSUl[R,"U5Ul[R"X40U D6Ul[R,"U5UlUR55 g) aInitialize attention module. Args: dim: Input feature dimension. num_heads: Number of attention heads. qkv_bias: If True, add learnable bias to query, key, value projections. qkv_bias_separate: If True, use separate bias for q, k, v projections. attn_drop: Dropout rate for attention weights. proj_drop: Dropout rate for output projection. window_size: Window size for relative position bias. If None, no relative position bias. attn_head_dim: Dimension per attention head. If None, uses dim // num_heads. r'Ngr&biasFk_bias persistentr%rrr:)super__init__rCscalerrArEnnLinearqkv Parameterr,emptyq_biasregister_bufferv_biasrKr"r5relative_position_bias_tabler/r:DropoutrFprojrGreset_parameters)selfrBrCrDrErFrGr"rHr(r)ddhead_dim all_head_dimr6 __class__s r;rOAttention.__init__ts2 / "#  $$H..0 % (*!299S"2EE"E ,,u{{<'F2'FGDK  5;;|+Jr+JW\ ],,u{{<'F2'FGDKDKDKDK * *+k!n*9K02  D66 HRH1JD -  ) [1_kAo>vUZU_U_`  !  $D 04D -+/D (I.IIl626 I. r=r#cXURURRS5RURSURS-S-URSURS-S-S5nUR SSS5R 5nUR S5$)zGet relative position bias for the attention window. Returns: Relative position bias tensor of shape (1, num_heads, window_area+1, window_area+1). r+rrr%)rYr:viewr"r1r2 unsqueezer]relative_position_biass r;_get_rel_pos_biasAttention._get_rel_pos_biass "&!B!B  ( ( - -b 1"3374   Q $"2"21"5 5 9   Q $"2"21"5 5 924? "8!?!?1a!H!S!S!U%//22r=xshared_rel_pos_biascRURup4nURcURU5nO[R"URUR UR 45nUR(aURU5nXg- nO)[R"XRRUS9nURX4SURS5RSSSSS5nURS5upn UR(akSn UR bUR#5n UbX-n OUbUn [R$"XU U UR&(aUR(R*OS S 9nOlXR,-nXR/S S5-n UR bXR#5-n UbX-n U R1SS 9n UR)U 5n X-nUR/SS5RX4U5nUR3U5nUR5U5nU$) zForward pass of attention module. Args: x: Input tensor of shape (batch_size, num_tokens, dim). shared_rel_pos_bias: Optional shared relative position bias from parent module. Returns: Output tensor of shape (batch_size, num_tokens, dim). N)weightrJr&r+r%rr) attn_mask dropout_prB)shaperVrSr,catrKrXrEFlinearrmreshaperCr1unbindrArYrhscaled_dot_product_attentiontrainingrFprP transposesoftmaxr[rG) r]rjrkBNCrSrDqkv rel_pos_biasattns r;forwardAttention.forwards''a ;; ((1+Cyy$++t{{DKK!HIH%%hhqkhhqxHkk!4>>26>>q!Q1M**Q-a ??L00<#557 &2#/#EL$02 ..a&.2mm$..**A JJA B++D00<4466".1<>$'DA KK1  % %aA . IIaL NN1 r=cNURbR[RRUR5 [RRUR5 UR b)[RRUR 5 UR 5 gz"Initialize parameters and buffers.N)rVrQinitzeros_rXrY _init_buffersr]s r;r\Attention.reset_parameterssb ;; " GGNN4;; ' GGNN4;; '  , , 8 GGNN4<< = r=cURbURR5 URbBURR[ UR URR S95 gg).Compute and fill non-persistent buffer values.Nr()rKzero_r:copy_r<r"r(rs r;rAttention._init_bufferss] ;; " KK     ' ' 3  ( ( . .+D,<,UUS.n[TU]5 U "U40UD6Ul[U4UUUUU US.UD6UlU S:a [ U 5O[ R"5UlU "U40UD6Ul U(a*[SU[X-5U(aU OSUS.UD6Ul O*[SU[X-5U U(aU OSUS.UD6Ul U S:a [ U 5O[ R"5UlXlU (aa[ R "["R$"U40UD65Ul[ R "["R$"U40UD65UlOSuUlUlUR+5 g) aInitialize transformer block. Args: dim: Input feature dimension. num_heads: Number of attention heads. qkv_bias: If True, add learnable bias to query, key, value projections. mlp_ratio: Ratio of MLP hidden dimension to input dimension. scale_mlp: If True, apply layer normalization in MLP. swiglu_mlp: If True, use SwiGLU activation in MLP. proj_drop: Dropout rate for projections. attn_drop: Dropout rate for attention. drop_path: Drop path rate for stochastic depth. init_values: Initial values for layer scale. If None, no layer scale. act_layer: Activation function class. norm_layer: Normalization layer class. window_size: Window size for relative position bias in attention. attn_head_dim: Dimension per attention head. r')rCrDrFrGr"rHroN) in_featureshidden_featuresrdrop)rrrrrNN)rNrOnorm1r?rrrQIdentity drop_path1norm2rrmlpr drop_path2rrTr,rUgamma_1gamma_2r\)r]rBrCrDrrrrGrFrrrrr"rHr(r)r^ras r;rOBlock.__init__srJ / *r*    #'     2;R(9-R[[]*r*  #CO 4)2:   DH #CO 4#)2:   DH2;R(9-R[[]& << C(>2(>?DL<< C(>2(>?DL)3 &DL$, r=r#cURbi[RRURUR5 [RRUR UR5 gg)zInitialize parameters.N)rrQr constant_rrrs r;r\Block.reset_parameterspsM << # GG  dllD,<,< = GG  dllD,<,< = $r=rjrkc URccXRURURU5US95-nXR UR UR U555-nU$XRURURURU5US9-5-nXR URUR UR U55-5-nU$)zForward pass of transformer block. Args: x: Input tensor of shape (batch_size, num_tokens, dim). shared_rel_pos_bias: Optional shared relative position bias. Returns: Output tensor of shape (batch_size, num_tokens, dim). rk)rrrrrrrr)r]rjrks r;r Block.forwardvs << OODIIdjjmQdI$effAOODHHTZZ]$;< HU\\X4S.n[TU]5 XlUSUS-UlSUS-S- SUS-S- -S-n[R "[ R"Xb40UD65UlURS[ R"URS-URS-4U[ RS9SS9 UR5 g ) zInitialize relative position bias module. Args: window_size: Height and width of the attention window. num_heads: Number of attention heads. r'rrr%r&r:FrLN) rNrOr"r6rQrTr,rUrYrWr/r\)r]r"rCr(r)r^r5ras r;rORelativePositionBias.__init__s / &&q>KN:!"[^!3a!7A AS6\\\\ \4S7\ S8\ S9\ S:\ S;\ S)\\ \R>\\R>\ \R>444S<jjr!SJS6\\\ \4S=\ S>\ S)\ \4S?jjr"S5\R>S)\R>4S@jr#SGS5\R>SA\ S)\R>4SBjjr$S5\R>S)\R>4SCjr%SDr&U=r'$)Kr!izBEiT: BERT Pre-Training of Image Transformers. Vision Transformer model with support for relative position bias and shared relative position bias across layers. Implements both BEiT v1 and v2 architectures with flexible configuration options. r&avg TrFroNgMbP?img_size patch_sizein_chans num_classes global_pool embed_dimdepthrCrDrrr drop_rate pos_drop_rateproj_drop_rateattn_drop_ratedrop_path_raterruse_abs_pos_embuse_rel_pos_biasuse_shared_rel_pos_biashead_init_scalec>UUS.n[T U]5 X@lX0lXPlU=Ul=UlUlSUlSUl [SUUUUS.UD6Ul URRn[URS5(aURR5OUn[R "["R$"SSU40UD65UlU(a0[R "["R$"SUS-U40UD65OSUl[R*"US9UlU(a([/SURR0US.UD6UlOSUl[5UU5n[R6"[9U5Vs/sH;n[;SUUU U U U UUUUUUU(aURR0OSS . UD6PM= sn5Ul[9U5Vs/sHn[?S U3UUS 9PM snUl URS :HnU(a[RB"5OU"U40UD6Ul"U(a U"U40UD6O[RB"5Ul#[R*"U 5Ul$US :a[RJ"Xd40UD6O[RB"5Ul&UUl'URQSS9 gs snfs snf)aInitialize BEiT model. Args: img_size: Input image size. patch_size: Patch size for patch embedding. in_chans: Number of input image channels. num_classes: Number of classes for classification head. global_pool: Type of global pooling ('avg' or ''). embed_dim: Embedding dimension. depth: Number of transformer blocks. num_heads: Number of attention heads. qkv_bias: If True, add learnable bias to query, key, value projections. mlp_ratio: Ratio of MLP hidden dimension to embedding dimension. swiglu_mlp: If True, use SwiGLU activation in MLP. scale_mlp: If True, apply layer normalization in MLP. drop_rate: Dropout rate. pos_drop_rate: Dropout rate for position embeddings. proj_drop_rate: Dropout rate for projections. attn_drop_rate: Dropout rate for attention. drop_path_rate: Stochastic depth rate. norm_layer: Normalization layer class. init_values: Initial values for layer scale. use_abs_pos_emb: If True, use absolute position embeddings. use_rel_pos_bias: If True, use relative position bias in attention. use_shared_rel_pos_bias: If True, share relative position bias across layers. head_init_scale: Scale factor for head initialization. r'rF)rrrr feat_ratioN)r|)r"rC) rBrCrDrrrrGrFrrrr"zblocks.)modulenum_chs reductionrr needs_resetr))rNrOrrr num_featureshead_hidden_sizernum_prefix_tokensgrad_checkpointingr patch_embed num_patcheshasattrrrQrTr,rU cls_token pos_embedrZpos_dropr grid_sizerr ModuleListrangerblocksdict feature_infornormfc_norm head_droprRheadr init_weights)!r]rrrrrrrrCrDrrrrrrrrrrrrrrr(r)r^rrdpri use_fc_normras! r;rO Beit.__init__sn / & &ENNND1DN!""'% !    &&22 -4T5E5E|-T-TD   ' ' )Zdekk!Q &HR&HI[jekk![1_i&VSU&VWpt ]3 " 4! ,,66#!!D  !%D '>mm 5\!%# "  #!##%((a&%':JD,,66PT  "!%#$ $QVV[P\^P\1D'! yA FP\^&&%/ %0BKKMj6Qb6Q 6Az)2r2r{{} I.?JQBIIi;;TVT_T_Ta . e,9%#"^s AK/K rr#cUR[URUS95 URb[ URSS9 [ UR SS9 UR 5 UR(a[UR[R5(a[ URRSS9 [R"5 URRRUR5 URR RUR5 SSS5 ggg!,(df  g=f)zInitialize model weights. Args: needs_reset: If True, call reset_parameters() on modules that have it. Set to False when modules have already self-initialized in __init__. rN{Gz?std)applyr _init_weightsrrrfix_init_weightr isinstancer rQrRrmr,no_gradmul_rJ)r]rs r;r Beit.init_weights?s 74--;GH >> % $..c 2dnn#.    Jtyy"))$D$D $))** 4   %%d&:&:; ##D$8$89!%E  s AE Ec[R"5 [UR5Hup[R "SUS--5nUR RRRU5 URRRRU5 M SSS5 g!,(df  g=f)zFix initialization weights according to BEiT paper. Rescales attention and MLP weights based on layer depth to improve training stability. g@rN) r,r enumeratermathsqrtrr[rmdiv_rfc2)r]layer_idlayerrPs r;rBeit.fix_init_weightSs{ ]]_#,T[[#9 #A"67 &&++E2 $$))%0$:__s BB88 Cmc0[U[R5(aM[URSS9 UR b+[R RUR S5 ggU(a#[US5(aUR5 ggg)zInitialize model weights. Args: m: Module to initialize. needs_reset: If True, call reset_parameters() on modules that have it. rrNrr\) rrQrRrrmrJrrrr\)r]r'rs r;rBeit._init_weights_sm a # # !(( ,vv!!!!&&!," WQ(:;;   <[r=crSS1nUR5Hup#SU;dM URU5 M U$)zyGet parameter names that should not use weight decay. Returns: Set of parameter names to exclude from weight decay. rrrY)named_parametersadd)r]nwdn_s r;no_weight_decayBeit.no_weight_decayms=K())+DA-2 , r=enablecXlg)zeEnable or disable gradient checkpointing. Args: enable: If True, enable gradient checkpointing. N)r)r]r2s r;set_grad_checkpointingBeit.set_grad_checkpointingzs #)r=coarsec[SSS/S9nU$)zCreate parameter group matcher for optimizer parameter groups. Args: coarse: If True, use coarse grouping. Returns: Dictionary mapping group names to regex patterns. z-^cls_token|pos_embed|patch_embed|rel_pos_bias)z^blocks\.(\d+)N)z^norm)i)stemr)r)r]r6matchers r; group_matcherBeit.group_matchers!A-/CD r=cUR$)zGGet the classifier head. Returns: The classification head module. )r rs r;get_classifierBeit.get_classifiersyyr=cXlUbX lUS:a'[R"URU5Ulg[R "5Ulg)z}Reset the classification head. Args: num_classes: Number of classes for new head. global_pool: Global pooling type. Nr)rrrQrRrrr )r]rrs r;reset_classifierBeit.reset_classifiersB'  "* >IAoBIIdnnk: SUS^S^S` r=rjindicesreturn_prefix_tokensr  stop_early output_fmtintermediates_onlyc US;dS5eUS:Hn/n [[UR5U5upURuppUR U5n[ R "URRURSSS5U4SS9nURbXR-nURU5nURbUR5OSn[ RR5(dU(d URnOURSU S-n[U5H~unnUR(a/[ RR5(d [!UUUS 9nOU"UUS 9nUU ;dMUU R#U(aUR%U5OU5 M UR&(aJU Vs/sHnUSS2SUR&24PM nnU Vs/sHnUSS2UR&S24PM n nU(acURR)X45unnU Vs/sH7nUR+U UUS5R-SS SS 5R/5PM9 n n[ RR5(dU(a[1[3U W55n U(aU $UR%U5nX4$s snfs snfs snf) aForward pass that returns intermediate feature maps. Args: x: Input image tensor of shape (batch_size, channels, height, width). indices: Block indices to return features from. If int, returns last n blocks. return_prefix_tokens: If True, return both prefix and spatial tokens. norm: If True, apply normalization to intermediate features. stop_early: If True, stop at last selected intermediate. output_fmt: Output format ('NCHW' or 'NLC'). intermediates_only: If True, only return intermediate features. Returns: If intermediates_only is True, returns list of intermediate tensors. Otherwise, returns tuple of (final_features, intermediates). )NCHWNLCz)Output format must be one of NCHW or NLC.rHrr+rrsNrr&r%)rlenrrtrr,rurexpandrrrr is_scriptingrrrappendr rdynamic_feat_sizerxr1r2listzip)r]rjrBrCr rDrErFrx intermediates take_indices max_indexrr/heightwidthrrrblky prefix_tokensHWs r;forward_intermediatesBeit.forward_intermediatessz2_,Y.YY,& "6s4;;7G"Q  ggf   Q  IIt~~,,QWWQZR@!D! L >> %NN"A MM! .2.?.?.Kt((*QU 99 ! ! # #:[[F[[)a-0F'FAs&&uyy/E/E/G/GsA<H|<L $$TTYYq\qA(  ! !ERS]Qq!D$:$:"::;]MSDQRMqQq$"8"8"99:MMR ##55voFDAq^kl^kYZQYYq!Q3;;Aq!QGRRT^kMlyy%%'',@ ]M!BCM  IIaLTRms K0K;>K! prune_norm prune_headc[[UR5U5upEURSUS-UlU(a[R"5UlU(a,[R"5UlURSS5 U$)aPrune layers not required for specified intermediate outputs. Args: indices: Indices of blocks to keep. prune_norm: If True, remove final normalization. prune_head: If True, remove classification head. Returns: List of indices that were kept. Nrr)rrJrrQrr r r@)r]rBr]r^rRrSs r;prune_intermediate_layersBeit.prune_intermediate_layerssh #7s4;;7G"Q kk.9q=1  DI ;;=DL  ! !!R (r=cURU5n[R"URR UR SSS5U4SS9nUR bXR -nURU5nURbUR5OSnURHInUR(a/[RR5(d [X1US9nMCU"XS9nMK URU5nU$)zForward pass through feature extraction layers. Args: x: Input tensor of shape (batch_size, channels, height, width). Returns: Feature tensor of shape (batch_size, num_tokens, embed_dim). rr+rrsNr)rr,rurrKrtrrrrrrrLrr )r]rjrrVs r;forward_featuresBeit.forward_features s   Q  IIt~~,,QWWQZR@!D! L >> %NN"A MM! .2.?.?.Kt((*QU ;;C&&uyy/E/E/G/Gs<H<  IIaLr= pre_logitscUR(a;URS:Xa"USS2URS24RSS9OUSS2S4nURU5nUR U5nU(aU$UR U5$)aForward pass through classification head. Args: x: Feature tensor of shape (batch_size, num_tokens, embed_dim). pre_logits: If True, return features before final linear layer. Returns: Logits tensor of shape (batch_size, num_classes) or pre-logits. rNrrsr)rrmeanr r r )r]rjrfs r; forward_headBeit.forward_head!s   =A=M=MQV=V!T++,,-22q29\]^_ab^b\cA LLO NN1 q0DIIaL0r=cJURU5nURU5nU$)zForward pass through the model. Args: x: Input tensor of shape (batch_size, channels, height, width). Returns: Logits tensor of shape (batch_size, num_classes). )rdri)r]rjs r;r Beit.forward1s)  ! !! $   a r=)rrrr rrrr r rrrr rrrrrrr)TrFr)NFFFrHF)rFT)(rrrrrrr rr strrrr rQrrrOr rrr,rignorer r0r4rrr:r=r@rrr[rardrirrrrs@r;r!r!s5868#$ !!$#!#%$&$&$&*3+/$(%*,1%*5u-CsCx01u-c5c?23u- u-  u-  u-u-u-u-u-u-u-u-u-!u- "!u-""#u-$"%u-&RYY'u-("%)u-*"+u-,#-u-.&*/u-0#1u-u-n:::( 1 !ryy !t ! YY S   YY)T)) YY D T#s(^   YY  aC ahsm a8<).$$',F ||F eCcN34F #' F  F  F F !%F  tELL!5tELL7I)I#JJ KF T./$# 3S >*  c 2%,,5<<01ell111  %,,  r=urlc $USSSSSSSSSS S S . UE$) zCreate a default configuration dictionary for BEiT models. Args: url: Model weights URL. **kwargs: Additional configuration parameters. Returns: Configuration dictionary. r)r&rrNg?bicubicT)?rsrszpatch_embed.projr z apache-2.0) rpr input_size pool_sizecrop_pct interpolationfixed_input_sizerhr first_conv classifierlicenser)rpkwargss r;_cfgr}?s6=t(    r=ztimm/) hf_hub_id)r&rg?)r~rtrviQU)r~r)r&r)r~rhr)r~rrhrgffffff?)r~rvrhr) z)beit_base_patch16_224.in22k_ft_in22k_in1kz)beit_base_patch16_384.in22k_ft_in22k_in1kz$beit_base_patch16_224.in22k_ft_in22kz*beit_large_patch16_224.in22k_ft_in22k_in1kz*beit_large_patch16_384.in22k_ft_in22k_in1kz*beit_large_patch16_512.in22k_ft_in22k_in1kz%beit_large_patch16_224.in22k_ft_in22kz*beitv2_base_patch16_224.in1k_ft_in22k_in1kz$beitv2_base_patch16_224.in1k_ft_in1kz%beitv2_base_patch16_224.in1k_ft_in22kz+beitv2_large_patch16_224.in1k_ft_in22k_in1kz%beitv2_large_patch16_224.in1k_ft_in1kz&beitv2_large_patch16_224.in1k_ft_in22k state_dictmodelrw antialiasc HURSU5nURSU5n0nUR5GHfupVSU;aMSU;abURRRR upxpUR SU :wdUR SU :wa[ UX4UUSS9nOUS :XaOUR S URR S :wa%S n [UURRU UUSS 9nOURS 5(aURUS S5n UR U RR :wd U RSU RS :wa)[UU RU RR S9nXdU'GMi U$)aFilter and process checkpoint state dict for loading. Handles resizing of patch embeddings, position embeddings, and relative position bias tables when model size differs from checkpoint. Args: state_dict: Checkpoint state dictionary. model: Target model to load weights into. interpolation: Interpolation method for resizing. antialias: If True, use antialiasing when resizing. Returns: Filtered state dictionary. rrr:zpatch_embed.proj.weightr+rrT)rwrverboserr)new_sizerrwrrrYNir)new_window_sizenew_bias_shape)getitemsrr[rmrtrrrrendswith get_submodulerYr"r) rrrwrout_dictrrOIrYrZrr's r;checkpoint_filter_fnrs4J*5JH  " $ )  $ )**//66<>>!--PQBRVWVcVcdeVfBf-$%MM#$#A#A#G#G  A#B Or=variant pretrainedc jURSS5n[[X4[[ USS9S.UD6nU$)zCreate a BEiT model. Args: variant: Model variant name. pretrained: If True, load pretrained weights. **kwargs: Additional model arguments. Returns: BEiT model instance. out_indicesr&getter)r feature_cls)pretrained_filter_fn feature_cfg)poprr!rr)rrr|rrs r; _create_beitrsF**]A.K  g 1[hG    E Lr=c T[SSSSSSSSS9n[S S U0[U40UD6D6nU$) z0BEiT base model @ 224x224 with patch size 16x16.rrrrnFT皙?rrrrCrrrrr)beit_base_patch16_224rrrr| model_argsrs r;rrsFB"#GJ  fZ f4PZKe^dKe fE Lr=c T[SSSSSSSSS9n[S S U0[U40UD6D6nU$) z0BEiT base model @ 384x384 with patch size 16x16.rrrrFTrrrrrrCrrrr)beit_base_patch16_384rrs r;rrsFs"#GJ  fZ f4PZKe^dKe fE Lr=c R[SSSSSSSS9n[S SU0[U40UD6D6nU$) z1BEiT large model @ 224x224 with patch size 16x16.rFTh㈵>rrrrCrrrr)beit_large_patch16_224rrs r;rrsDR2$HJ  gj gDQ[Lf_eLf gE Lr=c T[SSSSSSSSS9n[S S U0[U40UD6D6nU$) z1BEiT large model @ 384x384 with patch size 16x16.rrrrFTrrr)beit_large_patch16_384rrs r;rrFt2$HJ  gj gDQ[Lf_eLf gE Lr=c T[SSSSSSSSS9n[S S U0[U40UD6D6nU$) z1BEiT large model @ 512x512 with patch size 16x16.rrrrFTrrr)beit_large_patch16_512rrs r;rr rr=c T[SSSSSSSSS9n[S S U0[U40UD6D6nU$) z3BEiT v2 base model @ 224x224 with patch size 16x16.rrrrnFTrrr)beitv2_base_patch16_224rrs r;rrsFB"$HJ  hz hTR\Mg`fMg hE Lr=c R[SSSSSSSS9n[S SU0[U40UD6D6nU$) z4BEiT v2 large model @ 224x224 with patch size 16x16.rrrFTrrr)beitv2_large_patch16_224rrs r;rrsDR2$HJ  i idS]NhagNh iE Lr=r)r`)rrTrm)Crr  functoolsrtypingrrrrrr r r r r,torch.nnrQtorch.nn.functional functionalrv timm.datar r timm.layersrrrrrrrrrrrr_builderr _featuresr _manipulater _registryrr __all__rrr<rr?rrr!rnr} default_cfgsrrrrrrrrrrrr=r;rsNP OOO A    ++#< ( #U38_ #ell #Ff fRqBIIqh4D2994Dn|299|~ cT#s(^*%15226 32 -1- 37337 33 37 33 .2. 37 "(<3 -1 "(<- .2 5;O. 4817K4 .217K. /3 5;O/u?&? D5T#u||*;%<5RYY5_b5{5LPQTV[VbVbQbLc5p#4d*ddt$t$t$4Dr=