RЦijpSrSSKrSSKrSSKJrJrJr SSKJr SSK J r J r J r J r JrJrJrJrJr SSKrSSKJr SSKJs Jr SSKJrJr SSKJrJrJrJ r J!r!J"r"J#r#J$r$J%r%J&r&J'r'J(r( SS K)J*r* SS K+J,r, SS K-J.r.J/r/ SS K0J1r1J2r2 SS K3J4r4J5r5 SSK6J7r7 SSK8J9r9J:r: SS/r;\Rx"\=5r>\"SS55r?S\?S\?4Sjr@SGS\RS\\B\B4S\CS\\R\\B\B444SjjrDS\R4SjrE"SS5rFS\?S\ 4SjrG\/"S S!\R55rI\.SSS\R4S"\RS#\BS$\CS%\ \BS&\RS\ \R4 S'jj5rL\.SHS\RS"\ \RS(\MS#\BS)\CS\R4 S*jj5rN"S+S\R5rOS,rPSIS-\MS.\QS/\CS\ 4S0jjrRS1\ \M\4S2\OS\ \M\44S3jrSSJS4\MS\ \M\44S5jjrT\5"\T"S6S79\T"S6S79\T"S6S79\T"5\T"5\T"5\T"S6SS89\T"S6SS89S9.5rUSKS:\MS;\CS\O4S<jjrVSKS:\MS;\CS\O4S=jjrWSKS:\MS;\CS\O4S>jjrX\4SKS;\CS\O4S?jj5rY\4SKS;\CS\O4S@jj5rZ\4SKS;\CS\O4SAjj5r[\4SKS;\CS\O4SBjj5r\\4SKS;\CS\O4SCjj5r]\4SKS;\CS\O4SDjj5r^\4SKS;\CS\O4SEjj5r_\4SKS;\CS\O4SFjj5r`g)LaNaFlex Vision Transformer An improved version of the Vision Transformer with: 1. Encapsulated embedding and position encoding in a single module 2. Support for linear patch embedding on pre-patchified inputs 3. Support for NaFlex variable aspect, variable resolution 4. Support for FlexiViT variable patch size 5. Support for NaViT fractional/factorized position embedding Based on ideas from: - Original Vision Transformer: https://arxiv.org/abs/2010.11929 - FlexiViT: https://arxiv.org/abs/2212.08013 - NaViT: https://arxiv.org/abs/2307.06304 - NaFlex (SigLip-2): https://arxiv.org/abs/2502.14786 Hacked together by / Copyright 2025, Ross Wightman, Hugging Face N) dataclassfieldsreplace)partial) CallableDictListOptionalSetTupleTypeUnionAnyIMAGENET_INCEPTION_MEANIMAGENET_INCEPTION_STD) AttentionPoolLatentMlp LayerNormPatchDropoutWithIndicesPatchEmbedInterpolator_assert to_2tuple get_act_layerget_norm_layerapply_keep_indices_nlcdisable_compilercalculate_drop_path_rates)build_model_with_cfg)feature_take_indices)register_notrace_functionregister_notrace_module) checkpoint named_apply)register_modelgenerate_default_cfgs)EvaBlock)Blockglobal_pool_nlc NaFlexVitCfg NaFlexVitcZ\rSrSr%SrSr\\\\\44\ S'Sr \\ S'Sr \\ S'Sr \\ S 'S r \\ S 'S r\\ S 'Sr\\ S'S r\\ S'Sr\\ S'Sr\\ S'S r\\ S'Sr\\\ S'Sr\\ S'Sr\\ S'Sr\\ S'Sr\\ S'Sr\\ S'S r\\ S'Sr\\ S'Sr\ \ S 'S!r!\\\\4\ S"'S#r"\ \ S$'S r#\\ S%'S r$\\ S&'S'r%\ \ S('S)r&\\ S*'Sr'\\\\4\ S+'Sr(\\ S,'S-r)\ \ S.'S r*\\ S/'S r+\\ S0'Sr,\\ S1'Sr-\\\ S2'S3r.\ \ S4'S r/\\ S5'Sr0\\\ S6'Sr1\\\ S7'S'r2\ \ S8'Sr3\\ S9'S:r4\ \ S;'Sr5\\ \ S<'Sr6\\ \ S='Sr7\\ \ S>'Sr8\\ \ S?'Sr9\\ \ S@'Sr:\\ \ SA'Sr;\\ \ SB'SCr<\ \ SD'S r=\\ SE'Sr>\\ SF'S r?\\ SG'SHr@g)Ir+:zConfiguration for FlexVit model. This dataclass contains the bulk of model configuration parameters, with core parameters (img_size, in_chans, num_classes, etc.) remaining as direct constructor arguments for API compatibility.  patch_size embed_dim depth num_headsg@ mlp_ratioFscale_mlp_normTqkv_biasqk_norm proj_biasattn_drop_ratescale_attn_inner_normN init_values drop_rate pos_drop_ratepatch_drop_rateproj_drop_ratedrop_path_rate class_tokenr reg_tokenslearned pos_embed)r/r/pos_embed_grid_sizebicubicpos_embed_interp_modepos_embed_ar_preservingpos_embed_use_grid_sample rope_type@rope_temperaturerope_ref_feat_shaperope_grid_offsetijrope_grid_indexingdynamic_img_padpre_norm final_normfc_normmap global_poolpool_include_prefixattn_pool_num_headsattn_pool_mlp_ratio weight_initfix_initlinearembed_proj_typeinput_norm_layerembed_norm_layer norm_layer act_layerblock_fn mlp_layer attn_layerstandard attn_type swiglu_mlp qkv_fusedenable_patch_interpolator)A__name__ __module__ __qualname____firstlineno____doc__r0rintr __annotations__r2r4r5r6floatr7boolr8r9r:r<r=r>r r?r@rArBrCrDrErGstrrHrJrKrLrNrPrQrRrTrUrVrWrXrZr[r\r]r^r_rarbrcrdrerfrgrhrjrkrlrm__static_attributes__rnT/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/models/naflexvit.pyr+r+:s/1Jc5c?*+0IsE3OIsIu ND HdGTItNE"'4'$(K%'IuM5 OU NENEKJIs5=%S/2=!*3*$)T)&+t+Is%e%59%S/29 e """OT!HdJ"GXd^"K %%)-#-+/%/KHd$OS#&*hsm*&*hsm*!%J $#Ix}#"Hhsm"#Ix}# $J $ IsJIt',t+rzcfgreturnc [URR55nUR5VVs0sHup4X2;dM X4_M nnnU(a [ U40UD6nU$s snnf)zIOverlay kwargs onto config, replacing config values with provided kwargs.)set__dataclass_fields__keysitemsr)r|kwargs config_fieldskv config_kwargss r{_overlay_kwargsrs_005578M&,llnKnda8JTQTnMKc+]+ J Ls A$A$Txr0padcJURup4pVUupxU(a?XW-S:wdXh-S:wa/XuU-- U-n XU-- U-n [R"USU SU 45nXW-Xh-pURX4XX5R SSSSSS5R X;U -Xx-U-5n XU 44$)aHPatchify a batch of images. Args: x: Input tensor of shape [B, C, H, W]. patch_size: Patch dimensions (patch_h, patch_w). pad: Whether to pad images to be divisible by patch size. Returns: Tuple of (patches, grid_size) where patches has shape [B, N, P*P*C] and grid_size is (num_patches_h, num_patches_w). rr)shapeFrviewpermutereshape)rr0rBCHWphpwpad_hpad_wnhnwpatchess r{batch_patchifyrs JA! FB ! qv{"f""f" EE!a5) * WagffQ22*221aAq!DLLQUWPWY[Y`cdYdeG H rz_coordc USS2SS2S4RSS9S-nUSS2SS2S4RSS9S-n[X5VVs/sH7up4[UR55[UR554PM9 snn$s snnf)Nrrdim)amaxziprtitem)rmax_ymax_xhws r{calculate_naflex_grid_sizesrs 1a7O Q '! +E 1a7O Q '! +E7:57H I7HtqS]CM *7H II Is>B c \rSrSrSrS\\\\4\\4S\\\\4S\S\S\ RS\ R4 S jr S r \S 5rS rg )NaFlexRopeIteratorzXIterator for generating batched ROPE embeddings for mixed mode with multiple grid sizes.size_to_indices unique_sizes batch_sizeseq_lendevicedtypecFXlX lX0lX@lXPlXplX`lURUlURUlSUR-UR-Ul SUl 0Ul UH nURUS9n XRU'M" g)Nrrr)roperrrrrrr4r5rhead_dim _depth_idx_embeddings_per_size get_embed) self rope_modulerrrrrr grid_size rope_embeds r{__init__NaFlexRopeIterator.__init__s .($    && $..KOO+{/D/DD %'!%I$..Y.?J3= % %i 0&rzcSUlU$)Nr)rrs r{__iter__NaFlexRopeIterator.__iter__s rzc URUR:a[e[R"UR UR URURURURS9nURHYnUup4X4-nURUnURUURnUHnUSS2SU2SS24XSS2SU2SS24'M M[ U=RS- slU$)Nrrr)rr4 StopIterationtorchzerosrr5rrrrrrr) r batch_embedrrr actual_len batch_indicesembedbis r{__next__NaFlexRopeIterator.__next__s ??djj ( kk OOT^^T\\4==**T[[ **IDAJ 00;M--i8IE$5:1kzk1;L5M ;J;12$+ 1rz) rrrr4rrrr5rrrrN)rorprqrrrsrr rtr rrrrrrrryrnrzr{rrsb>eCHotCy89>5c?+ >  >  > >{{>:rzrc URS;=(d" URS;=(d URnU(aURnUS:XaURS;aSnUR(aSOSUR-n[ [ UURURURURUS9$UR=(d [n0nUR(dUR(aURUS'URUS 'UR(aURUS 'U(a [ U40UD6nU$) zGet appropriate block function based on configuration. Returns a partially applied block constructor with EVA-specific or conflicting parameters pre-configured if needed. )evar)rMnonerirrr)rjrk scale_mlpscale_attn_innerrlnum_prefix_tokensr7scale_attn_normrh) rjrNrkrDrErr(r7r=rlrfr)rh)r|use_eva_featuresrjrrf block_kwargss r{ get_block_fnrs (  \)   MM  "s}}L'HI"%//QqCNNJ ~~(( 66mm/  <<(5   !:!:-0-?-?L) *.1.G.GL* + >>),L & x8<8Hrzc)^\rSrSrSrS+S\\\\\44S\S\S\\ S\ S \ S \S \ S \\\\\\44S \ S\\\\4S\ S\ S\ S\\ \ RS\\ \\ \ R4S\\ \ RS\S\ SS4(U4SjjjrS,SjrS\\ \44SjrS-S\ S\\\\\444SjjrS\\\4S\\\44Sjr\S\R0S \R0SS4S!j5r\S\R0S \R0SS4S"j5rS\R0S#\\SS4S$jr\S\R0S \R0SS4S%j5r\S\R0S \R0SS4S&j5rS\R0S#\\SS4S'jrS.S\R0S \\R0S(\\R0S\\R0\\\\444S)jjr S*r!U=r"$)/ NaFlexEmbedsi1aqNaFlex Embedding module for Vision Transformers. This module encapsulates the complete embedding process for Vision Transformers, supporting both standard and NaFlex (NaViT + FlexiViT) functionality: 1. Patch embedding (via Conv2d or Linear) 2. Class and register token preparation 3. Position embedding addition with interpolation support 4. Pre-normalization (if requested) 5. Dropout application NaFlex capabilities include: - Variable aspect ratio and resolution via patch coordinates - Patch type indicators for handling padding tokens in attention - Flexible position embedding interpolation for arbitrary grid sizes - Support for factorized position embeddings The patch embedding can be one of two types: - Conv2d-based (default): For standard image inputs [B, C, H, W] - Linear-based: For pre-patchified inputs [B, N, P*P*C] Args: patch_size: Size of patches for patch embedding in_chans: Number of input image channels embed_dim: Dimensionality of patch embedding proj_type: Type of embedding projection layer ('conv' or 'linear') input_norm_layer: Normalization layer applied to input (linear mode only) proj_norm_layer: Normalization layer applied after projection pos_embed: Type of position embedding ('learned', 'factorized', 'none') pos_drop_rate: Dropout rate for position embeddings class_token: Whether to include a class token reg_tokens: Number of register tokens to include bias: Whether to use bias in projection layers dynamic_img_pad: Whether to enable dynamic padding for variable resolution pos_embed_grid_size: Grid size for position embedding initialization pos_embed_interp_mode: Interpolation mode for position embedding resizing pos_embed_ar_preserving: Whether to preserve aspect ratio during position embedding interpolation default_img_size: Default image size for position embedding grid calculation Nr0in_chansr2 proj_typer:rDrErUdefault_img_sizerGrHrJrKrLrbproj_norm_layerrdr@rmr}c&>UUS.n[TU]5 X`lXplXlXlXl[U5UlX l X0l Xl UUl U(aSOSUl U=RU- sl U(a-[R"[ R""SSU40UD65OSUlU(a,[R"[ R""SXs40UD65OSUlSUlSUlU bXlOZU bW[U 5Ul[-[/UR(UR5VVs/sH unnUU-PM snn5UlUS:XaURSURS-U-nUSLa UcS5eUSLaUO U=(d SnU(aU"U5OSUl[R2"UU4SU0UD6UlS UlSUlO?U(aeSUl[R:"UU4UUUS .UD6UlSUlS UlUR(a[=URUUU SS 9UlOSUlUSLa UcS 5eUSLaUO U=(d SnU(aU"U5O[R@"5Ul!U S ;aUR*c [ES5eSUl#SUl$SUl%U (aU S:XaSUl&OU S:XaUR*ceUR*unnSUl&[R"[ R""SUU40UD65Ul$[R"[ R""SUU40UD65Ul%OXUR*ceUR*unn[R"[ R""SUUU40UD65Ul#SUl&[RN"US9Ul(URS5 gs snnf)aInitialize NaFlexEmbeds module. Args: patch_size: Size of patches for patch embedding. in_chans: Number of input image channels. embed_dim: Dimensionality of patch embedding. proj_type: Type of embedding projection layer ('conv' or 'linear'). proj_bias: Whether to use bias in projection layers. class_token: Whether to include a class token. reg_tokens: Number of register tokens to include. dynamic_img_pad: Whether to enable dynamic padding for variable resolution. default_img_size: Default image size for position embedding grid calculation. pos_embed: Type of position embedding ('learned', 'factorized', 'none'). pos_embed_grid_size: Grid size for position embedding initialization. pos_embed_interp_mode: Interpolation mode for position embedding resizing. pos_embed_ar_preserving: Whether to preserve aspect ratio during interpolation. input_norm_layer: Normalization layer applied to input (linear mode only). proj_norm_layer: Normalization layer applied after projection. norm_layer: Default normalization layer. pos_drop_rate: Dropout rate for position embeddings. enable_patch_interpolator: Enable dynamic patch size support. rrrrNr`Tz5`norm_layer` must be given when input_norm_layer=TruebiasF) kernel_sizestrider)base_patch_sizerr2 interpolation antialiasz4`norm_layer` must be given when proj_norm_layer=True) factorizedrFzgCannot initialize position embeddings without grid_size.Please provide img_size or pos_embed_grid_size.rrrF)p)*superrhas_class_tokennum_reg_tokensrJrKrLrr0rr2rUrmrnn Parameterrempty cls_token reg_tokenrrHtupler norm_inputLinearprojflatten is_linearConv2drpatch_interpolatorIdentitynorm ValueErrorrG pos_embed_y pos_embed_xpos_embed_typeDropoutpos_dropreset_parameters)rr0rr2rr:rDrErUrrGrHrJrKrLrbrrdr@rmrrddsr patch_dimrr __class__s r{rNaFlexEmbeds.__init__[s\ / *(%:"'>$)B&#J/ ".)B&'2q *,NYekk!Q &HR&HI^bV`ekk!Z&Qb&QRfj<@>B  *': $  )$-.>$?D !',TEZEZ\`\k\kAl-mAlAa1fAl-m'nD $  *T__Q-??(JI(D0Z5G HG HH-=-EzL\Ld`d =M.y9SWDO )YMYM"MDI DL!DN( ''"DO '!   DI DL"DN  ) )&< $!#3 'D #'+D #$t+ 0B C B CC(74(?*oF]Y]2AOI.r{{}  1 1d6N6N6VBC C263737I/"(D  , &++7 77++DAq".D !||EKK1i,N2,NOD !||EKK1i,N2,NOD ++7 77++DAq\\%++aAy*OB*OPDN"+D  ]3  S.ns P cURb([RRURSS9 URb([RRURSS9 UR b([RRUR SS9 UR b([RRUR SS9 URb)[RRURSS9 gg)Ngư>)stdg{Gz?)rrinitnormal_rrGrrrs r{rNaFlexEmbeds.reset_parameterss >> % GGOODNNO 5 >> % GGOODNNO 5 >> % GGOODNNO 4    ' GGOOD,,#O 6    ' GGOOD,,#O 6 (rzc>[URURS9$)zGet feature information for feature extraction. Args: location: Feature extraction location identifier Returns: Dictionary containing feature channel count and reduction factor )num_chs reduction)dictr2r0)rlocations r{ feature_infoNaFlexEmbeds.feature_infosDNNdooFFrz as_scalarcRU(a[UR5$UR$)zGet the feature reduction ratio (stride) of the patch embedding. Args: as_scalar: Whether to return the maximum dimension as a scalar Returns: Feature reduction ratio as scalar or tuple )maxr0)rrs r{ feat_ratioNaFlexEmbeds.feat_ratio s  t' '?? "rzimg_sizecUR(aR[R"USURS- 5[R"USURS- 54$USURS-USURS-4$)zCalculate grid (feature) size for given image size. Takes into account dynamic padding when enabled. Args: img_size: Input image size as (height, width) Returns: Grid size as (grid_height, grid_width) rr)rUmathceilr0)rrs r{dynamic_feat_sizeNaFlexEmbeds.dynamic_feat_sizes   99Xa[4??1+==> (ST+X\XgXghiXjJj@kk kA;$//!"44hqkT__UVEW6WW Wrzr patch_coordc |^^^ ^ ^ [U5nTRRSSum m TRRSSSS5R 5m U U U UU4Sjn0n[ U5H&upgUR U/5RU5 M( UR5HupxU"U5n [TRSU RS5n TSS2SU 24RS[R"UTRS9U SS2SU 24R[U5SS55 M g) a/Apply learned position embeddings to NaFlex batch in-place. Interpolates learned 2D position embeddings for each sample in the batch based on their individual grid sizes. Args: x: Input tensor to add position embeddings to [B, N, C] patch_coord: Patch coordinates [B, N, 2] with (y, x) values rrrrc>UST:Xa*UST:Xa!TRRSTT-S5nO}TR(a[[ U55OUn[ R "TUTRSSS9SS2SS2SUS2SUS24RS5RSS5nURTRS 9$) z Return a flattened positional-embedding grid at an arbitrary spatial resolution. Converts the learned 2-D table stored in NCHW format (pos_embed_nchw) into a (1, H*W, C) sequence that matches the requested size. rrFTsizemode align_cornersrNrr) rGrrKrrr interpolaterJr transposetor)r'pos_embed_flat _interp_sizeorig_horig_wpos_embed_nchwrrs r{ _interp2d?NaFlexEmbeds._apply_learned_naflex_pos_embed.._interp2d;sQ6!Q6(9!%!7!76F?B!O7;7S7SyT3Y] !""%33"'" " Qa(47(* ", -4GAJyyA "$$177$3 3rzNrr%)rrGrrrv enumerate setdefaultappendrmin index_add_r as_tensorrexpandlen)rrr"naflex_grid_sizesr3rrrrr.rr0r1r2s`` @@@r{_apply_learned_naflex_pos_embed,NaFlexEmbeds._apply_learned_naflex_pos_embed's  8 D--a2//1a;AAC 4 4*=?01EB  & &q" - 4 4R 82!0 5 5 7 A'q\N!''!*n&:&:1&=>G a'kN % % ahh?q(7({+223}3Er2N  !8rzc|URnURupEnURSS9RS-nUR(a(UR SS9nUR 5n U =pX- =pO'UR SS9upXSS2S4- n XSS2S4- n [ R"USSU[ RS9nXSS2SS4'XSS2SS4'U S- USS2SS4'U S- USS2SS4'[R"XXjU 4SS 9n[R"URRSSSS5RUS S S 5R5UUR SS S 9R#UR$S 9n[ R&"XC[ R(S9R+S5nUUUSS2USUS4- ng)afApply learned position embeddings to NaFlex batch using grid_sample. Uses F.grid_sample for efficient interpolation of learned 2D position embeddings based on patch coordinates. Based on proposal by https://github.com/stas-sl Args: x: Input tensor to add position embeddings to [B, N, C] patch_coord: Patch coordinates [B, N, 2] with (y, x) values rrrNrrrFr)r%borderr(r) padding_moder*.r.r)rrrvaluesrKrrrfloat32r affine_grid grid_samplerGrr<rvrJr-rarangelong unsqueeze)rrr"rrNrshapesL_iL_global grid_size_y grid_size_xscale_xscale_ythetagridrGrs r{+_apply_learned_naflex_pos_embed_grid_sample8NaFlexEmbeds._apply_learned_naflex_pos_embed_grid_sample`s''aQ'..2  ' '++!+$CxxzH(0 0K ( .Gg'-{{q{'9 $K!1a4L0G!1a4L0G Aq!F%--H aAg aAg 1aAg 1aAg}}U $DTYZMM NN " "1aA . 5 5aR D J J L ++!  "177"  \\!%** = G G J Yr1k&1;v3FF GGrzrc6URRSSup4USU:Xa)USU:Xa URRSX4-S5nOUR(a[ U5nXf4nOUn[ R "URRSSSS5R5UURSSS9S S 2S S 2S US2S US24RS5RSS5nURURS 9nURU5 g ) aApply learned position embeddings to standard 2D batch in-place. Interpolates learned 2D position embeddings to match the specified grid size. Args: x: Input tensor to add position embeddings to [B, H*W, C] grid_size: Target grid size as [height, width] rrrr%rFTr&Nr*)rGrrrKrrr+rrvrJrr,r-radd_)rrrr0r1r.Lr/s r{_apply_learned_pos_embed%NaFlexEmbeds._apply_learned_pos_embeds--a2 Q<6 !ilf&<!^^33AvKN++ N t ( ]]&&q!Q288:!//#  MYq\M=IaL=0 2 3:'!*YYq!_ (***9 ~rzc ^[U5n[U5TRS5:XdeURRSUR RSpT0n[ U5H&upxURU/5RU5 M( S[RS[S[S[R4U4Sjjn UR5GHupUupUR(a [X5=pOXpU "URX5SS2SU 24nU "UR X5SS2SU 24nURS 5URS5-nUR!SS 5n[#TRSURS5nTSS2SU24R%S[R&"U TR(S 9USS2SU24R+[U 5S S 55 GM g) a3Apply factorized position embeddings to NaFlex batch in-place. Uses separate Y and X position embedding tables that are interpolated and combined for each sample's grid size. Args: x: Input tensor to add position embeddings to [B, N, C] patch_coord: Patch coordinates [B, N, 2] with (y, x) values rrtable new_length orig_lengthr}c>X:XaURTRS9$[R"UR SSS5R 5USSS9R SSS5RTRS9$)zt Resample a 1-D positional-embedding table to specified length and return it in (1, L, C) layout, dtype matching x. r*rrrr`Fr'r(r)r-rrr+rrvrarbrcrs r{ _interp1dBNaFlexEmbeds._apply_factorized_naflex_pos_embed.._interp1dsr (xxaggx..== aA&,,.#  gaArrr0  1rzNrr5r%)rr=r'rrrr6r7r8rTensorrtrrKrrNrr9r:r;rr<)rrr"r>r0r1rrrrhrtarget_htarget_wlen_ylen_xpe_ype_xposrs ` r{"_apply_factorized_naflex_pos_embed/NaFlexEmbeds._apply_factorized_naflex_pos_embeds 8 D$%222))//2D4D4D4J4J14M=?01EB  & &q" - 4 4R 82 1U\\ 1s 1 1QVQ]Q] 1!0 5 5 7 A!" H++ #H 77'uT--u=a(lKDT--u=a(lKD..#dnnQ&77C++a#C!''!*ciil3G a'kN % % ahh?AxxK ''M(:BC !8rzc^^^TRnTRumnmURSS9S-nUR(a(URSS9nUR5nU=pXv- =pO)URS5upXSS2S4- n XSS2S4- n S[R S[R S[R S[R 4UUU4S jjn U "UR XS 9n U "URXS 9n[R"TU[RS 9RS5nTXSS2SUS 4XSS2SUS 4-- mg)auApply factorized position embeddings to NaFlex batch using grid_sample. Uses F.grid_sample for efficient interpolation of separate Y and X position embedding tables based on patch coordinates. Based on proposal by https://github.com/stas-sl Args: x: Input tensor to add position embeddings to [B, N, C] patch_coord: Patch coordinates [B, N, 2] with (y, x) values rrrNrascale out_lengthr}c>URSSS5RS5RTSSS5R5n[R "TSSTR S9nXSS2SS4'US- USS2SS4'SUSS2SS4'[R"UTTSU4SS9n[R"X5S SS S 9nURTR5$) Nrrrr%rr5FrBbilinearrCrD) rrNr<rvrrrrrJrKr-r) rarurvperWrXrrrs r{rhNNaFlexEmbeds._apply_factorized_naflex_pos_embed_grid_sample.._interp1dsq!Q'11!4;;Ar2rJPPRBKK1a9E"!Q'N"QYE!Q'NE!Q'N==Aq*(=USDrj\deB55> !rz)rurvrrGrF) rrrrKrrjrrrLrMrN)rrr"r_rPrQrRrSrTrUrVrhrprorrrs ` @@r{._apply_factorized_naflex_pos_embed_grid_sample;NaFlexEmbeds._apply_factorized_naflex_pos_embed_grid_samplesP''1a!!a!(1,  ' '++!+$CxxzH(0 0K ( .Gg(.{{1~ $K!1a4L0G!1a4L0G "U\\ "%,, "ELL "]b]i]i " "))Q))Q \\!F%** = G G J TaK// 04Aq+fBU8U3V VVrzc^URRSURRSpCUupVUR(a [ XV5=pxOXVpS[ R S[S[S[ R 4U4Sjjn U "URXs5SS2SU24n U "URX5SS2SU24n U RS5U RS5-n U RSS5n TRU 5 g) a-Apply factorized position embeddings to standard 2D batch in-place. Uses separate Y and X position embedding tables that are interpolated and combined for the specified grid size. Args: x: Input tensor to add position embeddings to [B, H*W, C] grid_size: Target grid size as [height, width] rrarbrcr}c>X:XaURTRS9$[R"UR SSS5R 5USSS9R SSS5RTRS9$)Nr*rrrr`Frerfrgs r{rh;NaFlexEmbeds._apply_factorized_pos_embed.._interp1d6sp(xxaggx..== aA&,,.#  gaArrr0  1rzNr) rrrrKrrrjrtrNrr\)rrrr0r1rkrlrmrnrhrorprGr.s ` r{_apply_factorized_pos_embed(NaFlexEmbeds._apply_factorized_pos_embed s))//2D4D4D4J4J14M&  ' '3 3EE#5 1U\\ 1s 1 1QVQ]Q] 1))59!YhY,G))59!YhY,GNN1%q(99 "**1a0 ~rz patch_validc SnURSnUR(Ga:Uc:[URS:HS5 [ XR UR S9upO/[URS:H=(d URS:HS5 UR(a{URS:Xak[URSLS 5 URUURRURR[URS S5S S 9nGOLURS 5nURbURU5nURU5nGO [URS:HS 5 UR (aURSSupgUR SX`R S-- UR S-nUR SXpR S-- UR S-n [R "USU SU45nURU5nURSSnUR(a!URS 5R#SS 5nUR%U5nUR&S:XaDUbUR)XS9 OUR*(aUR-XS9 OcUR/XS9 OSUR&S:XaCUbUR1XS9 O0UR*(aUR3XS9 OUR5XS9 /n UR6b,U R9UR6R;USS55 UR<b,U R9UR<R;USS55 U (a[>R@"X/-SS9nURCU5nX4$)awForward pass for patch embedding with position encoding. Args: x: Input tensor. Supported formats: - [B, C, H, W] for conv mode - [B, N, P*P*C] for pre-patchified linear mode (normal) - [B, N, Ph, Pw, C] for pre-patchified linear mode (variable patch size) patch_coord: Optional patch coordinates [B, N, 2] for NaFlex mode. patch_valid: Optional validity mask for patches [B, N] for NaFlex mode. Returns: Tuple of (embedded_tensor, grid_size) where: - embedded_tensor: [B, num_prefix_tokens + N, embed_dim] - grid_size: (H, W) tuple for standard mode, None for NaFlex mode Nrrz-Expecting 2D image input with input ndim == 4)rrrz/Expecting patchified input with ndim == 3 or 5.z,input norm not supported with patch resizingrT)r0rzConvolutional input must be 4DrrF)r)r"rr%r)"rrrndimrr0rUrmrrrweightrrrrrr,rrr^rLrYr?rr|rrrr8r<rrcatr) rrr"rrrrrrrto_cats r{forwardNaFlexEmbeds.forwardJs2*04 GGAJ >>>"! %TU-adFZFZ[ 9! 2qvv{4ef--!&&A+4/1_`++II$$IINN$QWWQq\2" ,IIaL??.*AIIaL AFFaK!A B##wwrs|+a//!2D.DDXYHZZ+a//!2D.DDXYHZZEE!a512 ! A I||IIaL**1a0 IIaL   ) +$--a-E11DDQD`888T  L 0$000H11GGGc;;A;W >> % MM$..//2r: ; >> % MM$..//2r: ;  &3,A.A MM! |rz)rrrUr2rmrrrrrrrrrr0rrGrKrHrJrrLrrrr)r/rr1NTTrFNrF)rrIFFNNNr;FNNr}NTNN)#rorprqrrrsrrtr r rxrwr rModulervrrrrrrr rrrjr?rYr r^rrr|rrry __classcell__r s@r{rr1s&T79 '+" $$)FJ&=E)2,1.3:>FJ48#%.3-U c5c?23U U  U  } U  U U U "U 'uS%S/-A'BCU U "*%S/!:U $'U &*U (,U 'tBII7!U "#4$ryy/)B#BC#U $!bii1%U &!'U ((,)U . /U U n 7 GS#X G #D #E#uS#X:N4O #X%S/XeCHoX 6||66  66p,H||,H,H  ,H,H\!||!Cy!  !F;||;;  ;;z-W||-W-W  -W-W^(||(Cy(  (Z3726 g||g"%,,/g"%,,/ g u||XeCHo66 7 ggrzrrr symmetricq_lenrc UcgUR5nURupVUnUS:a9URXQ4[RS9n[R"X/SS9nXq- nU(a5UR S5UR S5-n U R S5n O)U=(d UnUSS2SSSS24R USX75n [R"XS9n U RU )[R"U5R5 U $)aCreates an attention mask from patch validity information. Supports two modes controlled by `symmetric`: 1. `symmetric=True` (default): Creates a symmetric mask of shape [B, 1, seq_len, seq_len]. An attention pair (i, j) is allowed only if both token i and token j are valid. Suitable for standard self-attention. 2. `symmetric=False`: Creates a potentially non-square mask of shape [B, 1, q_len, kv_len]. An attention pair (q, k) is allowed only if the key/value token k is valid. Query token validity is not checked in the mask itself. Useful for cross-attention or specific self-attention implementations `q_len` can be specified. Used for NaFlex mode to handle variable token counts and padding tokens. Args: patch_valid: Tensor of shape [B, N] with True for valid patches, False for padding. num_prefix_tokens: Number of prefix tokens (class token, register tokens) to prepend, which are always considered valid. symmetric: If True, create a symmetric mask. If False, create an expanded mask based only on key/value validity. q_len: Query sequence length override. Only used when `symmetric` is False. Defaults to the key/value sequence length (`kv_len`) if None. dtype: Dtype of the output attention mask (e.g., torch.float32). Returns: Attention mask tensor. Additive mask (-inf for masked, 0 for unmasked). Shape is [B, 1, seq_len, seq_len] if symmetric=True, or [B, 1, q_len, kv_len] if symmetric=False. Nrr*rrr%) rwrnew_onesrrrNr< zeros_like masked_fill_finfor9) rrrrrrrOkv_len prefix_valid mask_bool mask_floats r{create_attention_maskrsJ""$K   DA F1"++Q,B%**+U ii ;C #))"- 0E0Ea0HH ''* 4q 0188AuM !!)9JYJ E(:(>(>? rz pool_typereduce_include_prefixcpUbUS;a[UUUUS9nU$US:aHU(a6URURSU5n[R"XQ/SS9nO USS2US24nUR UR 5nUS:XaCXRS5-RSS9nURSS S 9RSS 9nXx- n U $US :XaXRS5-RSS9nURSS S 9RSS 9nXx- n UR5n [R"U R 5RX)'U RSS9n S X--$US:XaLUR5n [R"U R 5RX)'U RSS9$e)a$Global pooling with NaFlex support for masked tokens. Applies global pooling while respecting patch validity masks to exclude padding tokens from pooling operations. Args: x: Input tensor with shape [B, N, C] patch_valid: Optional validity mask for patches [B, N-num_prefix_tokens] pool_type: Type of pooling ('token', 'avg', 'avgmax', 'max') num_prefix_tokens: Number of prefix tokens (class/register) reduce_include_prefix: Whether to include prefix tokens in pooling reduction Returns: Pooled tensor with shape [B, C] N)avgavgmaxrrrrrrrrr%T)rkeepdim)r9rg?r)r*rrrrr-rrNsumclampclonerr9r) rrrrrrpatch_valid_float masked_sums valid_countspooled masked_avgmasked_x masked_maxs r{global_pool_naflexrs.i/GG  /"7   1 '// [TU]5 XVS.nU=(d [5nU(a [U40UD6nURS;deUR (dURS:wdeUR S;de[UR5=(d [n [UR5n [UR5=(d [Rn [U5n UR =(d ["n X0lX lURUlUR(=Ul=UlUlUR (aSOSUlU=R.UR0- slUR0UlUR UlUR6UlSUl[;S+0SUR<_S U_S UR(_S UR>_S UR@(+_S UR _SUR0_SU_SURB_SUR _SURD_SURF_SURH_SURJ_SU _SURL_S[OUSS5_UD6Ul(UR@(aU "UR(40UD6O[RR"5Ul*SUl+SUl,URZ(aURZS:waSSK.J/nJ0n URZS:XaNU"UR(4URbURdURfSURhS.UD6Ul+SUl,OURZS:Xa\U"UR(URd-4URfSSURjURlURhS .UD6Ul+SUl,O[oS!URZ35eURpS:a$[sURpUR.S"9Ul:OSUl:[wURxURb5n[Rz"[}URb5Vs/sHwnU "S+UR(URdUR~URURURURURURUUU U U US#.UD6PMy sn6UlFURPRSS$9n[}URb5Vs/sHn[S%U3UR(US&9PM snUlIUR(a$UR(dU "UR(40UD6O[RR"5UlLURS':Xa[[UR(4UR=(d URdUR=(d UR~U U S(.UD6UlPOSUlPURnUcURS):HnUR(aU(aU "UR(40UD6O[RR"5UlK[R"UR5UlSUS:a"[R"UR(U40UD6O[RR"5UlUURUlWURUlXURURSS*9 gs snfs snf),aXInitialize NaFlexVit model. Args: cfg: Model configuration. If None, uses default NaFlexVitCfg. in_chans: Number of input image channels. num_classes: Number of classification classes. img_size: Input image size (for backwards compatibility with classic vit). **kwargs: Additional config parameters to override cfg values. rrMrrrtokenrYr)rMrrFrrrFr0rr2rr:rDrErrUrGrHrJrKrLrr@rmNr)RotaryEmbeddingCatRotaryEmbeddingMixedmixed)r4r5 temperature feat_shape grid_indexingTaxial)r in_pixelsrref_feat_shape grid_offsetrzUnknown rope_type: )r)rr5r6r8r9r:r> proj_drop attn_drop drop_pathrdrergr4)rzblocks.)modulerrrY)r5r6rdrer needs_resetrn)Zrrr+rrZrDrGrrdrrcrrerGELUrrgrrrr2 num_featureshead_hidden_sizerrErrr[grad_checkpointingrr0rarVrUrHrJrKrLr@getattrembedsrnorm_prer rope_is_mixedrNtimm.layers.pos_embed_sincosrrr4r5rPrTrQrRrrAr patch_droprrC Sequentialranger6r8r9r:r>rBr<blocksrrrrWrXrrr\r] attn_poolrr? head_droprheadr^weight_init_moder_ init_weights)rr|rrrrrrrrdrcrerfrgrrdpripatch_reductionrXr s r{rNaFlexVit.__init__Os!&  /#\^ !#00C"NNNN#//W"<<<}} EEEE$CNN3@y )#*>*>?!#--0;BGG $MM(S ' ??EH]]RRD1DN&)oo1 #..0!nn"#&#:#: "'# ~~  mm ))  ,,&    ~~ &  // mm !$ 7 7 #&";"; %($?$? '*&C&C - ++! "'.c3NPU&V%  (\"\c.S[RS[SS4Sjn[UR5Hup#[ US5(a*U"UR RRUS-5 [ US5(a*U"URRRUS-5 [ US 5(a U"URRUS-5 [ US 5(dMU"URRUS-5 M g) z8Apply initialization weight fix with layer-wise scaling.param _layer_idr}Nc[R"5 UR[R"SU-55 SSS5 g!,(df  g=f)Ng@)rno_graddiv_rsqrt)rrs r{rescale*NaFlexVit.fix_init_weight..rescales. 499S9_56!s )A Aattnrmlp attn_out_proj mlp_out_proj) rrjrtr6rhasattrrrrrfc2rr)rrlayer_idlayers r{fix_init_weightNaFlexVit.fix_init_weights 75<< 7C 7D 7 )5OHuf%% ..1 =ue$$ ,,hl;uo..++22HqLAun--**118a<@ 6rzr(rcU=(d URnUS;deSU;a![R"UR5*OSn[ [ XUS9U5 UR (aUR5 gg)a>Initialize model weights according to specified scheme. Args: mode: Initialization mode ('jax', 'jax_nlhb', 'moco', or '') needs_reset: If True, call reset_parameters() on modules (default for after to_empty()). If False, skip reset_parameters() (for __init__ where modules already self-initialized). )jaxjax_nlhbmocorMnlhbr;rN)rrlogrr%get_init_weights_vitr_r)rr(r head_biass r{rNaFlexVit.init_weightssm,t,,666639T>TXXd..//r (kRTXY ==  " rzcheckpoint_pathprefixc4^SSKJm SU4SjjnU"XU5 g)Nr) _load_weightsc>>[R"USS9n[U[5(a SU;aUSn[ UR 55HnUR S5(aURU5USU-'M2UR S5(aURU5USU-'MaUR S5(aURU5USU-'MUR S5(dMURU5USUS S -'M T"XU5$) z8Adapter function to handle the different model structurecpu) map_location state_dictrembeds.rrG patch_embedr3N)rload isinstancerlistr startswithpop)modelrrrr_orig_load_weightss r{_load_weights_adapter8NaFlexVit.load_pretrained.._load_weights_adapter$sO%HJ*d++ 0J' 5 *//+,<< ,,0:q0AJy1}-\\+..0:q0AJy1}-\\+..0:q0AJy1}-\\-005?^^A5FJy1RS612-&e@ @rzrM)vision_transformerr)rrrr rs @r{load_pretrainedNaFlexVit.load_pretraineds L A& dVembeds.cls_tokenembeds.pos_embedembeds.reg_tokenno_weight_decay)rrupdater)r skip_lists r{rNaFlexVit.no_weight_decay9sCQ 99,=>>   TYY668 9rzcoarsec[SSS/S9$)zGet parameter group matcher for optimizer parameter grouping. Args: coarse: Whether to use coarse-grained grouping Returns: Dictionary mapping group names to regex patterns z^embeds)z^blocks\.(\d+)N)z^norm)i)stemr)r)rrs r{ group_matcherNaFlexVit.group_matcherEs-/CD  rzenablecXl[URS5(aL[URRS5(a&URRR U5 ggg)z|Enable or disable gradient checkpointing for memory efficiency. Args: enable: Whether to enable gradient checkpointing rset_grad_checkpointingN)rrrrr)rrs r{r NaFlexVit.set_grad_checkpointingTsS#) 4;; . .74;;;R;RTl3m3m KK # # : :6 B4n .rzcUR$)zMGet the classification head module. Returns: Classification head module )rrs r{get_classifierNaFlexVit.get_classifier_syyrzrr"c [U5n0n/n[U5H2upgXt;a/XG'URU5 XGRU5 M4 URupn XR- n UR (a/[ URUUUU URUR5$[R"XURRS-URURS9n [URS5(a^URRU5n [X]UR!55H$up~nUunnUU-nUHnUSUXSU24'M M& OGUR#5H3upvURR%US9nUunnUU-nUSUXSU24'M5 U R'S5n U $)aGenerate ROPE position embeddings for NaFlex batch with variable grid sizes. Args: x: Input tensor [B, N, C] patch_coord: Patch coordinates [B, N, 2] with (y, x) values Returns: ROPE embeddings: - Axial mode: Tensor of shape [B, 1, N, dim*2] - Mixed mode: List of tensors, each of shape [B, num_heads, N, dim], one per depth layer - Mixed mode with iterator: Iterator yielding tensors per depth rrget_batch_embedsNrr)rr6r8rrrrrrrrrrrr$rrHrrrN)rrr"r>rrrrrrOrr rope_embeds unique_embedsrrrrrrs r{_generate_rope_naflexNaFlexVit._generate_rope_naflexhs&8 D &'89MB/-/*##I.  & - -b 1 : ''a,,,   %  kk!diimma.?qwwWXW_W_` 4990 1 1 II66|DM36|TcTjTjTl3m/ - 1U 'B38*3EKKZK0(4n"1!6!6!8 !YY00y0A  1U /9+:/F  O, "9 "++A. rzrZc(XlUbBUS;deUS:XaURcS5eUS:waURbSUlX lUS:a'[R"UR U5Ulg[R "5Ulg)zReset the classification head with new number of classes and pooling. Args: num_classes: Number of classes for new classification head global_pool: Optional new global pooling type NrrYz=Cannot currently add attention pooling in reset_classifier().r)rrrZrrr2rr)rrrZs r{reset_classifierNaFlexVit.reset_classifiers'  ""NN NNe#(>]]]u%$..*D!%* >IAoBIIdnnk: SUS^S^S` rzc2USLnURUUUS9upSnURb9UbURX5nO$UbURRUS9nOS5eSnUR(akUR b^UR U5upUbUR SU5nUb3UR(d"[XXS9nU(dURS5nUc[UURURS9nURU5nUUUUUS.$) zQForward pass through patch / abs pos / rope pos embeds and patch dropout N)r"rrz4Expected one of patch_coord or grid_size to be validrpos_embed_has_batch)rr)rrr% attn_mask keep_indices)rrr'rtrainingrgatherrrrNrrrr) rrr"rr/ naflex_moderr%r0s r{_forward_embedsNaFlexVit._forward_embedss9"- {{ ###  99 &"88H &"ii11 1B TTTu04 ==T__8"ooa0OA&)00LA &t/A/A5Q\s ""-"7"7":K  -"&"8"8ggI MM! &&"(   rzindicesreturn_prefix_tokensr stop_early output_fmtintermediates_only output_dictrr/c US;dS5eUS:Hn /n [[UR5U5up[U[5(aUSn USn USnS5eUnUR S S unnUR RUU45unnURUU U U S 9nUSnURS S 5nURS S 5nURSS 5n [RR5(dU(d URnOURS US-nUR=(a# [RR5(+nUR(aUb[[!URU55HununnUR"(a,UR$bUb['UUUURSS5S9nU(a [)UUUU S9nOU"UUU S9nUU;dMjU R+U(aUR-U5OU5 M O~[U5HounnUbU(a [)UUUU S9nO!U"UUU S9nOU(a [)UXS9nOU"XS9nUU;dMFU R+U(aUR-U5OU5 Mq UR.(aKU Vs/sHnUS S 2SUR.24PM nnU Vs/sHnUS S 2UR.S 24PM n nOS nU (aQU Vs/sHDnUR1UR SUUS5R3SSSS5R55PMF n nU(a50nU US'Ub U(aUUS'U(dUR-U5n U US'U$[RR5(dU(aUb[7[!U U55n U(aU $UR-U5nX4$s snfs snfs snf)aForward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence return_prefix_tokens: Return both prefix and spatial intermediate tokens norm: Apply norm layer to all intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features output_dict: Return outputs as a dictionary with 'image_features' and 'image_intermediates' keys patch_coord: Optional patch coordinates [B, N, 2] for NaFlex mode patch_valid: Optional patch type indicators (1=patch, 0=padding) for NaFlex attn_mask: Optional attention mask for masked attention Returns: A tuple with (final_features, intermediates), a list of intermediate features, or a dictionary containing 'image_features' and 'image_intermediates' (and optionally 'image_intermediates_prefix') )NCHWNLCz)Output format must be one of NCHW or NLC.r=r"rrFzWIP, patch mode needs more workrNr"rr/r%r0r/rr3r-rr/r/rr%rrimage_intermediatesimage_intermediates_prefiximage_features)r!r=rrrrrr r4getrjit is_scriptingrrr6rr1rrr$r8rrrr contiguousr)!rrr6r7rr8r9r:r;r"rr/r intermediates take_indices max_indexrheightwidthrrrr%r0rdo_checkpointingrblkry prefix_tokens result_dictx_finals! r{forward_intermediatesNaFlexVit.forward_intermediatessF_,Y.YY,& "6s4;;7G"Q a  M*KM*K lG ;; ;5GGGBCLMFE;;00&%ADAq%% ## &  9 jj5 zz.$7 JJ{D1  99 ! ! # #:[[F[[)a-0F22S599;Q;Q;S7S   +"9(1#dkk;2O(P$$C==T__%@\E]!7"$,2JJ}e,L "J $"3 iPAAJ)DA $!((11E!)Q$$F+3*'&sAK9U yI'&sAC7 $!((11E,"  ! !ERS]Qq!D$:$:"::;]MSDQRMqQq$"8"8"99:MMRM M '&A !''!*aB/771aCNNP&  K1>K- .(-A  rzrcZURbk[UUR(a UROSSSURS9nUR(dUSS2URS24nURXS9nU$Uc UR OUn[ UUUURURS9nU$)NrFr)rrrrrAr)rrr[rrrZr)rrrrr/s r{_poolNaFlexVit._pools >> %-<@{$'> I,eeM;? eeM;? EE+y9  Kt+-U V Kt+-U V,,''# -H!!-H-A%%a(A!!!$Arz)rrr2rrXrr_rZrrrrrrrrrrrrrr[rrr)NrNNNr)rMTr Fr)N) NFFFr=FFNNN)NNNr)FN))rorprqrrrsr r+rtrr rrrxrwrrrFignorer r rrrrrrr!rrjr rr'r*r4rTrWrZr^rryrrs@r{r,r,Bs +/#>Bl>,'l>l> l> uS%S/%9:; l> l>l>\A"####" YY=s=C===2 YY    YY  D  T     YYCTCTCC YY B||BB u||T%,,/4 5 BBHaCahsmaW[a"< c5<< < B8<).$$', %262604S U\\4U\\(9#::;S eCcN34S #' S  S  S S !%S S "%,,/S "%,,/S   -S  tELL!5tELL7I)I#JDQTVYQYNZ [S p372604 >\\>"%,,/>"%,,/ >   - > u||T#u||"344 5 >F(,26 || }"%,,/   B %26 1\\ 1 1"%,,/ 1  1372604 0U\\4U\\(9#::;0"%,,/0"%,,/ 0   - 0  00rzcUSnUSnUSn[[U55HnX4X$nXSS2S4R5S-R5nXSS2S4R5S-R5nUR XgSSS5R SSS SS5nUR SUS-US-5nSS KJn U"US US 3S S9 M g)Nr"rrrrr/rrr) save_imagepatch_z.jpgT) normalize)rr=rrrrtorchvision.utilsrh) rr"rrrpatchrrrhs r{_debug_dump_patchesrm-sM"KM"K lG 3w<  ;>* Aq ! % % '! + 1 1 3 Aq ! % % '! + 1 1 3 aBA.66q!Q1E a2qt,05F1#T*d;!rzr(rrcjSSKJnJnJnJn SU;a [ X1US9$SU;a [ XBS9$[ XRS9$)zFFunction imported from vision_transformer.py to maintain compatibilityr)init_weights_vit_jaxinit_weights_vit_mocoinit_weights_vit_timminit_weights_reset_parametersr)rrrr)r rorprqrrr)r(rrrorprqrrs r{rr<sB }+kZZ 4,FF,FFrzrrc20nUR5GHup4US:XGa[URS5(GaURS:XGaSnSnSU;aUSRSnSU;aUSRSnXV-nURSnX- n [ R "U 5n [ R "U5n X:waU R5(a}U R5(dhU nUSS2SU24n U R5(a US==U - ss'USS2XV24n U R5(a US==U - ss'USS2US24nU n [U 5n X-U:Xa!URSXURS5nO[URRS 5(aiURRRupX-U:Xa!URSXURS5nO[RS US X-S 35 XBS 'GMUS:XaXBS'GMUS:XaXBS'GM+URS5(a:USSnUS:Xa#UR!SSSS5R#S5nSU-nXBU'GM{XBU'GM U$)zZHandle state dict conversion from original ViT to the new version with combined embedding.rGrrrrrNrrz-Position embedding size mismatch: checkpoint=z, model=z?. Using default initialization and will resize in forward pass.rrrz patch_embed.r3z proj.weightr)rrrrrrr is_integernumelrtrrr_loggerwarningrrr)rrout_dictrr num_cls_token num_reg_tokenr num_patchesnum_patches_no_prefixgrid_size_no_prefixr cls_token_emb reg_token_embrrsuffixnew_keys r{checkpoint_filter_fnrMsH  " u||[11affk ! ! *,$.{$;$A$A!$DM*,$.{$;$A$A!$DM$1$A! ggaj (3(G%&*ii0E&F# IIk2 '40;;==iFZFZF\F\#8K$%a=&8$9M$**,,";/=@/$%a)D&D$EM$**,,";/=@/!.//0A 3I N (K7 !Y1771:FAu||77EE$||77AA5K/ ! !Q1771: >A$OO"OP[}\dfgfkdmn`!a ,-' ( + +,' ( + +,' ( \\. ) )rsVF&IIaAq)11!4&(G !W QKC#F Orzurlc 2USSSSS[[SSSS. UE$) Nrd)rrg?rIz embeds.projrz apache-2.0) rr input_size pool_sizecrop_pctrmeanr  first_conv classifierlicenser)rrs r{_cfgrs7#"'%#    rzztimm/) hf_hub_id)rr)z)naflexvit_base_patch16_gap.e300_s576_in1kz-naflexvit_base_patch16_par_gap.e300_s576_in1kz0naflexvit_base_patch16_parfac_gap.e300_s576_in1kz$naflexvit_base_patch16_map.untrainedz,naflexvit_so150m2_patch16_reg1_gap.untrainedz,naflexvit_so150m2_patch16_reg1_map.untrainedz&naflexvit_base_patch16_siglip.v2_webliz(naflexvit_so400m_patch16_siglip.v2_weblivariant pretrainedc URSS5nURS[55n[[5Vs1sHoURiM nn[ U5Vs0sHowU;dM XrRU5_M nnU(a [ U40UD6n[ [X4[U[USS9S.UD6n U $s snfs snf)N out_indicesrr|getter)r feature_cls)pretrained_filter_fnr| feature_cfg) rr+rnamerrr r,rr) rrrrr|fcfg_field_namesr cfg_updatesrs r{_create_naflexvitrs**]A.K **ULN +C'-l';<';!vv';O<-1&\R\/=Q#1jjm#\KRc1[1 7 1 [hG     E L=RsB<" C/Cc 6URSS5 URSS5 URSS5nURSS5nUcUS:XaSnSURS S5UUURS S 5URS S 5S .UEn[X40UD6$)aCreate FlexVit model from classic VisionTransformer configuration. This function handles the parameter mapping and configuration logic needed to create FlexVit models that are compatible with classic VisionTransformer configurations and pretrained weights. Args: variant: Model variant name pretrained: Whether to load pretrained weights **kwargs: Classic VisionTransformer parameters Returns: FlexVit model instance no_embed_classNdynamic_img_sizerZrrXrTrDr7Fr)rHrDrZrXr7r=rrEr)rrrgprX flex_kwargss r{_create_naflexvit_from_classicrs( JJ& JJ!4( M7 +BjjD)G2; $zz-6 **%5u=!',=u!E  K W @K @@rzc URSS5 URSS5nURSS5nURSS5nURSS 5nURS S 5nU(a U(aS OS nOSnURSS5n URSS5n URSS5n URSS5n U cU S:Hn 0SS_SURSS5_SURSURSS55_SU _SU _SU _SU _SURSS5(aSOS_S U_SU_SU_S U_S!URS"S5_S#URS#S$5_S%URS%S5_S&URS&S5_S'URS(S5_S)URS*S50EUEn [X40U D6$)+a~Create NaFlexVit model from EVA configuration. This function handles the parameter mapping and configuration logic needed to create NaFlexVit models that are compatible with EVA configurations and pretrained weights. Args: variant: Model variant name pretrained: Whether to load pretrained weights **kwargs: EVA model parameters Returns: NaFlexVit model instance rNuse_rot_pos_embFrope_mixed_moderPrOrRr;rTrSrrrrZruse_pre_transformer_normuse_post_transformer_normT use_fc_normrHrDrErrrVrWrXrGuse_abs_pos_embrFrNrQrrjrrkrlr7rr=rr)rrrrrrPrRrTrNrrrr naflex_kwargss r{_create_naflexvit_from_evarsH( JJ&jj!2E:Ojj!2E:Ozz"4f=zz"4b9$8$?.GG   M5 )B%zz*DeL & +F M**]D1KEk tvzz-6 vzz"2FJJ|Q4OP r  ,  /  ; &**->"E"EY6 Y , , 0 vzz*:DA VZZ U3 fjju5 VZZ T2!" &**[%8#$ ,>!F%& 'M, W BM BBrzc @[SSSSSSSSS9n[S XS .UD6nU$) zCViT-Base with NaFlex functionality and global average pooling. r/r1r3h㈵>rrT)r0r2r4r5r>rZrErXrr|)naflexvit_base_patch16_gapr+rrrr|rs r{rrBs@  C  ez e^d eE Lrzc B[SSSSSSSSSS9 n[S XS .UD6nU$) z]ViT-Base with NaFlex functionality, aspect preserving pos embed, global average pooling. r/r1r3rTrr) r0r2r4r5r>rKrZrErXr)naflexvit_base_patch16_par_gaprrs r{rrTsC  $ C  i: ibh iE Lrzc D[SSSSSSSSSSS 9 n[S XS .UD6nU$) zjViT-Base with NaFlex functionality, aspect preserving & factorized pos embed, global average pooling. r/r1r3rTrrr) r0r2r4r5r>rKrGrZrErXr)!naflexvit_base_patch16_parfac_gaprrs r{rrgsF  $ C  lj lek lE Lrzc >[SSSSSSSS9n[S XS.UD6nU$) zBViT-Base with NaFlex functionality and MAP attention pooling. r/r1r3rrYr)r0r2r4r5r>rZrEr)naflexvit_base_patch16_maprrs r{rr{s=  C  ez e^d eE Lrzc D[SSSSSSSSS S S 9 n[S XS .UD6nU$)ViT-SO150M2 with NaFlex functionality for variable aspect ratios and resolutions. This model supports: 1. Variable aspect ratios and resolutions via patch coordinates 2. Position embedding interpolation for arbitrary grid sizes 3. Explicit patch coordinates and valid token masking r/@ NN@rFrrT) r0r2r4r5r6r>r8rErZrXr)"naflexvit_so150m2_patch16_reg1_gaprrs r{rrsF  C  mz mfl mE Lrzc B[SSSSSSSSS S 9 n[S XS .UD6nU$) rr/rrrrrFrrY) r0r2r4r5r6r>r8rErZr)"naflexvit_so150m2_patch16_reg1_maprrs r{rrsC  C  mz mfl mE Lrzc <[SSSSSSS9n[SXS.UD6nU$) zGViT-Base with NaFlex functionality and SigLIP-style configuration. r/r1r3 gelu_tanhrY)r0r2r4r5rerZr)naflexvit_base_patch16_sigliprrs r{rrs:   C  h* hag hE Lrzc >[SSSSSSSS9n[S XS.UD6nU$) zUViT-SO400M with NaFlex functionality for variable aspect ratios and resolutions. r/igZӼ @rrY)r0r2r4r5r6rerZr)naflexvit_so400m_patch16_sigliprrs r{rrs=  C  jJ jci jE Lrzr)NrrF)rr;Tr re)arsloggingr dataclassesrrr functoolsrtypingrrr r r r r rrrtorch.nnrtorch.nn.functional functionalr timm.datarr timm.layersrrrrrrrrrrrr_builderr _featuresr! _features_fxr"r# _manipulater$r% _registryr&r'rr(r r)r*__all__ getLoggerrorvr+rrjrtrwrrrrrrrIrrrxrr,rmrvrrr default_cfgsrrrrrrrrrrrrnrzr{rs$ 22OOO E    ++L0<6 ; '   H % T,T, T,n  L  <<#s(O 5<<sCx () @J J==@)l)x)X 299  D"##"]] A\\AAA} A {{ A ell AAH/3 !"&+ E <<Eell+EE E $ E  \\ EEPh  h V <GsGuGQUGaiG"HT#s(^HIH$sTWx.HVcT#s(^"%1526:69=9-1F48F48F/3/151%& 0s9*!(A(A(A (AZ!@C@C@C @CF4i"t)$$Y& 4 i   4i04i. d     9  rz