RЦid SrSSKrSSKrSSKrSSKJr SSKJrJ r J r J r J r SSK r SSKJs Jr SSK Jr SSKJrJr SSKJrJrJrJrJrJrJrJrJrJrJ r J!r! SS K"J#r# SS K$J%r% SS K&J'r' SS K(J)r)J*r* SS K+J,r,J-r-J.r. S/r/\R`"\15r2"SS\Rf5r4"SS\Rf5r5"SS\Rf5r6S\74Sjr8\'S\74Sj5r9"SS\Rf5r:"SS\Rf5r;S1S\RfS\Sr?S r@S2S!jrAS3S"jrB\-"\B"5\B"5\B"5\B"S#S$9\B"S#S$9\B"S#S$9S%.5rC\,S2S&\;4S'jj5rD\,S2S&\;4S(jj5rE\,S2S&\;4S)jj5rF\,S2S&\;4S*jj5rG\,S2S&\;4S+jj5rH\,S2S&\;4S,jj5rI\."\1S-S.S/S0.5 g)4aNested Transformer (NesT) in PyTorch A PyTorch implement of Aggregating Nested Transformers as described in: 'Aggregating Nested Transformers' - https://arxiv.org/abs/2105.12723 The official Jax code is released and available at https://github.com/google-research/nested-transformer. The weights have been converted with convert/convert_nest_flax.py Acknowledgments: * The paper authors for sharing their research, code, and model weights * Ross Wightman's existing code off which I based this Copyright 2021 Alexander Soare N)partial)ListOptionalTupleTypeUnion)nnIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) PatchEmbedMlpDropPathcalculate_drop_path_ratescreate_classifier trunc_normal__assert create_conv2d create_pool2d to_ntupleuse_fused_attn LayerNorm)build_model_with_cfg)feature_take_indices)register_notrace_function)checkpoint_seq named_apply)register_modelgenerate_default_cfgsregister_model_deprecationsNestc ^\rSrSr%Sr\R R\\ S'S S\ S\ S\S\ S\ 4 U4S jjjr S r S rU=r$) Attention6z This is much like `.vision_transformer.Attention` but uses *localised* self attention by accepting an input with an extra "image block" dim fused_attndim num_headsqkv_bias attn_drop proj_dropcX>XgS.n[T U]5 X lX-n U S-Ul[ 5Ul[ R"USU-4SU0UD6Ul[ R"U5Ul [ R"X40UD6Ul [ R"U5Ul g)Ndevicedtypegbias) super__init__r(scalerr&r LinearqkvDropoutr*projr+) selfr'r(r)r*r+r.r/ddhead_dim __class__s O/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/models/nest.pyr3Attention.__init__=s / "#% (*99S!C%=h="=I.IIc-"- I.c URup#pEURU5RX#USURXPR-5R SSSSSS5nUR S5upxn UR (a=[R"XxXR(aURROSS9nOFXpR-nXxRS S 5-n U RS S 9n URU 5n X-nUR SSSSS5RX#XE5nURU5nUR!U5nU$) z] x is shape: B (batch_size), T (image blocks), N (seq length per image block), C (embed dim) r0rr) dropout_p)r')shaper6reshaper(permuteunbindr&Fscaled_dot_product_attentiontrainingr*pr4 transposesoftmaxr8r+) r9xBTNCr6qkvattns r=forwardAttention.forwardSsWW ahhqk!!!1dnna>>>QRZZ[\^_abdeghjkl**Q-a ??..qQVcVc$..BRBRiklAJJA{{2r**D<>$'DA IIaAq! $ , ,Q1 8 IIaL NN1 r?)r*r&r(r8r+r6r4)FrDrDNN)__name__ __module__ __qualname____firstlineno____doc__torchjitFinalbool__annotations__intfloatr3r[__static_attributes__ __classcell__r<s@r=r$r$6su %% "!!/// /  /  //,r?r$c^\rSrSrSrSSSSS\R \RSS4 S\S\S \ S \ S \ S \ S \ S\ \RS\ \R4U4Sjjjr SrSrU=r$)TransformerLayerlz This is much like `.vision_transformer.Block` but: - Called TransformerLayer here to allow for "block" as defined in the paper ("non-overlapping image blocks") - Uses modified Attention layer that handles the "block" dimension @FrDNr'r( mlp_ratior)r+r* drop_path act_layer norm_layerc >XS.n [TU]5 U "U40U D6Ul[U4UUUUS.U D6UlUS:a [ U5O[ R"5UlU "U40U D6Ul [X-5n [SUU UUS.U D6Ul US:a[ U5Ul g[ R"5Ul g)Nr-)r(r)r*r+rD) in_featureshidden_featuresrsdrop)r2r3norm1r$rZrr Identity drop_path1norm2rhrmlp drop_path2)r9r'r(rqr)r+r*rrrsrtr.r/r:mlp_hidden_dimr<s r=r3TransformerLayer.__init__rs / *r*        2;R(9-R[[]*r* S_- *    2;R(9-R[[]r?cURU5nXRURU55-nXRUR UR U555-nU$N)rzr|rZrr~r})r9rRys r=r[TransformerLayer.forwardsO JJqM  ! - - A 78 8r?)rZr|rr~rzr})r^r_r`rarbr GELUrrhrirfrModuler3r[rjrkrls@r=rnrnls ""!!!)+*,,,#S#S#S #S  #S  #S#S#SBII#SRYY#S#SJr?rnc f^\rSrSrS S\S\S\\RS\4U4Sjjjr Sr Sr U=r $) ConvPool in_channels out_channelsrtpad_typec>XVS.n[TU]5 [X4SUSS.UD6UlU"U40UD6Ul[ SSSUS9Ulg)Nr-r0T) kernel_sizepaddingr1maxrB)rstrider)r2r3rconvnormrpool) r9rrrtrr.r/r:r<s r=r3ConvPool.__init__sX / !+nT\cgnkmn |2r2 !%Qq(S r?c0[URSS-S:HS5 [URSS-S:HS5 URU5nURUR SSSS55R SSSS5nUR U5nU$)z* x is expected to have shape (B, C, H, W) rFrBrz1BlockAggregation requires even input spatial dimsrGr0r)rrHrrrJrr9rRs r=r[ConvPool.forwards  a1$&YZ a1$&YZ IIaL IIaii1a+ , 4 4Q1a @ IIaLr?)rrr)NN) r^r_r`rarhrr rstrr3r[rjrkrls@r=rrsV  T T TRYY T  T T  r?r block_sizecURup#pE[X1-S:HS5 [XA-S:HS5 X1-nXA-nURX&XX5nURSS5RX&U-SU5nU$)zimage to blocks Args: x (Tensor): with shape (B, H, W, C) block_size (int): edge length of a single square block in units of H, W rz,`block_size` must divide input height evenlyz+`block_size` must divide input width evenlyrBr0rG)rHrrIrP)rRrrSHWrV grid_height grid_widths r=blockifyrs~ ''JA! ANa !OP ANa !NO/KJ !**HA Aq!!!:%=r1EA Hr?cURup#pE[[R"U55nXa-=pxUR X&XaX5nUR SS5R X'X5nU$)zblocks to image Args: x (Tensor): with shape (B, T, N, C) where T is number of blocks and N is sequence size per block block_size (int): edge length of a single square block in units of desired H, W rBr0)rHrhmathsqrtrIrP) rRrrSrT_rV grid_sizeheightwidths r= deblockifyrsbJA!DIIaL!I++F ! zEA Aq!!!U6A Hr?c^\rSrSrSrSS\S\S\S\S\S\S \\S \S \S \S \S\\ \S\\ \ RS\\ \ RS\ 4U4SjjjrSrSrU=r$) NestLevelz6Single hierarchical level of a Nested Transformer num_blocksr seq_lengthr(depth embed_dimprev_embed_dimrqr)r+r*rrrtrsrc>UUS.n[TU]5 X lSUl[R "[ R"SXU40UD65UlUb[Xv4XS.UD6Ul O[R"5Ul [U 5(a[U 5U:XdS5e[R"[U5Vs/sH$n[SUUUU U U U (aU UOSU US. UD6PM& sn6Ulgs snf)Nr-Fr)rtrzDMust provide as many drop path rates as there are transformer layers) r'r(rqr)r+r*rrrtrsry)r2r3rgrad_checkpointingr Parameterrczeros pos_embedrrr{len Sequentialrangerntransformer_encoder)r9rrrr(rrrrqr)r+r*rrrtrsrr.r/r:ir<s r=r3NestLevel.__init__s ( / $"'ekk!ZY&]Z\&]^  % kzkhjkDI DI y>>y>U* r,r r*#%==5\ 3#"  ##!##*3)A,%#  " 3# $$ 3#s+C=cURU5nURSSSS5n[XR5nXR-nUR (a:[ RR5(d[URU5nOURU5n[XR5nURSSSS5$)z expects x as (B, C, H, W) rrBr0r) rrJrrrrrcrd is_scriptingrrrrs r=r[NestLevel.forwards IIaL IIaAq ! Q (    " "599+A+A+C+Ct77;A((+A q// *yyAq!$$r?)rrrrr) NrpTrDrDNNNrNN)r^r_r`rarbrhrrirfrrr rrr3r[rjrkrls@r=rrs-1!!!!/34837%0$0$0$ 0$  0$  0$0$%SM0$0$0$0$0$ U ,0$!bii10$ RYY00$ !0$0$d%%r?rc'f^\rSrSrSrS.S\S\S\S\S\\S4S \\S4S \\S4S \S \S \S\S\S\S\S\ \ \ RS\ \ \ RS\ S\ S\ 4&U4Sjjjr\R R"S/Sj5r\R R"S5r\R R"S0Sj5r\R R"S1Sj5r\R R"S\ R4Sj5rS2S \S\ 4SjjrS3S\R0S \ \\\\4S!\S"\S#\ S$\S\\\R0\\R0\\R0444S%jjrS4S \\\\4S&\S'\4S(jjrS)rS0S*\4S+jjrS,rS-r U=r!$)5r"i zxNested Transformer (NesT) A PyTorch impl of : `Aggregating Nested Transformers` - https://arxiv.org/abs/2105.12723 img_sizein_chans patch_size num_levels embed_dims.r(depths num_classesrqr) drop_rateproj_drop_rateattn_drop_ratedrop_path_ratertrsr weight_init global_poolc2>[T U]5 UUS.nSHUn[5Un[U[R R 5(dM;[U5U:XaMLSUS35e [U5"U5n[U5"U5n[U5"U5nXl X l US=Ul Ul /Ul U=(d [nU=(d [R nXlX@l[U[R R 5(aUSUS:XdS5eUSnX-S:XdS 5eX0lS [(R*"US [(R,S9-R/S5R15UlX-[4R6"UR2S5-S:XdS 5e[9X-[4R6"UR2S5-5Ul[=SUUUUSS S.UD6UlUR>R@Ul UR@UR2S-Ul!/n[EXSS9nSnS n[G[UR255HnUUnURI[KUR2UUR:URBUUUUUU4U U U U UUUUUS.UD65 U=R[MUUSU3S9/- sl UnUS-nM [RN"U6Ul(U"US40UD6Ul)[UURUR4SU0UD6unnUUl+[RX"U 5Ul-UUl.UR_U5 g)a Args: img_size (int, tuple): input image size in_chans (int): number of input channels patch_size (int): patch size num_levels (int): number of block hierarchies (T_d in the paper) embed_dims (int, tuple): embedding dimensions of each level num_heads (int, tuple): number of attention heads for each level depths (int, tuple): number of transformer layers for each level num_classes (int): number of classes for classification head mlp_ratio (int): ratio of mlp hidden dim to embedding dim for MLP of transformer layers qkv_bias (bool): enable bias for qkv if True drop_rate (float): dropout rate for MLP of transformer layers, MSA final projection layer, and classifier attn_drop_rate (float): attention dropout rate drop_path_rate (float): stochastic depth rate norm_layer: (nn.Module): normalization layer for transformer layers act_layer: (nn.Module): activation layer in MLP of transformer layers pad_type: str: Type of padding to use '' for PyTorch symmetric, 'same' for TF SAME weight_init: (str): weight init scheme global_pool: (str): type of pooling operation to apply to final feature map Notes: - Default values follow NesT-B from the original Jax code. - `embed_dims`, `num_heads`, `depths` should be ints or tuples with length `num_levels`. - For those following the paper, Table A1 may have errors! - https://github.com/google-research/nested-transformer/issues/2 r-rr(rz Require `len(z) == num_levels`rGrrz Model only handles square inputsz*`patch_size` must divide `img_size` evenlyrAcpuzUFirst level blocks don't fit evenly. Check `img_size`, `patch_size`, and `num_levels`F)rrrrflattenT) stagewiseN)rqr)r+r*rrrtrsrzlevels.)num_chs reductionmodulerB pool_typery)0r2r3locals isinstance collectionsabcSequencerrrr num_featureshead_hidden_size feature_inforr rrrrrcarangelongfliptolistrrrrhrr patch_embed num_patchesrrrappendrdictrlevelsrrrr7 head_drophead init_weights)!r9rrrrrr(rrrqr)rrrrrtrsrrrr.r/r: param_name param_valuerdp_ratesprev_dim curr_striderr'rr<s! r=r3 Nest.__init__'sf  /?J (:.K+{'?'?@@;':5czlRb7cc5@ z*:6 j))4 :&v.& 4>rNBD1,9 ( "$ h  8 8 9 9A;(1+- Q/Q Q-{H$)W+WW)$ ZUZZ XX^^_`ahhj&$))DOOA4F*GG1L e d eL x5$))DOOTUDV:WWX& ! m     ++77**dooa.@@,^tT s4??+,AQ-C MM)"! q $!(("1+%#! ! $   $skT[\][^R_"`!a a H 1 K--.mmV, z"~44 .d.?.?AQAQo]holno T&I.  +&r?cUS;deSU;a![R"UR5*OSnURHn[ UR SSSS9 M [ [[US9U5 g) N)nlhbrrrD{Gz?rFrBstdab) head_bias) rlogrrrrrr_init_nest_weights)r9moderlevels r=rNest.init_weightssa|###39T>TXXd..//r [[E %//sbA >!G.)DdKr?cr[[UR55Vs1sH nSUS3iM sn$s snf)Nzlevel.z .pos_embed)rrr)r9rs r=no_weight_decayNest.no_weight_decays305c$++6F0GH0G1&:&0GHHHs4c6[SU(aSOSS4SS/S9nU$)Nz ^patch_embedz^levels\.(\d+)z*^levels\.(\d+)\.transformer_encoder\.(\d+))z"^levels\.(\d+)\.(?:pool|pos_embed))r)z^norm)i)stemblocks)r)r9coarsematchers r= group_matcherNest.group_matchers0 &,"2_aef=$ r?c6URH nXlM gr)rr)r9enablels r=set_grad_checkpointingNest.set_grad_checkpointingsA#) r?returncUR$r)r)r9s r=get_classifierNest.get_classifiers yyr?chXl[URURUS9uUlUlg)N)r)rrrrr)r9rrs r=reset_classifierNest.reset_classifiers0&&7   t//;'H#$)r?rRindicesr stop_early output_fmtintermediates_onlyc US;dS5e/n[[UR5U5upURU5n[UR5S- n [ R R5(dU(d URn OURSU S-n [U 5HyupU "U5nX;dMU(aMX:XaHURURSSSS55RSSSS5nURU5 MhURU5 M{ U(aU$W U :Xa5URURSSSS55RSSSS5nX4$)aForward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence norm: Apply norm layer to compatible intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features Returns: )NCHWzOutput shape must be NCHW.rNrrBr0) rrrrrrcrdr enumeraterrJr)r9rRrrrrr intermediates take_indices max_indexlast_idxstagesfeat_idxstagex_inters r=forward_intermediatesNest.forward_intermediatessI*Y&D(DD& "6s4;;7G"Q    Q t'!+ 99 ! ! # #:[[F[[)a-0F(0OHaA'H0"ii !Q1(=>FFq!QPQRG!((1!((+ 1  x  !))Aq!Q/088Aq!DAr? prune_norm prune_headc[[UR5U5upEURSUS-UlU(a[R"5UlU(aUR SS5 U$)z?Prune layers not required for specified intermediates. Nrrr)rrrr r{rr)r9rr&r'rrs r=prune_intermediate_layersNest.prune_intermediate_layers s[#7s4;;7G"Q kk.9q=1  DI   ! !!R (r?cURU5nURU5nURURSSSS55RSSSS5nU$)NrrBr0r)rrrrJrs r=forward_featuresNest.forward_featuressR   Q  KKN IIaii1a+ , 4 4Q1a @r? pre_logitsczURU5nURU5nU(aU$URU5$r)rrr)r9rRr.s r= forward_headNest.forward_head!s5   Q  NN1 q0DIIaL0r?cJURU5nURU5nU$r)r,r0rs r=r[ Nest.forward&s'  ! !! $   a r?)rrrrrrrrrrrrrrrrrr)r0rAr0irAr]rBrBrpTrDrDrDg?NNrravgNNrF)T)r=)NFFrF)rFT)"r^r_r`rarbrhrrirfrrr rrr3rcrdignorerrrr rrTensorrrr$r)r,r0r[rjrkrls@r=r"r" s *9)3&0#!!!$&$&$'4837!$-H'H'H' H'  H' c3h H'S#XH'#s(OH'H'H'H'H'"H'"H'"H' !bii1!H'" RYY0#H'$%H'&'H'()H'H'T YYLL YYII YY   YY** YY HCHcH8<$$',1 ||1 eCcN341  1  1  1 !%1  tELL!5tELL7I)I#JJ K1 j./$# 3S >*  1$1 r?rnamercf[U[R5(aURS5(aA[ UR SSSS9 [R RURU5 g[ UR SSSS9 URb*[R RUR5 gg[U[R5(aN[ UR SSSS9 URb*[R RUR5 ggg)zeNesT weight initialization Can replicate Jax implementation. Otherwise follows vision_transformer.py rrrFrBrN) rr r5 startswithrweightinit constant_r1zeros_Conv2d)rrBrs r=rr,s&"))$$ ??6 " " &--SB! < GG  fkk9 5 &--SB! <{{&v{{+' FBII & &fmma8 ;; " GGNN6;; ' # 'r?c[RSURUR5 URSnURSSup4[[R "X4-55n[ U[[R "U555RSSSS5n[R"XU/SSS9n[URSSSS5[[R "U555nU$) z Rescale the grid of position embeddings when loading from state_dict Expected shape of position embeddings is (1, T, N, C), and considers only square images z$Resized position embedding: %s to %srBrr0rbicubicF)sizer align_corners) _loggerinforHrhrrrrJrL interpolater)posemb posemb_newseq_length_oldnum_blocks_newseq_length_newsize_news r=resize_pos_embedrW>s  LL7zGWGWX\\!_N%/%5%5a%:"N499^:; ? G G1aQR SF ]]68(<9\a bF fnnQ1a0#dii6O2P QF Mr?cUR5Vs/sHo"RS5(dMUPM nnUHCnXR[X5R:wdM*[ X[X55X'ME U$s snf)z3resize positional embeddings of pretrained weights pos_embed_)keysrDrHgetattrrW) state_dictmodelrXpos_embed_keyss r=checkpoint_filter_fnr_Osp!+!2Q!2All<6Pa!2NQ  =  '%"3"9"9 9,Z]GE[SSSSS.UD6n[SSU0UD6nU$)Nest-B @ 224x224 r5r8r:rrfry) nest_baserrhrfrg model_kwargsr]s r=ryry|s<W"jWOUWL  L L| LE Lr?c >[SSSSS.UD6n[SSU0UD6nU$)Nest-S @ 224x224 `ir0 r:rrfry) nest_smallrzr{s r=rrs3e>ZPZe^deL  M* M ME Lr?c >[SSSSS.UD6n[SSU0UD6nU$)Nest-T @ 224x224 rrrBrBr]rrfry) nest_tinyrzr{s r=rrs3d>ZPYd]cdL  L L| LE Lr?c bURSS5 [SSSSS.UD6n[S SU0UD6nU$) rxrsamer5r8r:rrfry) nest_base_jx setdefaultrrhr{s r=rrsL j&)W"jWOUWL  OJ O, OE Lr?c bURSS5 [SSSSS.UD6n[S SU0UD6nU$) r~rrrrr:rrfry) nest_small_jxrr{s r=rrsC j&)e>ZPZe^deL  PZ P< PE Lr?c bURSS5 [SSSSS.UD6n[S SU0UD6nU$) rrrrrrrrfry) nest_tiny_jxrr{s r=rrsC j&)d>ZPYd]cdL  OJ O, OE Lr?rrr) jx_nest_base jx_nest_small jx_nest_tiny)rrDr?r>)Jrbcollections.abcrloggingr functoolsrtypingrrrrrrctorch.nn.functionalr functionalrL timm.datar r timm.layersr rrrrrrrrrrr_builderr _featuresr _features_fxr _manipulaterr _registryrr r!__all__ getLoggerr^rNrr$rnrrhrrrr"rrirrWr_rhru default_cfgsryrrrrrryr?r=rs" 55 A    ++34YY (   H %3 3l/ryy/dryy:  C    c    C% C%LI299IX(ryy((U($"  %6 F6"W5#g6"W5& TdTH"$"'r?