QЦi}R%SSKrSSKJr SSKJr SSKJrJrJrJ r J r J r SSK r SSK Jr SSKJrJr SSKJrJr SSKJr S S KJrJrJr S S KJr S S KJrJr /S Qr "SS\ 5r!"SS\5r""SS\RF5r$"SS\RF5r%"SS\RF5r&S\'S\'S\'S\'S\'S\ \S\(S\S \&4S!jr)S"\0r*\\+\4\,S#'0\*ES$S%S&.Er-"S'S(\5r."S)S*\5r/"S+S,\5r0"S-S.\5r1"S/S0\5r2\"5\"S1\.Rf4S29SS3S4.S\ \.S\(S\S \&4S5jj55r4\"5\"S1\/Rf4S29SS3S4.S\ \/S\(S\S \&4S6jj55r5\"5\"S1\0Rf4S29SS3S4.S\ \0S\(S\S \&4S7jj55r6\"5\"S1\1Rf4S29SS3S4.S\ \1S\(S\S \&4S8jj55r7\"5\"S9S29SS3S4.S\ \2S\(S\S \&4S:jj55r8SAS;\'S\'S\+S?\(S S=4 S@jjr9g)BN) OrderedDict)partial)AnyCallableDictList NamedTupleOptional)Conv2dNormActivationMLP)ImageClassificationInterpolationMode)_log_api_usage_once)register_modelWeights WeightsEnum)_IMAGENET_CATEGORIES)_ovewrite_named_paramhandle_legacy_interface) VisionTransformerViT_B_16_WeightsViT_B_32_WeightsViT_L_16_WeightsViT_L_32_WeightsViT_H_14_Weightsvit_b_16vit_b_32vit_l_16vit_l_32vit_h_14c\rSrSr%\\S'\\S'\\S'\Rr\ S\R4\S'\Rr \ S\R4\S'Sr g ) ConvStemConfig out_channels kernel_sizestride. norm_layeractivation_layerN)__name__ __module__ __qualname____firstlineno__int__annotations__nn BatchNorm2dr)rModuleReLUr*__static_attributes__r+d/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/torchvision/models/vision_transformer.pyr$r$ sJ K+->>Jbii(913hsBII~.8r7r$cL^\rSrSrSrSrS\S\S\4U4SjjrU4Sjr S r U=r $) MLPBlock(zTransformer MLP block.r in_dimmlp_dimdropoutcv>[TU]XU/[RSUS9 UR 5Hn[ U[R 5(dM$[RRUR5 URcM\[RRURSS9 M g)N)r*inplacer>ư>std) super__init__r2GELUmodules isinstanceLinearinitxavier_uniform_weightbiasnormal_)selfr<r=r>m __class__s r8rEMLPBlock.__init__-s} 6!2RWWVZdklA!RYY''''166%GGOOAFFO5  r7c >URSS5nUbUS:aN[S5H?n SH6n USU S-SU 3n USU -SU 3n X;dM#URU 5X'M8 MA [T U]UUUUUUU5 g)Nversionr )rLrMlinear_r.)getrangepoprD_load_from_state_dict)rO state_dictprefixlocal_metadatastrict missing_keysunexpected_keys error_msgsrTitypeold_keynew_keyrQs r8r[MLPBlock._load_from_state_dict6s!$$Y5 ?gk1X.D!'!uAdVattention_dropoutr).c>[TU]5 XlU"U5Ul[R "X!USS9Ul[R"U5UlU"U5Ul [X#U5Ul g)NT)r> batch_first) rDrErrln_1r2MultiheadAttentionself_attentionDropoutr>ln_2r:mlp)rOrrrsr=r>rtr)rQs r8rEEncoderBlock.__init__Ysh "z*  33JSdrvwzz'* z* J9r7inputc*[R"UR5S:HSUR35 UR U5nUR X"USS9up#UR U5nX!-nURU5nURU5nX$-$)NrW2Expected (batch_size, seq_length, hidden_dim) got F) need_weights) torch_assertdimshaperwryr>r{r|)rOr~x_ys r8forwardEncoderBlock.forwardns eiikQ&*\]b]h]h\i(jk IIe ""1"? LLO I IIaL HHQKu r7)r>rwr{r|rrryr,r-r.r/rhrr2 LayerNormr0rjrrr4rETensorrr6rkrls@r8rnrnVs$6=R\\t5T::: :  : ! :S%((//12::* U\\  r7rnc^\rSrSrSr\"\RSS94S\S\S\S\S \S \ S \ S \ S \ R R44U4Sjjjr S\ R4SjrSrU=r$)Encoderzz?Transformer Model Encoder for sequence to sequence translation.rArp seq_length num_layersrrrsr=r>rtr).c >[T U]5 [R"[R "SX5R SS95Ul[R"U5Ul [5n [U5Hn [UUUUUU5U SU 3'M [R"U 5UlU"U5Ulg)Nrg{Gz?rBencoder_layer_)rDrEr2 ParameterremptyrN pos_embeddingrzr>rrYrn Sequentiallayersln) rOrrrrrsr=r>rtr)rrcrQs r8rEEncoder.__init__}s  \\%++a*P*X*X]a*X*bczz'* .9mz"A+7! ,F^A3' (#mmF+ Z(r7r~c[R"UR5S:HSUR35 XR-nUR UR URU555$)NrWr)rrrrrrrr>)rOr~s r8rEncoder.forwardsZ eiikQ&*\]b]h]h\i(jk***wwt{{4<<#6788r7)r>rrrrrls@r8rrzsI6=R\\t5T))) )  )  ))!)S%((//12)):9U\\99r7rc.^\rSrSrSrSSSS\"\RSS9S4S\S \S \S \S \S \S\ S\ S\S\ \S\ S\ R R4S\ \\4U4SjjjrS\ R"S\ R"4SjrS\ R"4SjrSrU=r$)rz;Vision Transformer as per https://arxiv.org/abs/2010.11929.iNrArp image_size patch_sizerrrrsr=r>rt num_classesrepresentation_sizer).conv_stem_configsc  >[TU]5 [U5 [R"X-S:HS5 XlX lXPlX`lXl Xpl Xl Xl Xl U b[R"5n Sn[!U 5HeunnU R#SU3[%UUR&UR(UR*URUR,S95 UR&nMg U R#S[R."XSS95 XlO[R."SXRUS 9UlX-S -n[R2"[R4"SSU55UlUS- n[9UUUUUUUU 5UlUUl[?5nU c[R@"XY5US 'OJ[R@"XZ5US '[RB"5US '[R@"X5US '[R"U5Ul"[GUR0[R.5(aUR0RHUR0R(S-UR0R(S-n[RJRMUR0RN[PRR"SU- 5S9 UR0RTb3[RJRWUR0RT5 GOUR0RXGb[GUR0RX[R.5(a[RJR[UR0RXRNS[PRR"SUR0RXR&- 5S9 UR0RXRTb=[RJRWUR0RXRT5 []URDS 5(a[GURDR^[R@5(aURDR^R`n[RJRMURDR^RN[PRR"SU- 5S9 [RJRWURDR^RT5 [GURDRb[R@5(a{[RJRWURDRbRN5 [RJRWURDRbRT5 gg)Nrz&Input shape indivisible by patch size!rW conv_bn_relu_) in_channelsr&r'r(r)r* conv_lastr)rr&r')rr&r'r(r head pre_logitsactrBrg@)meanrC)2rDrErrrrrrsr=rtr>rrr)r2r enumerate add_moduler r&r'r(r*Conv2d conv_projrzeros class_tokenrencoderrrrITanhheadsrHrrJ trunc_normal_rLmathsqrtrMzeros_rrNhasattrr in_featuresr)rOrrrrrrsr=r>rtrrr)rseq_proj prev_channelsrcconv_stem_layer_configr heads_layersfan_inrQs r8rEVisionTransformer.__init__s? D! j-24\]$$$ !2 &#6 $  (}}HM-67H-I))###A3'($1%;%H%H$:$F$F5<<#9#D#D)?)P)P  !7 C C .J   RYY=ghi )1NYYJWaDN!.14 << Aq*(EFa            %4?M  &#%99Z#EL )+:)SL &"$'')L #%99-@#NL ]]<0 dnnbii 0 0^^//$..2L2LQ2OORVR`R`RlRlmnRooF GG ! !$.."7"7TYYq6z=R ! S~~"".t~~223 ^^ % % 1jAYAY[][d[d6e6e GGOO((//ctyyt~~OgOgOtOtIt?u  ~~'',,8t~~77<<= 4::| , ,DJJ">DIIaRXjDY ! Z GGNN4::0055 6 djjooryy 1 1 GGNN4::??11 2 GGNN4::??// 0 2r7rreturncURup#pEURn[R"X@R:HSURSUS35 [R"XPR:HSURSUS35 XF-nXV-nUR U5nUR X RXx-5nURSSS5nU$)NzWrong image height! Expected z but got !zWrong image width! Expected rr r) rrrrrrreshaperspermute) rOrnchwpn_hn_ws r8_process_input VisionTransformer._process_input sWW a OO a??*.KDOOK\\efgehhi,jk a??*.J4??J[[defdggh,ijff NN1  IIa#) 4 IIaA r7cURU5nURSnURRUSS5n[R "X1/SS9nUR U5nUSS2S4nURU5nU$)Nrrr)rrrexpandrcatrr)rOrrbatch_class_tokens r8rVisionTransformer.forward!s~    " GGAJ!,,33Ar2> II(,! 4 LLO adG JJqMr7)rtrrr>rrrsrr=r)rrrr)r,r-r.r/rhrr2rr0rjr rrr4rr$rErrrr6rkrls@r8rrsE#&-15`_'s training recipe.  num_paramsrr_metrics_ops _file_size_docsurl transformsrz>https://download.pytorch.org/models/vit_b_16_swag-9ac1b537.pthr resize_size interpolationi^-)rrg~jtSU@giX@gˡEK@g|?5^t@ These weights are learnt via transfer learning by end-to-end fine-tuning the original `SWAG `_ weights on ImageNet-1K data. rrrrrrzAhttps://download.pytorch.org/models/vit_b_16_lc_swag-4e70ced5.pth+https://github.com/pytorch/vision/pull/5793gbX9xT@gQ X@ These weights are composed of the original frozen `SWAG `_ trunk weights and a linear classifier learnt on top of them trained on ImageNet-1K data. rrrrrrrr+Nr,r-r.r/rrrr IMAGENET1K_V1rBICUBIC_COMMON_SWAG_METAIMAGENET1K_SWAG_E2E_V1IMAGENET1K_SWAG_LINEAR_V1DEFAULTr6r+r7r8rr_s# G.#>  ""f##  ! M,% L +33    ""##  ! 4!( O +33    C""##  ! !6Gr7rc T\rSrSr\"S\"\SS90\ESSSSS S S .0S S SS.ES9r\r Sr g)riz9https://download.pytorch.org/models/vit_b_32-d86f8d99.pthrri1BrzNhttps://github.com/pytorch/vision/tree/main/references/classification#vit_b_32rg|?5^R@gW@rgA`Т@gl u@rrrr+N r,r-r.r/rrrrrrr6r+r7r8rrs\ G.#>  ""f##  ! M,Gr7rc \rSrSr\"S\"\SSS90\ESSSS S S S .0S SSS.ES9r\"S\"\SS\ RS90\ ESSS SSS .0SSSS.ES9r \"S\"\SS\ RS90\ ESSSS SS S .0S SS!S".ES9r \rS#rg$)%riz9https://download.pytorch.org/models/vit_l_16-852ce7e3.pthr)rri#rzNhttps://github.com/pytorch/vision/tree/main/references/classification#vit_l_16rg|?5^S@gFԨW@rgףp= N@g;O$@a These weights were trained from scratch by using a modified version of TorchVision's `new training recipe `_. rrz>https://download.pytorch.org/models/vit_l_16_swag-4f3808c9.pthri0)r r gjtV@gT㥛ĠX@gƟv@gy&11@rrzAhttps://download.pytorch.org/models/vit_l_16_lc_swag-4d563306.pthrgMbXIU@g^I [X@rrr+Nrr+r7r8rrs% G.#3O  #"f##  " M.% L +33    #"##  " 4!( O +33    C#"##  " !6Gr7rc T\rSrSr\"S\"\SS90\ESSSSS S S .0S S SS.ES9r\r Sr g)riz9https://download.pytorch.org/models/vit_l_32-c7638314.pthrri[ErzNhttps://github.com/pytorch/vision/tree/main/references/classification#vit_l_32rg|?5>S@gGzDW@rgK7.@gE@rrrr+Nr r+r7r8rrs\ G.#>  #"f#"  " M,Gr7rc \rSrSr\"S\"\SS\RS90\ ESSSSS S .0S S S S.ES9r \"S\"\SS\RS90\ ESSSSSSS .0SSSS.ES9r \ r Sr g)ri2z>https://download.pytorch.org/models/vit_h_14_swag-80465313.pthri%)rrrgS#V@g#~jX@rg~jŏ@gK7I@rrrzAhttps://download.pytorch.org/models/vit_h_14_lc_swag-c1eb923e.pthrri@%rgZd;OmU@gQnX@g= ףpd@gI k֢@rrr+N)r,r-r.r/rrrrrrrrrr6r+r7r8rr2s$ L +33    #"##  " 4!( O +33    C#"##  " !6%Gr7r pretrained)rT)rrc R[RU5n[SSSSSSUUS.UD6$)au Constructs a vit_b_16 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_B_16_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_B_16_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_B_16_Weights :members:   rrrrrsr=rrr+)rverifyrrrrs r8rrkE(%%g.G      r7c R[RU5n[SSSSSSUUS.UD6$)au Constructs a vit_b_32 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_B_32_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_B_32_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_B_32_Weights :members: r%rrrrr+)rrrrs r8rrrr7c R[RU5n[SSSSSSUUS.UD6$)au Constructs a vit_l_16 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_L_16_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_L_16_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_L_16_Weights :members: rrr+)rrrrs r8r r E(%%g.G      r7c R[RU5n[SSSSSSUUS.UD6$)au Constructs a vit_l_32 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_L_32_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_L_32_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_L_32_Weights :members: r%rrrrrr+)rrrrs r8r!r!r r7)rNc R[RU5n[SSSSSSUUS.UD6$)au Constructs a vit_h_14 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_H_14_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_H_14_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_H_14_Weights :members: r%riirr+)rrrrs r8r"r"r r7r model_statezOrderedDict[str, torch.Tensor]interpolation_mode reset_headscUSnURupgnUS:wa[SUR35eX-S-S-n X:wGa3US-nU S-n USS2SS2SS24n USS2SS2SS24n U RSSS5n [[R "U55n X-U:wa[SX-SU35eU R SXU 5n X-n [RRU U US S 9nUR SX5nURSSS5n[R"X/SS 9nXS'U(aC[5nUR5H#unnURS 5(aMUUU'M% UnU$) awThis function helps interpolate positional embeddings during checkpoint loading, especially when you want to apply a pre-trained model on images with different resolution. Args: image_size (int): Image size of the new model. patch_size (int): Patch size of the new model. model_state (OrderedDict[str, torch.Tensor]): State dict of the pre-trained model. interpolation_mode (str): The algorithm used for upsampling. Default: bicubic. reset_heads (bool): If true, not copying the state of heads. Default: False. Returns: OrderedDict[str, torch.Tensor]: A state dict which can be loaded into the new model. zencoder.pos_embeddingrz%Unexpected position embedding shape: r NrzPseq_length is not a perfect square! Instead got seq_length_1d * seq_length_1d = z and seq_length = T)sizemode align_cornersrr)r ValueErrorrr0rrrr2 functional interpolaterrritems startswith)rrr$r%r&rrrrsnew_seq_lengthpos_embedding_tokenpos_embedding_img seq_length_1dnew_seq_length_1dnew_pos_embedding_imgnew_pos_embeddingmodel_state_copykvs r8interpolate_embeddingsr:s* 78M - 3 3A:Av@ATAT@UVWW .14q8N #a !+Arr1H5)!QR(3.55aA>DIIj12  (J 6bcpdAcCCUV`Uab  .55aTab&4!# 9 9 "# !:! !6 = =a \!6 = =aA F!II':&RXYZ/@+, AL #))+1||G,,*+$Q',+K r7)bicubicF):r collectionsr functoolsrtypingrrrrr r rtorch.nnr2ops.miscr r transforms._presetsrrutilsr_apirrr_metar_utilsrr__all__r$r:r4rnrrr0boolrrstrr1rrrrrrrrrr r!r"r:r+r7r8rIsh #BB 0H'66'B 9Z9+ s+ \!299!H#9bii#9LQ Qh    k " B&  d38n8KL{L^{4M{M`{46%{6%r,0@0N0N!OP6:T"23d]`evQ@,0@0N0N!OP6:T"23d]`evQ@,0@0N0N!OP6:T"23d]`evQ@,0@0N0N!OP6:T"23d]`evQ@!566:T"23d]`ev7H( KKK2K K  K & Kr7