RЦi (0SSKJrJrJr SSKrSSKJr SSKJr SSK J r SSK J r SSK Jr \RR \ SS \R"S \\R"4S jj55r"S S \R&5r"SS\R&5rg))FinalOptionalTypeN)nn) functional)register_notrace_function)use_fused_attn)apply_rot_embed_catscores attn_maskcUcU$X-$N)r r s T/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/layers/attention.pymaybe_add_maskr s&6>F,>>c^\rSrSr%Sr\\\S'SS\S\S\ \S\ \S \S \S \S \S \ S\ S\ \ \ RSS4U4SjjjrSS\R S\ \R S\R 4SjjrSrU=r$) AttentionaStandard Multi-head Self Attention module with QKV projection. This module implements the standard multi-head attention mechanism used in transformers. It supports both the fused attention implementation (scaled_dot_product_attention) for efficiency when available, and a manual implementation otherwise. The module includes options for QK normalization, attention dropout, and projection dropout. fused_attnNdim num_heads attn_head_dimdim_outqkv_biasqk_norm scale_norm proj_bias attn_drop proj_drop norm_layerreturnc >[TU]5 XS.nU=(d UnUnUcX-S:XdS5eX-nU(dU(a U cS5eX lXlX/-UlUS-Ul[ 5Ul[R"XRS-4SU0UD6Ul U(a U "U40UD6O[R"5Ul U(a U "U40UD6O[R"5Ul [R"U 5UlU(aU "UR40UD6O[R"5Ul[R"URU4SU0UD6Ul[R"U 5Ulg) aInitialize the Attention module. Args: dim: Input dimension of the token embeddings. num_heads: Number of attention heads. attn_head_dim: Dimension of each attention head. If None, computed as dim // num_heads. dim_out: Output dimension. If None, same as dim. qkv_bias: Whether to use bias in the query, key, value projections. qk_norm: Whether to apply normalization to query and key vectors. scale_norm: Whether to apply normalization to attention output before projection. proj_bias: Whether to use bias in the output projection. attn_drop: Dropout rate applied to the attention weights. proj_drop: Dropout rate applied after the output projection. norm_layer: Normalization layer constructor for QK normalization if enabled. devicedtypeNr$dim should be divisible by num_heads  /.S   ?a' O)O O''H j) i+i i)" !, % (*99S--!"3I(IbI4;j0R0 4;j0R0 I.7AJt}}33r{{} IIdmmWK9KK I.rxr c"URup4nURU5RX4SURUR5R SSSSS5nUR S5upxn URU5URU5pUR(a?[R"XxU UUR(aURROSS9nOQXpR-nXxR!SS 5-n [#X5n U R%S S 9n URU 5n X-nUR!SS5RX4UR&5nUR)U5nUR+U5nUR-U5nU$) Nr+rrr dropout_pr)shaper3reshaperr/permuteunbindr5r6rFscaled_dot_product_attentiontrainingr pr1 transposersoftmaxr0r8r9r!) r:r>r BNCr3qkvattns rforwardAttention.forwardSsH ''ahhqk!!!4>>4==IQQRSUVXY[\^_`**Q-a{{1~t{{1~1 ??..a#.2mm$..**A JJA{{2r**D!$2D<>$'DA KK1  % %aDMM : IIaL IIaL NN1 r) r0r rr/r6r8rr9r!r5r3r1) NNFFFTrBrBNNNr)__name__ __module__ __qualname____firstlineno____doc__rbool__annotations__intrfloatrrModuler.torchTensorrY__static_attributes__ __classcell__r<s@rrrs d  +/%)"!$"!!485/5/5/$C= 5/ c] 5/  5/5/5/5/5/5/!bii15/ 5/5/t15||  -  rrc8^\rSrSr%Sr\R R\\ S'SS\ S\ S\ \ S\S\S \ S \ S \ S \ \ S \ \RS\S\S\S\4U4SjjjrSS\ \R"S\ \R"4SjjrSrU=r$) AttentionRoperzA Self Attention module with ROPE support. Includes options for: * QK normalization option * Attention output (scale) normalization * Fused or unfused QKV projection support rrrrr qkv_fusednum_prefix_tokensr r!rr"rrr rotate_halfcv>[TU]5 UUS.nU=(d UnU nUcX-S:XdS5eX-nU (dU (a U cS5eX lUUlUU-UlUS-UlX`l[5UlXl U(aA[R"XRS-4SU0UD6Ul S=Ul =UlUlOSUl [R"XR4SU0UD6Ul [R"XR4SU0UD6Ul[R"XR4SU0UD6UlU (a U "U40UD6O[R "5UlU (a U "U40UD6O[R "5Ul[R&"U5UlU (aU "UR40UD6O[R "5Ul[R"URU4SU 0UD6Ul[R&"U5Ulg) a3Initialize the Attention module. Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads dim_out: Output dimension. If None, same as dim. qkv_bias: Whether to add a bias term to the query, key, and value projections qkv_fused: Whether to use fused QKV projection (single linear) or separate projections num_prefix_tokens: Number of reg/cls tokens at the beginning of the sequence that should not have position embeddings applied attn_drop: Dropout rate for attention weights proj_drop: Dropout rate for the output projection attn_head_dim: Dimension of each attention head. If None, computed as dim // num_heads. norm_layer: Normalization layer constructor to use for QK and scale normalization qk_norm: Enable normalization of query (Q) and key (K) vectors with norm_layer scale_norm: Enable normalization (scaling) of attention output with norm_layer proj_bias: Whether to use bias in the output projection rotate_half: Use 'half' ROPE layout instead of default 'interleaved' r%Nrr(r)r*r+r,)r-r.rr/r0r1ror rrprr2r3q_projk_projv_projr4r5r6r7r r8r9r!)r:rrrrrnror r!rr"rrrrpr&r'r;r/r<s rr.AttentionRope.__init__|sL  /.S   ?a' O)O O''H ) i+i i)"   9, % !2(*& yymma&7MhM"MDH6: :DK :$+ DH))CLXLLDK))CLXLLDK))CLXLLDK4;j0R0 4;j0R0 I.7AJt}}33r{{} IIdmmWK9KK I.rroper c NURupEnURbaURU5nURXESURUR5R SSSSS5nUR S5upn OURU5RXEURUR5RSS5nURU5RXEURUR5RSS5n URU5RXEURUR5RSS5n URU5URU 5pUbURn [USS5n [R "USS2SS2SU 2SS24[#USS2SS2U S2SS24X,S 9/SS 9R%U 5n[R "U SS2SS2SU 2SS24[#U SS2SS2U S2SS24X,S 9/SS 9R%U 5n UR&(a?[(R*"XU UUR,(aUR.R0OS S 9nOQXR2-nXRS S5-n [5X5n U R7SS 9n UR/U 5n X-nURSS5RXEUR85nUR;U5nUR=U5nUR?U5nU$)aSForward pass for the attention module. Args: x: Input tensor of shape (batch_size, sequence_length, embedding_dim) rope: Rotary position embeddings tensor for position-aware attention attn_mask: Optional attention mask to apply during attention computation Returns: Tensor of shape (batch_size, sequence_length, dim_out) Nr+r@rrrArpF)halfrGrBrCrErF) rHr3rIrr/rJrKrrrPrsrtr5r6rogetattrrfcatr type_asrrLrMrNr rOr1rrQr0r8r9r!)r:r>rvr rRrSrTr3rUrVrWnptrxrXs rrYAttentionRope.forwards ''a 88 ((1+C++aAt~~t}}EMMaQRTUWXZ[\CjjmGA! A&&qT^^T]]KUUVWYZ[A A&&qT^^T]]KUUVWYZ[A A&&qT^^T]]KUUVWYZ[A{{1~t{{1~1  ((C46D 1Q4C4]+-@1aq=AQSW-cdjklttuvwA 1Q4C4]+-@1aq=AQSW-cdjklttuvwA ??..a#.2mm$..**A JJA B++D!$2D<>$'DA KK1  % %aDMM : IIaL IIaL NN1 r)r0r rr/r6rsr8rror9r!r5rrr3rpr1rt)r[NTTrrBrBNNFFTFNN)NN)r\r]r^r_r`rfjitrrarbrcrrdrrrer.rgrYrhrirjs@rrlrlrs9 %% %)!"%&!!+/*.!$" %#F/F/F/c] F/  F/  F/ #F/F/F/$C=F/RYYF/F/F/F/F/F/V,004 65<<(6  - 66rrlr)typingrrrrfrtorch.nnrrL_fxr configr pos_embed_sincosr fxwraprgrrerrlrrrrs(( $*"1?5<<?HU\\4J??] ]@HBIIHr