RЦiSrSSKJrJrJr SSKrSSKJr SSKJs J r SSK J r J r SSKJr SSKJr S\\4S jr"S S \R(5r"S S \R(5rg)aOBottleneck Self Attention (Bottleneck Transformers) Paper: `Bottleneck Transformers for Visual Recognition` - https://arxiv.org/abs/2101.11605 @misc{2101.11605, Author = {Aravind Srinivas and Tsung-Yi Lin and Niki Parmar and Jonathon Shlens and Pieter Abbeel and Ashish Vaswani}, Title = {Bottleneck Transformers for Visual Recognition}, Year = {2021}, } Based on ref gist at: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 This impl is a WIP but given that it is based on the ref gist likely not too far off. Hacked together by / Copyright 2021 Ross Wightman )ListOptionalTupleN) to_2tuplemake_divisible) trunc_normal_)_assert permute_maskcURup4pVXRSS5-nURSUSU-S- 5n[R"USS/5R S5n[R"USUS- /5nURSUS-SU-S- 5nUSS2SU2US- S24nURX4SXU5R SSUSS5nURU5$)agCompute relative logits along one dimension As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 Args: q: (batch, heads, height, width, dim) rel_k: (2 * width - 1, dim) permute_mask: permute output dim according to this rrN)shape transposereshapeFpadflattenexpandpermute) qrel_kr BHWdimxx_pads Z/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/layers/bottleneck_attn.py rel_logits_1dr!s77LA! __R $ $A "aQ"A EE!aV  $ $Q 'E EE%!QU $E MM"a!eQUQY /E a!QUVmA !1 ''B2r:A 99\ ""cZ^\rSrSrSrS S\\\4S\S\4U4SjjjrSr Sr S r U=r $) PosEmbedRel8zRelative Position Embedding As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 feat_sizedim_headscalec>XES.n[TU]5 [U5uUlUlX lX0l[R"[R"URS-S- U40UD65Ul [R"[R"URS-S- U40UD65Ul UR5 g)Ndevicedtyperr)super__init__rheightwidthr'r(nn Parametertorchempty height_rel width_relreset_parameters)selfr&r'r(r+r,dd __class__s r r.PosEmbedRel.__init__=s / "+I"6 TZ  ,,u{{4;;?Q3F'WTV'WXekk$**q.12Dh&URT&UV r"c[RRRURUR S9 [RRRUR UR S9 g)Nstd)r3r1initnormal_r5r(r6r8s r r7PosEmbedRel.reset_parametersPsH  doo4::>  dnn$**=r"cURup#nURX RURS5n[ XR SS9nUR SS5n[ XRSS9nXe-nURX#U5nU$)Nr )rrr)r rr)rrDrrEr)rrr/r0r!r6rr5)r8rrHW_ rel_logits_w rel_logits_h rel_logitss r forwardPosEmbedRel.forwardTs}77q IIadjj" 5$Q_U  KK1 $QoV !0 ''r2 r")r'r/r5r(r0r6)NN) __name__ __module__ __qualname____firstlineno____doc__rintfloatr.r7rK__static_attributes__ __classcell__r:s@r r$r$8sM  S#X     &>  r"r$c^\rSrSrSrSS\S\\S\\\\4S\S\S\\S \S \ S \ 4U4S jjjr S r Sr Sr U=r$)BottleneckAttnda=Bottleneck Attention Paper: `Bottleneck Transformers for Visual Recognition` - https://arxiv.org/abs/2101.11605 The internal dimensions of the attention module are controlled by the interaction of several arguments. * the output dimension of the module is specified by dim_out, which falls back to input dim if not set * the value (v) dimension is set to dim_out // num_heads, the v projection determines the output dim * the query and key (qk) dimensions are determined by * num_heads * dim_head if dim_head is not None * num_heads * (dim_out * attn_ratio // num_heads) if dim_head is None * as seen above, attn_ratio determines the ratio of q and k relative to the output if dim_head not used Args: dim (int): input dimension to the module dim_out (int): output dimension of the module, same as dim if not set stride (int): output stride of the module, avg pool used if stride == 2 (default: 1). num_heads (int): parallel attention heads (default: 4) dim_head (int): dimension of query and key heads, calculated from dim_out * attn_ratio // num_heads if not set qk_ratio (float): ratio of q and k dimensions to output dimension when dim_head not set. (default: 1.0) qkv_bias (bool): add bias to q, k, and v projections scale_pos_embed (bool): scale the position embedding as well as Q @ K rdim_outr&stride num_headsr'qk_ratioqkv_biasscale_pos_embedc >XS.n [T U]5 UcS5eU=(d UnX%-S:XdeXPlU=(d [X'-SS9U-UlX R-UlXPR-UlXPR -UlURS-UlXl [R"XR S-UR-S4S U0U D6Ul [U4URURS .U D6UlUS:Xa[R"SS5O[R "5UlUR%5 g) Nr*zBA concrete feature size matching expected input (H, W) is requiredr)divisorrrbias)r'r()r-r.r\r dim_head_qk dim_head_v dim_out_qk dim_out_vr(r_r1Conv2dqkvr$ pos_embed AvgPool2dIdentitypoolr7)r8rrZr&r[r\r'r]r^r_r+r,r9r:s r r.BottleneckAttn.__init__zs/ / $j&jj$.S"a'''"#a~g6HRS'TXa'a!^^3#&6&66"__4%%- .99S//A"5"F_PX_\^_%Yb9I9IQUQ[Q[b_ab*0A+BLLA&2;;=  r"c([URRURRRSS-S9 [URR UR S9 [URRUR S9 g)Nrrcr=)r rjweightrrkr5r(r6rAs r r7BottleneckAttn.reset_parameterss\dhhoo488??+@+@+Ct+KLdnn//TZZ@dnn..DJJ?r"cURup#pE[X@RR:HS5 [XPRR:HS5 UR U5n[ R"XRURUR/SS9upgnURX R-URS5RSS5nURX R-URS5nURX R-URS5RSS5nUR(a$Xg-URU5-UR -n O#Xg-UR -URU5-n U R#SS9n X-RSS5RX RXE5n UR%U 5n U $)Nr)rr r)rr rkr/r0rjr3splitrgrhrr\rerrfr_r(softmaxrn) r8rrCrrrkvattnouts r rKBottleneckAttn.forwardsuWW a^^***B/^^)))2. HHQK++a//4??DNN!SYZ[a IIa..($*:*:B ? I I"b Q IIa..($*:*:B ? IIa..($//2 > H HR P   EDNN1--;DETZZ'$..*;;D|||#x""2r*221nnaKiin r") rerfrgrhr\rnrkrjr(r_) NNrrENg?FFNN)rMrNrOrPrQrRrrrSboolr.r7rKrTrUrVs@r rXrXds0&*37&*!"$)" " c]"  c3h0 "  "  " sm" " " "" " H@ r"rX)rQtypingrrrr3torch.nnr1torch.nn.functional functionalrhelpersrr weight_initr trace_utilsr rRr!Moduler$rXr"r rsZ )( .& #$s)#8)")))XURYYUr"