RЦi,SrSSKJrJrJrJr SSKrSSKJr SSKJs J r SSK J r SSK Jr SSKJr S \\4S jr"S S \R&5r"S S\R&5rg)a\Halo Self Attention Paper: `Scaling Local Self-Attention for Parameter Efficient Visual Backbones` - https://arxiv.org/abs/2103.12731 @misc{2103.12731, Author = {Ashish Vaswani and Prajit Ramachandran and Aravind Srinivas and Niki Parmar and Blake Hechtman and Jonathon Shlens}, Title = {Scaling Local Self-Attention for Parameter Efficient Visual Backbones}, Year = {2021}, } Status: This impl is a WIP, there is no official ref impl and some details in paper weren't clear to me. The attention mechanism works but it's slow as implemented. Hacked together by / Copyright 2021 Ross Wightman )ListOptionalTupleUnionN)nn)make_divisible) trunc_normal_)_assert permute_maskcURup4pVURSnUS-S-nXRSS5-n U RSXW5n [R"U SS/5R S5n [R"U SXu- /5n U RSUS-U5n U SS2SU2US- S24n U RX4SXX5R SSUSS5n U RU5$)aaCompute relative logits along one dimension As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 Args: q: (batch, height, width, dim) rel_k: (2 * window - 1, dim) permute_mask: permute output dim according to this rrN)shape transposereshapeFpadflattenexpandpermute) qrel_kr BHWdimrel_sizewin_sizexx_pads T/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/layers/halo_attn.py rel_logits_1dr$s77LA!{{1~H1 "H __R $ $A "a"A EE!aV  $ $Q 'E EE%!X\* +E MM"a!eX .E a!X\]"#A !1'..r2xRHA 99\ ""c T^\rSrSrSrS S\S\S\S\4U4SjjjrSrS r S r U=r $) PosEmbedRel=zRelative Position Embedding As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 block_sizer dim_headscalecJ>XVS.n[TU]5 XlX0lX@l[ R "[R"US-S- U40UD65Ul [ R "[R"US-S- U40UD65Ul UR5 g)z Args: block_size: block size win_size: neighbourhood window size dim_head: attention head dim scale: scale factor (for init) devicedtyperrN) super__init__r)r*r+r Parametertorchempty height_rel width_relreset_parameters) selfr)r r*r+r.r/dd __class__s r#r1PosEmbedRel.__init__Cs  / $  ,,u{{8a  dnn$**=r%c*URup#pEURSURURUR5n[ XR SS9nUR SS5n[ XRSS9nXv-nURX#US5nU$)Nr)rrr)r rr)rrCrrDr)rrr)r*r$r6rr5) r8rrBBHW_ rel_logits_w rel_logits_h rel_logitss r#forwardPosEmbedRel.forwardbsww r IIb$//4??DMM J$Q_U  KK1 $QoV !0 ''r26 r%)r)r*r5r+r6)NN) __name__ __module__ __qualname____firstlineno____doc__intfloatr1r7rK__static_attributes__ __classcell__r:s@r#r'r'=sO        6>  r%r'c^\rSrSrSrSS\S\\S\\\\4S\S\S\\S \S \S \S \ S \ S\ 4U4Sjjjr Sr Sr Sr U=r$)HaloAttnracHalo Attention Paper: `Scaling Local Self-Attention for Parameter Efficient Visual Backbones` - https://arxiv.org/abs/2103.12731 The internal dimensions of the attention module are controlled by the interaction of several arguments. * the output dimension of the module is specified by dim_out, which falls back to input dim if not set * the value (v) dimension is set to dim_out // num_heads, the v projection determines the output dim * the query and key (qk) dimensions are determined by * num_heads * dim_head if dim_head is not None * num_heads * (dim_out * attn_ratio // num_heads) if dim_head is None * as seen above, attn_ratio determines the ratio of q and k relative to the output if dim_head not used Args: dim (int): input dimension to the module dim_out (int): output dimension of the module, same as dim if not set feat_size (Tuple[int, int]): size of input feature_map (not used, for arg compat with bottle/lambda) stride: output stride of the module, query downscaled if > 1 (default: 1). num_heads: parallel attention heads (default: 8). dim_head: dimension of query and key heads, calculated from dim_out * attn_ratio // num_heads if not set block_size (int): size of blocks. (default: 8) halo_size (int): size of halo overlap. (default: 3) qk_ratio (float): ratio of q and k dimensions to output dimension when dim_head not set. (default: 1.0) qkv_bias (bool) : add bias to q, k, and v projections avg_down (bool): use average pool downsample instead of strided query blocks scale_pos_embed (bool): scale the position embedding as well as Q @ K rdim_out feat_sizestride num_headsr*r) halo_sizeqk_ratioqkv_biasavg_downscale_pos_embedc>XS.n[TU]5 U=(d UnX%-S:XdeUS;deXPlU=(d [X)-SS9U-UlX R-UlXPR-UlXPR -UlURS-UlXl U=Ul Ul Xl XxS--Ul SUlS nUS:a>U =(d Xt-S:gnU(aSOUUlURUR-Ul [R "XR S4URU S .UD6Ul[R "XR UR-S4S U 0UD6Ul['S URURURURS .UD6UlU(a[R*"SS5O[R,"5UlUR15 g)Nr-r)rr)divisorrrF)r\biasrg)r)r r*r+)r0r1r]r dim_head_qk dim_head_v dim_out_qk dim_out_vr+rbr) block_size_dsr^r block_striderConv2drkvr' pos_embed AvgPool2dIdentitypoolr7)r8rrZr[r\r]r*r)r^r_r`rarbr.r/r9 use_avg_poolr:s r#r1HaloAttn.__init__s" / .S"a'''"#a~g6HRS'TXa'a!^^3#&6&66"__4%%- ./99$,""]2  A:#?z':a'?L%1vD !%D4E4E!ED  3b4;L;LS[b_ab))C4>>!A1Z8ZWYZ$ ))]]%%**    +7BLLA&BKKM  r%chURRRSS-n[URRUS9 [URRUS9 [UR R URS9 [UR RURS9 g)Nrrfr=) rweightrr rprqr5r+r6)r8r>s r#r7HaloAttn.reset_parametersssffmm!!!$,dffmm-dggnn#.dnn//TZZ@dnn..DJJ?r%clURup#pE[X@R-S:HS5 [XPR-S:HS5 X@R-nXPR-nXg-nURU5n U R SUR X`R XpR 5RSSSSSS5n U R X R-UR SU5RSS5n URU5n [R"XRURURUR/5n U RSURUR5RSURUR5R X R-UR UR -US5RSSSS5n ["R$"XR UR /SS 9upUR&(a4XRSS 5-UR)U 5-UR*-n O3XRSS 5-UR*-UR)U 5-n U R-SS 9n X-RSS5nUR SUR UR Xg5nURSSSSS5R/5R1X R2X@R4-XPR4-5nUR7U5nU$) NrrrrCrrD)rr)rr r)rrrirmrr]rrprrr^unfoldr rjr3splitrbrqr+softmax contiguousviewrlrnrt)r8r!rCrr num_h_blocks num_w_blocks num_blocksrrpkvattnouts r#rKHaloAttn.forwardsWW aOO#q("-OO#q("-OO+ OO+ !0 FF1I II   ,,lrs`$0/ #& #$s)#>2"))2jEryyEPr%