RЦi/)SrSSKJrJrJr SSKrSSKJr SSKJs J r SSK J r "SS\R5r"SS \R5r"S S \R5r"S S \R5rg)aGNon-Local Attention Pooling Layers A collection of global pooling layers that go beyond simple avg/max pooling. LSEPool - LogSumExp pooling, a smooth approximation between avg and max pooling SimPool - Attention-based pooling from 'Keep It SimPool' (ICCV 2023) Based on implementations from: * LSE Pooling: custom implementation by Bill Psomas * SimPool: https://arxiv.org/abs/2309.06891 - 'Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit?' by Bill Psomas et al. Hacked together by / Copyright 2024 Ross Wightman, original code by Bill Psomas )OptionalTypeUnionN)use_fused_attnc^\rSrSrSrS S\S\S\4U4SjjjrS\RS\R4S jr S r U=r $) LsePlus2damLogSumExp (LSE) Pooling for 2D inputs. A smooth approximation to max pooling that provides a learnable interpolation between average and max pooling. When r is large, LSE approaches max pooling; when r is small, it approaches average pooling. Implements: (1/r) * log((1/n) * sum(exp(r * (x - x_max)))) + x_max The x_max subtraction provides numerical stability. r r_learnableflattenc >[TU]5 U(a/[R"[R "XUS95UlO%URS[R "XUS95 X0lg)z Args: r: Initial value of the pooling parameter. Higher = closer to max pooling. r_learnable: If True, r is a learnable parameter. flatten: If True, flatten spatial dims in output. devicedtyper N) super__init__nn Parametertorchtensorr register_bufferr )selfr r r rr __class__s U/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/layers/other_pool.pyrLsePlus2d.__init__$sN  \\%,,qu"MNDF  ell15&Q R xreturnc4[R"US5n[R"URX- -5nUR SSS9nUSUR- [R "U5--nUR(aURS5nU$)Nr)Tdimkeepdim?)Fadaptive_max_pool2drexpr meanlogr rrx_maxexp_xsum_expouts rforwardLsePlus2d.forward9sx%%a+ $&&AI./***6sTVV|uyy'999 <<++a.C r)r r )$@TTNN __name__ __module__ __qualname____firstlineno____doc__floatboolrrTensorr1__static_attributes__ __classcell__rs@rr r s_  $   *%,,rr c~^\rSrSrSrS S\S\4U4SjjjrS\RS\R4Sjr S r U=r $) LsePlus1dCaLogSumExp (LSE) Pooling for sequence (NLC) inputs. A smooth approximation to max pooling that provides a learnable interpolation between average and max pooling. When r is large, LSE approaches max pooling; when r is small, it approaches average pooling. r r c >[TU]5 U(a/[R"[R "XUS95UlgURS[R "XUS95 g)z Args: r: Initial value of the pooling parameter. Higher = closer to max pooling. r_learnable: If True, r is a learnable parameter. rr N)rrrrrrr r)rr r rrrs rrLsePlus1d.__init__KsI  \\%,,qu"MNDF  ell15&Q RrrrcURSSS9Rn[R"URX- -5nUR SSS9nUSUR- [R "U5--nURS5$)NrTr#r&)maxvaluesrr)r r*r+squeezer,s rr1LsePlus1d.forward]ss!T*11 $&&AI./**D*1sTVV|uyy'999{{1~r)r )r3TNNr4r?s@rrArACsV $ SSSS$%,,rrAc^\rSrSr%Sr\R R\\ S'SS\ S\ S\S\S\ \ S \ \ \R4 U4S jjjrS \R"S \R"4S jrSrU=r$) SimPool2dfaNSimPool: Simple Attention-Based Pooling for 2D (NCHW) inputs. From 'Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit?' https://arxiv.org/abs/2309.06891 Uses GAP as query initialization and applies cross-attention between the GAP query and spatial features to produce a weighted pooled representation. fused_attnr$ num_headsqkv_biasqk_normgamma norm_layerc V>[T U]5 XxS.n X-S:XdS5eX lX-UlURS-UlXPl[ 5UlU=(d [RnU"U40U D6Ul [R"X4SU0U D6Ul [R"X4SU0U D6Ul U(a1U"UR40U D6UlU"UR40U D6Ulg[R "5Ul[R "5Ulg)a Args: dim: Input feature dimension (number of channels). num_heads: Number of attention heads. qkv_bias: If True, add bias to query and key projections. qk_norm: If True, apply normalization to queries and keys. gamma: If provided, apply power normalization to values with this exponent. norm_layer: Normalization layer for patches and optionally qk_norm. flatten: If True, flatten output to (B, C). rr"dim must be divisible by num_headsbiasNrrrNhead_dimscalerQrrMr LayerNormnormLinearqkq_normk_normIdentity rr$rNrOrPrQrRrrddrs rrSimPool2d.__init__qs*  /!#I%II#"( ]]d*  (*/2<< s)b) 39(9b939(9b9 $T]]9b9DK$T]]9b9DK++-DK++-DKrrrcURup#pEXE-nURS5RSS5nURSSS9nUR U5nUR U5R USURUR5RSS5nURU5R X&URUR5RSS5n UR X&URUR5RSS5n URU5URU 5pURbU RSSS9n X- S-n UR(a1[R "XyU R#UR55n OMXpR$-U RSS5-nUR'SS9nXR#UR5-n U R#S UR- 5nO]UR(a[R "XyU 5nO4XpR$-U RSS5-nUR'SS9nX-nURSS5R X#5nU$) Nr!rTr#ư>r$r&)shaper transposer*r[r]reshaperNrXr^r_r`rQaminrMr'scaled_dot_product_attentionpowrYsoftmax)rrBCHWNr]x_normr^vv_min v_shiftedattn_outattnr0s rr1SimPool2d.forwardsWW a E IIaL " "1a ( FFq$F '1 FF1I  aDNNDMM B L LQPQ R FF6N " "1 G Q QRSUV W NN1 ? I I!Q O{{1~t{{1~1 :: !FFr4F0E D(I99! djj@YZJJ!++b"*==|||+-- ";;,,sTZZ/0C44Q1=JJ!++b"*==|||+hmmAq!))!/ r rMrQrXr^r`r[rNr]r_rYrFFNNNNr5r6r7r8r9rjitFinalr;__annotations__intrr:rrModulerr<r1r=r>r?s@rrKrKfs %% "!%)48'('('( '(  '( E? '(!bii1'('(R))%,,))rrKc^\rSrSr%Sr\R R\\ S'SS\ S\ S\S\S\ \ S \ \ \R4 U4S jjjrS \R"S \R"4S jrSrU=r$) SimPool1daRSimPool: Simple Attention-Based Pooling for sequence (NLC) inputs. From 'Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit?' https://arxiv.org/abs/2309.06891 Uses GAP as query initialization and applies cross-attention between the GAP query and sequence tokens to produce a weighted pooled representation. rMr$rNrOrPrQrRc V>[T U]5 XxS.n X-S:XdS5eX lX-UlURS-UlXPl[ 5UlU=(d [RnU"U40U D6Ul [R"X4SU0U D6Ul [R"X4SU0U D6Ul U(a1U"UR40U D6UlU"UR40U D6Ulg[R "5Ul[R "5Ulg)ag Args: dim: Input feature dimension. num_heads: Number of attention heads. qkv_bias: If True, add bias to query and key projections. qk_norm: If True, apply normalization to queries and keys. gamma: If provided, apply power normalization to values with this exponent. norm_layer: Normalization layer for tokens and optionally qk_norm. rrrTrUrVNrWrbs rrSimPool1d.__init__s(  /!#I%II#"( ]]d*  (*/2<< s)b) 39(9b939(9b9 $T]]9b9DK$T]]9b9DK++-DK++-DKrrrcURup#nURSSS9nURU5nURU5R USUR UR 5RSS5nURU5R X#UR UR 5RSS5nUR X#UR UR 5RSS5nURU5URU5puURbURSSS9n X- S-n UR(a1[R"XWU R!UR55n OMXPR"-URSS5-n U R%SS9n XR!UR5-n U R!S UR- 5n O]UR(a[R"XWU5n O4XPR"-URSS5-n U R%SS9n X-n U RSS5R X$5n U $) NrTr#r!rfrgrhrir&)rjr*r[r]rlrNrXrkr^r_r`rQrmrMr'rnrorYrp)rrrqrurrr]rvr^rwrxryrzr{r0s rr1SimPool1d.forwards''a FFq$F '1 FF1I  aDNNDMM B L LQPQ R FF6N " "1 G Q QRSUV W NN1 ? I I!Q O{{1~t{{1~1 :: !FFr4F0E D(I99! djj@YZJJ!++b"*==|||+-- ";;,,sTZZ/0C44Q1=JJ!++b"*==|||+hmmAq!))!/ rr}r~rr?s@rrrs %% "!%)48&(&(&( &(  &( E? &(!bii1&(&(P%%%,,%%rr)r9typingrrrrtorch.nnrtorch.nn.functional functionalr'configrrr rArKrrrrsh )( "( (V  F] ]@X Xr