RЦiSrSSKJrJrJrJrJr SSKrSSKJr SSK J r SSK J r SSK Jr S r"S S \R 5r"S S \R 5rg)zSelective Kernel Convolution/Attention Paper: Selective Kernel Networks (https://arxiv.org/abs/1903.06586) Hacked together by / Copyright 2020 Ross Wightman )ListOptionalTupleTypeUnionN)nn) ConvNormAct)make_divisible)_assertc[U[[45(aUHn[U5s $ US:a US-(deg)N) isinstancelisttuple _kernel_valid)kkis [/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/layers/selective_kernel.pyrrs=!dE]##B $ $ 6a!e ec ^\rSrSrSS\R \R SS4S\S\S\S\\RS \\R4 U4S jjjr S r S r U=r $) SelectiveKernelAttnr Nchannels num_paths attn_channels act_layer norm_layerc>XgS.n[T U]5 X l[R"X4SSS.UD6UlU"U40UD6UlU"SS9Ul[R"X1U-4SSS.UD6Ulg)zkSelective Kernel Attention Module Selective Kernel attention mechanism factored out into its own module. devicedtyper F) kernel_sizebiasT)inplaceN) super__init__rrConv2d fc_reducebnact fc_select) selfrrrrr r#r$dd __class__s rr)SelectiveKernelAttn.__init__s| / "8\PU\Y[\]1b1T*=Y2FhTU\aheghrc[URSUR:HS5 URS5R SSS9nUR U5nUR U5nURU5nURU5nURup#pEURX RX0R-XE5n[R"USS9nU$)Nr )rrT)keepdimdim) r shapersummeanr+r,r-r.viewtorchsoftmax)r/xBCHWs rforwardSelectiveKernelAttn.forward0s dnn,b1 EE!HMM&$M / NN1  GGAJ HHQK NN1 WW a FF1nna>>&91 @ MM! #r)r-r,r+r.r)__name__ __module__ __qualname____firstlineno__rReLU BatchNorm2dintrModuler)rC__static_attributes__ __classcell__r1s@rrrsz!#)+*,..iii i BII i RYY ii.  rrc!P^\rSrSrSSSSSSSSSS\R \R SSSS4S\S\\S \\ \\ \4S \S \S \S \ S\\S\S\ S\ S\ \RS\ \RS\\ \RS\\ \R4U4SjjjrSrSrU=r$)SelectiveKernel=Nr g?T in_channels out_channelsr%stridedilationgroupsrd_ratio rd_channels rd_divisorkeep_3x3 split_inputrr aa_layer drop_layerc >UUS.n[TU]5 U=(d UnU=(d SS/n[U5 [U[5(dU/S-nU (a+UVs/sHnUUS- -S-PM nnS/[ U5-nOU/[ U5-n[ U5UlXlX lXl UR(a"XR -S:XdeXR -n[X&5n[S XFXXS.UD6n[R"[X55VVs/sHunn[X4UUS.UD6PM snn5UlU=(d [#X'-U S 9n[%X R U40UD6Ulg s snfs snnf) aSelective Kernel Convolution Module As described in Selective Kernel Networks (https://arxiv.org/abs/1903.06586) with some modifications. Largest change is the input split, which divides the input channels across each convolution path, this can be viewed as a grouping of sorts, but the output channel counts expand to the module level value. This keeps the parameter count from ballooning when the convolutions themselves don't have groups, but still provides a noteworthy increase in performance over similar param count models without this attention layer. -Ross W Args: in_channels: module input (feature) channel count out_channels: module output (feature) channel count kernel_size: kernel size for each convolution branch stride: stride for convolutions dilation: dilation for module as a whole, impacts dilation of each branch groups: number of groups for each branch rd_ratio: reduction factor for attention features keep_3x3: keep all branch convolution kernels as 3x3, changing larger kernels for dilations split_input: split input channels evenly across each convolution branch, keeps param count lower, can be viewed as grouping by path, output expands to module out_channels count act_layer: activation layer to use norm_layer: batchnorm/norm layer to use aa_layer: anti-aliasing module drop_layer: spatial drop module in convs (drop block, etc) r"rrr r)rVrXrr r^r_)r%rW)divisorN)r(r)rrrlenrrTrUr]mindictr ModuleListzipr pathsr rattn)r/rTrUr%rVrWrXrYrZr[r\r]rr r^r_r#r$r0r conv_kwargsdrr1s rr)SelectiveKernel.__init__?sZ / #2{ !+aV k"+t,,&-!+K 9DEAAE*a/HE#K 00K zC $44H[)&(&   /14 44%7K\*<I<8:< ]]K2$421  \q1 \P[ \2$45 $b~l6MWa'b ' nnmZWYZ +F $4s E<!F cUR(a`[R"XRUR-S5n[ UR 5VVs/sHup4U"X#5PM nnnO!UR Vs/sH oD"U5PM nn[R"USS9nURU5nX-n[R"USS9nU$s snnfs snf)Nr r6) r]r<splitrTr enumerateristackrjr9)r/r>x_splitiopx_pathsx_attns rrCSelectiveKernel.forwards   kk!%5%5%GKG3rsI 65 $# ""))"JXbiiXr