{-jJ 2ddlmZddlmZmZmZddlZddlmcm Z ddlm Z ddl m Z ddlmZddlmZddlmZmZmZer dd lmZdd lmZe dadbdZdcdZe ddded%Ze dddfd(Ze dddgd*Z dhd,Zdid0Zdid1Ze djdddddd2dkd:Ze djdddddd2dld;Ze djdddddd2dmd<Z dndd=d ddd2d>Ze djddddd?dodAZe djddddd?dpdBZe djddddd?dqdCZ dndd=d dd?dDZe drdsdIZ e drdtdJZ e drdudKZ dvdLZ dwdOZ! dwdPZ"e dxdydRZ#e dxdzdSZ#e dxd{dTZ# d|dUZ# d}d+d dd d dd=d ddddV d~d]Z$dd`Z%dS)) annotations) TYPE_CHECKINGLiteraloverloadN)_C_ops)in_dynamic_or_pir_mode) LayerHelper)signature_safe_contextmanager) SDPBackend_get_enabled_backends sdpa_kernel) Generator)TensorFT enable_mathbool enable_flashenable_mem_efficientreturnGenerator[None, None, None]c#^Kg}|r|tj|r|tj|r|tj|st dt |5} |Vn#wxYw ddddS#1swxYwYdS)z With the sdp_kernel context manager, different algorithm implementations can be selected for scaled_dot_product_attention. z$At least one backend must be enabledN)appendr FLASH_ATTENTIONEFFICIENT_ATTENTIONMATH ValueErrorr )rrr backend_listcontexts d/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/paddle/nn/functional/flash_attention.py sdp_kernelr%s L8J6777<J:;;;-JO,,, A?@@@ \ " "g MMM DDDDD s*B" BB"BB""B&)B&xrcxtj|d}d|_tj|d}d|_|S)NgT)diagonal)paddle full_like stop_gradienttriu)r masks rget_triangle_upper_maskr)Bs=  At $ $DD ;ta ( ( (DD K.querykeyvaluer( dropout_ratefloatcausalreturn_softmaxLiteral[False]trainingscale float | Nonetuple[Tensor, None]c dSN r+r,r-r(r.r0r1r3r4s r_math_attentionr;J #r* Literal[True]tuple[Tensor, Tensor]c dSr8r9r:s rr;r;X  Cr*tuple[Tensor, Tensor | None]c dSr8r9r:s rr;r;f $'3r*c |jd} tj|gd}tj|gd}tj|gd}|p | dkr| dznd}tj||z|d} |s|| |z} t j| } ntj} d | vsBd | vs>| jdd ks-| jdd ks| jd| jd kr)t| }| |z} t j| } ntj | } |dkrt j | ||d} tj| |} tj| gd} | |r| ndfS)z This is a basic implementation of scaled dot product attention composed of combinations of fundamental components. )rr"rg?T)r y transpose_yNxpucpu i@rDupscale_in_train)r3mode) shaper$ transposematmulFsoftmax get_devicer)incubate softmax_mask_fuse_upper_triangledropout)r+r,r-r(r.r0r1r3r4head_dimproductweightsplaceouts rr;r;ts {2H  ULLL 1 1E  3 - -C  ULLL 1 1E  ?A hnn3EmeemsEEEG P  nG)G$$!## UNN~~}R 2%%}R 5((}R GM"$555+733DnGi((GGoFFwOOGc) \H;M    - ' 'C  3 - -C >3t 33r*r[intstrc|dkrdSdS)N flash_attn mem_efficientr9)r[s r_select_sdp_cudarfs3|r*c,tj}d|vrdSt}|stdtj|v}tj|v}tj|v}|dur|dur|durdSd|vrdS|dur|durt|S|durdSdS) z There are currently three different implementation options available for scaled dot product attention, and the chosen approach depends on whether it is determined by the sdp_kernel configuration or specified through input values. rLrdz@No available backend for scaled_dot_product_attention was found.TFmathgpure) r$rWr AssertionErrorr rrrrf)r[r^enabled_backendsrrrs r _select_sdprls    E ~~|,..   N   /%55K-1AAL%9=MMd 5 %9U%B%B6   6t 4 < <)))t| ?r*)fixed_seed_offsetrng_namer3name softmax_scalerZrm Tensor | Nonernro str | NonerpcdSr8r9 r+r,r-rZr0r1rmrnr3rorps rflash_attentionrus #r*cdSr8r9rts rrurus  Cr*cdSr8r9rts rrurus $'3r*c|jd} t| } | dkr|dtjvrd} nadtjvrd} nItjdgdrd} n+tjjdgd} ts| dks Jd|d ks| dks Jd |r| dks Jd || dks Jd |dks| dks Jd|s| dks Jd| | dks Jd| | dks Jdtr| dkr+tj ||||d |||| | \}}}}||r|nd fS| dkr<| |jddz} tj |||d d d d | |ddd dddd\}}|d fStd| td#it}|d}||}||}|tj}|tj}||||d}||||d}|d|||||| |d||r|nd fS| dkrdd lm}||||d |d |!}|d fSt+|||||||"S)$a> The equation is: .. math:: result=softmax(\frac{ Q * K^T }{\sqrt{d}}) * V where : ``Q``, ``K``, and ``V`` represent the three input parameters of the attention module. The dimensions of the three parameters are the same. ``d`` represents the size of the last dimension of the three parameters. Warning: This API is only support inputs with dtype float16 and bfloat16. Args: query(Tensor): The query tensor in the Attention module. 4-D tensor with shape: [batch_size, seq_len, num_heads, head_dim]. The dtype can be float16 or bfloat16. key(Tensor): The key tensor in the Attention module. 4-D tensor with shape: [batch_size, seq_len, num_heads, head_dim]. The dtype can be float16 or bfloat16. value(Tensor): The value tensor in the Attention module. 4-D tensor with shape: [batch_size, seq_len, num_heads, head_dim]. The dtype can be float16 or bfloat16. dropout(float): The dropout ratio. causal(bool): Whether enable causal mode. return_softmax(bool): Whether to return softmax. fixed_seed_offset(Tensor|None, optional): With fixed seed, offset for dropout mask. training(bool): Whether it is in the training phase. rng_name(str): The name to select Generator. name(str|None, optional): The default value is None. Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. Returns: out(Tensor): The attention tensor. 4-D tensor with shape: [batch_size, seq_len, num_heads, head_dim]. The dtype can be float16 or bfloat16. softmax(Tensor): The softmax tensor. None if return_softmax is False. Examples: .. code-block:: python >>> import paddle >>> paddle.seed(2023) >>> q = paddle.rand((1, 128, 2, 16)) >>> output = paddle.nn.functional.flash_attention.flash_attention(q, q, q, 0.9, False, False) >>> print(output) (Tensor(shape=[1, 128, 2, 16], dtype=float32, place=Place(cpu), stop_gradient=True, [[[[0.34992966, 0.34456208, 0.45826620, ..., 0.39883569, 0.42132431, 0.39157745], [0.76687670, 0.65837246, 0.69117945, ..., 0.82817286, 0.76690865, 0.71485823]], ..., [[0.71662450, 0.57275224, 0.57053083, ..., 0.48108247, 0.53336465, 0.54540104], [0.59137970, 0.51350880, 0.50449550, ..., 0.38860250, 0.40526697, 0.60541755]]]]), None) rHrdrLrG iluvatar_gpuFLAGS_cudnn_deterministicFLAGS_flash_attn_versionz2flash attention 3 only support dynamic or pir moderDz*flash attention 3 does not support dropoutz1flash attention 3 does not support return softmaxNz6flash attention 3 does not support setting seed_offsetrxz3flash attention 3 does not support setting rng_namez3flash attention 3 does not support setting trainingz/flash attention 3 does not support setting namez8flash attention 2 does not support setting softmax_scalerFrIr"Fr!Invalid flash attention version: qinput_param_name)r~kvrmr_rV softmax_lse seed_offsetrZr0r1is_testrntypeinputsoutputsattrsrememory_efficient_attention attn_biaspr4r3r.r0r1r3)rd)rRrlr$rW get_flagsbase frameworkrrrd flash_attn_v3rr locals input_dtype"create_variable_for_type_inferencefloat32int64 append_op-paddle.incubate.nn.memory_efficient_attentionrr;)r+r,r-rZr0r1rmrnr3rorpr[ sdp_func_name fa_versionresult_attentionresult_softmax_r_rhelperdtyperVrrrroutputs rrurusQ^{1~H))M $$ F%'' ' 'JJ v022 2 2JJ  :; < < '  *JJ.88+,(*J&'' :??? @,;?:#~~q 8"10" Z1___ ?&5_4!(J!OOO D-1a(&4>NN$q ($)KO$=M#)#7!!$$ [$Dy  D DD66VXX66""C"8877>>;;EBB??OO ?? MM !2   &&    " "0'<$   ~7GG477 O + +      0/!F4< "$-! r*)rmrnr3roqkvcdSr8r9rrZr0r1rmrnr3ros rflash_attn_qkvpackedrr<r*cdSr8r9rs rrrr@r*cdSr8r9rs rrrrCr*c \|jd}t|} | dkrtr)tj||d|||| |\} } } } | |r| ndfSt dit } | d}| |}| |}| tj }| tj }||d}||||d}| d|||||| |d  ||r|ndfS|dddddd f d d d|jdg}|ddddd f}|dddddf}| d krd dlm}||||d|d|}|dfSt!|||||||S)a' The equation is: .. math:: result=softmax(\frac{ Q * K^T }{\sqrt{d}}) * V where : ``Q``, ``K``, and ``V`` represent the three input parameters of the attention module. The dimensions of the three parameters are the same. ``d`` represents the size of the last dimension of the three parameters. Warning: This API only supports inputs with dtype float16 and bfloat16. Don't call this API if flash_attn is not supported. Args: qkv(Tensor): The query/key/value packed tensor in the Attention module. 5-D tensor with shape: [batchsize, seqlen , num_heads/num_heads_k + 2, num_heads_k, head_dim]. The dtype can be float16 or bfloat16. dropout(float): The dropout ratio. causal(bool): Whether enable causal mode. return_softmax(bool): Whether to return softmax. fixed_seed_offset(Tensor|None, optional): With fixed seed, offset for dropout mask. training(bool): Whether it is in the training phase. rng_name(str): The name to select Generator. name(str|None, optional): The default value is None. Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. Returns: - out(Tensor). The attention tensor. 4-D tensor with shape: [batch_size, seq_len, num_heads, head_dim]. The dtype can be float16 or bfloat16. - softmax(Tensor). The softmax tensor. None if return_softmax is False. Examples: .. code-block:: python >>> # doctest: +SKIP('flash_attn need A100 compile') >>> import paddle >>> paddle.seed(2023) >>> q = paddle.rand((1, 128, 2, 16)) >>> qkv = paddle.stack([q, q, q], axis=2) >>> output = paddle.nn.functional.flash_attn_qkvpacked(qkv, 0.9, False, False) >>> print(output) (Tensor(shape=[1, 128, 2, 16], dtype=float32, place=Place(cpu), stop_gradient=True, [[[[0.34992966, 0.34456208, 0.45826620, ..., 0.39883569, 0.42132431, 0.39157745], [0.76687670, 0.65837246, 0.69117945, ..., 0.82817286, 0.76690865, 0.71485823]], ..., [[0.71662450, 0.57275224, 0.57053083, ..., 0.48108247, 0.53336465, 0.54540104], [0.59137970, 0.51350880, 0.50449550, ..., 0.38860250, 0.40526697, 0.60541755]]]]), None) >>> # doctest: -SKIP rFrdNrrr)rrmrrrrOrrerrr)r)rRrlrrrr rrrr$rrrreshaperrr;)rrZr0r1rmrnr3ror[rrrrrrr_rVrrrrr+r,r-rrs rrrsJy}H))M $$ ! # # P +!      $~%O^^4O O@@vxx@@""E"::77>>;;EBB??OO ?? MM !2   &&    '" "0'<$   ~7GG477AAAqqq#2#I&&1b#)B-'@AA!!!QQQ(mAAAqqq"H  O + +      0/!F4< "$-! r* cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_kcdSr8r9r+r,r-rrrrr4rZr0r1rmrnr3ros rflash_attn_unpaddedrs "#r*cdSr8r9rs rrrs " Cr*cdSr8r9rs rrrs "$'3r*ctr.tj|||||| d||||| | | | \}}|| r|ndfStd it }|d}||}||}|tj}|tj }|||||| d}||||d}| d||||||| | | | d|| r|ndfS) a The equation is: .. math:: result=softmax(\frac{ Q * K^T }{\sqrt{d}}) * V where : ``Q``, ``K``, and ``V`` represent the three input parameters of the attention module. The dimensions of the three parameters are the same. ``d`` represents the size of the last dimension of the three parameters. Warning: This API is only support inputs with dtype float16 and bfloat16. Args: query(Tensor): The query tensor in the Attention module. 3-D tensor with shape: [total_seq_len, num_heads, head_dim]. The dtype can be float16 or bfloat16. key(Tensor): The key tensor in the Attention module. 3-D tensor with shape: [total_seq_len, num_heads, head_dim]. The dtype can be float16 or bfloat16. value(Tensor): The value tensor in the Attention module. 3-D tensor with shape: [total_seq_len, num_heads, head_dim]. The dtype can be float16 or bfloat16. cu_seqlens_q(Tensor): The cumulative sequence lengths of the sequences in the batch, used to index query. cu_seqlens_k(Tensor): The cumulative sequence lengths of the sequences in the batch, used to index key and value. max_seqlen_q(int): Maximum sequence length of query in the batch. max_seqlen_k(int): Maximum sequence length of key/value in the batch. scale(float): The scaling of QK^T before applying softmax. dropout(float, optional): The dropout ratio. causal(bool, optional): Whether enable causal mode. return_softmax(bool, optional): Whether to return softmax. fixed_seed_offset(Tensor|None, optional): With fixed seed, offset for dropout mask. rng_name(str, optional): The name to select Generator. training(bool, optional): Whether it is in the training phase. name(str|None, optional): The default value is None. Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. Returns: out(Tensor): The attention tensor. 3-D tensor with shape: [total_seq_len, num_heads, head_dim]. The dtype can be float16 or bfloat16. softmax(Tensor): The softmax tensor. None if return_softmax is False. Examples: .. code-block:: python >>> import paddle >>> paddle.seed(2023) >>> q = paddle.rand((2, 128, 8, 16), dtype='float16') >>> cu = paddle.arange(0, 384, 128, dtype='int32') >>> qq = paddle.reshape(q, [256, 8, 16]) >>> output = paddle.nn.functional.flash_attention.flash_attn_unpadded(qq, qq, qq, cu, cu, 128, 128, 0.25, 0.0, False, False) Nrr~r)r~rrrrrmrrrr4rZr0r1rrnr)r) rrrr rrrr$rrr)r+r,r-rrrrr4rZr0r1rmrnr3rorrrrr_rVrrrrs rrrs\L  &             L     $ >!KtKK  ; ;&(( ; ;F     4 4E  3 3E : :C77>>G;;FNKKK;;FLIIK   $$. F"" G   "((,#|     >3t 33r*rFrFr"cFt|||||||||| | | | | ||||||Sr8)flash_attn_varlen_func)r+r,r-rrrr seqused_q seqused_krpr0qv q_descale k_descale v_descale window_sizesoftcap num_splitspack_gqa sm_margins rflash_attention_v3_varlenrmsR, "    )  r*cdtjvs Jdtjdgdr Jdtjjdgddks Jdt s Jd| Jd | "|jd | | jd nd zd z} tj|||||||| | | |||| | |d |d|||d u||nd|\}}||fS)a The equation is: .. math:: result=softmax(\frac{ Q * K^T }{\sqrt{d}}) * V where : ``Q``, ``K``, and ``V`` represent the three input parameters of the attention module. The dimensions of the three parameters are the same. ``d`` represents the size of the last dimension of the three parameters. This is the varlen version of flash attention. Warning: This API is only support inputs with dtype float16 and bfloat16. Args: query(Tensor): The query tensor in the Attention module. 3-D tensor with shape: [token_num, num_heads, head_dim]. The dtype can be float16 or bfloat16. key(Tensor): The key tensor in the Attention module. 3-D tensor with shape: [token_num, num_heads, head_dim]. The dtype can be float16 or bfloat16. value(Tensor): The value tensor in the Attention module. 3-D tensor with shape: [token_num, num_heads, head_dim]. The dtype can be float16 or bfloat16. cu_seqlens_q(Tensor): The cumulative sequence lengths of the sequences in the batch, used to index query. cu_seqlens_k(Tensor): The cumulative sequence lengths of the sequences in the batch, used to index key and value. causal(bool): Whether enable causal mode. softmax_scale(float): The softmax scale of the attention. max_seqlen_q(int): Maximum sequence length of query in the batch. Note it's the padding length, not the max actual seqlen. max_seqlen_k(int): Maximum sequence length of key/value in the batch. Returns: out(Tensor): The attention tensor. 3-D tensor with shape: [token_num, num_heads, head_dim]. The dtype can be float16 or bfloat16. softmax(Tensor): The softmax tensor. None if return_softmax is False. Examples: .. code-block:: python >>> # doctest: +SKIP('flash_attn_v3 need H100 compile') >>> import paddle >>> paddle.seed(2023) >>> q = paddle.rand((10, 2, 128), dtype="bfloat16") >>> cu_seqlens_q = paddle.to_tensor([0, 10], dtype="int32") >>> max_seq_len_q = 10 >>> output = paddle.nn.functional.flash_attention.flash_attention_v3_varlen(q, q, q, cu_seqlens_q, cu_seqlens_q, max_seqlen_q=max_seq_len_q, max_seqlen_k=max_seq_len_q, causal=True) >>> # doctest: -SKIP rLz.flash_attn_varlen_func is not supported on xpur{z5flash_attn_varlen_func does not support deterministicr|rHzDFLAGS_flash_attn_version is 2, conflicts with flash_attn_varlen_funcz7flash_attn_varlen_func only support dynamic or pir modeNz2flash_attn_varlen_func does not support setting qvrFrrIr"F) r$rWrrrrrRrflash_attn_v3_varlen)r+r,r-rrrrrrrpr0rrrrrrrrrr_rs rrrsF )++ + + +8 , + +!< =>>#??>??   '')C(DEE &     N    " # #A # ::K::: KOr~rx||1 E 2    AA(e-C0  r* varlen_paddedcdSr8r9rrrrrr4rZr0r1rmrnrr3ros rflash_attn_varlen_qkvpackedrs #r*cdSr8r9rs rrr&s  Cr*cdSr8r9rs rrr9s $'3r*ctr-tj|||| d||||||| | | \}}||r|ndfStd it }|d}||}||}|tj}|tj }|||| d}||||d}| d||||||||| | d||r|ndfS) a The equation is: .. math:: result=softmax(\frac{ Q * K^T }{\sqrt{d}}) * V where : ``Q``, ``K``, and ``V`` represent the three input parameters of the attention module. The dimensions of the three parameters are the same. ``d`` represents the size of the last dimension of the three parameters. Warning: This API only supports inputs with dtype float16 and bfloat16. Args: qkv(Tensor): The padded query/key/value packed tensor in the Attention module. The padding part won't be computed 4-D tensor with shape: [total_seq_len, num_heads/num_heads_k + 2, num_heads_k, head_dim]. The dtype can be float16 or bfloat16. cu_seqlens_q(Tensor): The cumulative sequence lengths of the sequences in the batch, used to index query. cu_seqlens_k(Tensor): The cumulative sequence lengths of the sequences in the batch, used to index key and value. max_seqlen_q(int): Maximum sequence length of query in the batch. Note it's the padding length, not the max actual seqlen max_seqlen_k(int): Maximum sequence length of key/value in the batch. scale(float): The scaling of QK^T before applying softmax. dropout(float, optional): The dropout ratio. causal(bool, optional): Whether enable causal mode. return_softmax(bool, optional): Whether to return softmax. fixed_seed_offset(Tensor|None, optional): With fixed seed, offset for dropout mask. rng_name(str, optional): The name to select Generator. training(bool, optional): Whether it is in the training phase. name(str|None, optional): The default value is None. Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. Returns: - out(Tensor). The attention tensor. The tensor is padded by zeros. 3-D tensor with shape: [total_seq_len, num_heads, head_dim]. The dtype can be float16 or bfloat16. - softmax(Tensor). The softmax tensor. None if return_softmax is False. Examples: .. code-block:: python >>> # doctest: +SKIP('flash_attn need A100 compile') >>> import paddle >>> paddle.seed(2023) >>> q = paddle.rand((2, 128, 8, 16), dtype='float16') >>> cu = paddle.arange(0, 384, 128, dtype='int32') >>> qq = paddle.reshape(q, [256, 8, 16]) >>> qkv = paddle.stack([qq, qq, qq], axis=2) >>> output = paddle.nn.functional.flash_attn_varlen_qkvpacked(qkv, cu, cu, 128, 128, 0.25, 0.0, False, False) >>> # doctest: -SKIP Nrrr)rrrrmrrr)r) rrrr rrrr$rrr)rrrrrr4rZr0r1rmrnrr3rorrrrr_rVrrrrs rrrLsLL  .           L      " >!KtKK  C C&(( C CF     6 6E  3 3E : :C77>>G;;FNKKK;;FLIIK$$. F"" G   *((,#|     >3t 33r*) rZr0rreturn_softmax_lsereturn_seed_offsetrmrnr3rorp block_maskstartend_row_indicesrint | tuple | Nonerrrc z |Yt|tr||f}|jd}|jd}| Jd|rntj|ddz||dzdzddd|df}tj|||d}ntjdd|dfd}tj|ddz||dzdzd|dddddf<tj|d ||dz d|dddddf<tj|d| |d}| | Jd |%tj |||| d||d | | \}}}}n{|j tj ksJd |j t|jd ksJd|j|jd|jdks&Jd|jdd|jd|jd|jdks&Jd|jdd|jd|jdd|jd|jdfvs Jd||j tj ksJd|j |jd|jdks&Jd|jdd|jd|jd|jdks&Jd|jdd|jd|jd|jddzdzks Jd|jd|jddzdzks Jd|jddks Jd|rE|jddkrd }nv|jddkrd}nbtd|jd|jddkrd }n1|jdd krd}ntd|jddtjvr'tjd gd r | Jd!dtjvrd}ntjjd"gd"dkr?tjjd gd r|jddkrd}n+tjjd"gd"}|dkr<| Jd#| Jd$tj||||| ||d | | \}}}}n|dkr|d%ks Jd&|r Jd'| Jd(| d)ks Jd*| s Jd+| Jd,| |jdd-z} tj|||||| |\}}ntd.||g}|r||gz }|r||gz }t|dkr|dS|S)/a FlashMask: Official Implementation This module provides the official implementation of the FlashMask algorithm as described in the paper. For more details, please refer to the paper available at: https://arxiv.org/abs/2410.01359. The core equation utilized in FlashMask is as follows: .. math:: \text{result} = \text{softmax}\left(\frac{Q \cdot K^T}{\sqrt{d}} + M\right) \cdot V In this equation: - ``Q``, ``K``, and ``V`` are the input tensors to the attention module. - All these tensors share the same dimensions. - ``d`` denotes the size of the last dimension of these tensors. - ``M`` represents the column-wise sparse mask introduced by FlashMask. Args: query (Tensor): The query tensor in the attention module. A 4-D tensor with shape [batch_size, q_seq_len, num_heads, head_dim]. The dtype can be float16 or bfloat16. key (Tensor): The key tensor in the attention module. A 4-D tensor with shape [batch_size, k_seq_len, k_num_heads, head_dim]. The dtype can be float16 or bfloat16. value (Tensor): The value tensor in the attention module. A 4-D tensor with shape [batch_size, k_seq_len, k_num_heads, head_dim]. The dtype can be float16 or bfloat16. startend_row_indices(Tensor): A column-wise sparse attention mask row indices tensor. A 4-D tensor with shape [batch_size, k_num_heads, k_seq_len, {1, 2, 4}]. The dtype must be int32. k_num_heads can be 1 or the same as key's num_heads. When num_heads is 1, it will be broadcast to match key's num_heads. Depending on the value of the causal parameter, startend_row_indices can take different shapes and meanings. - When `causal=True` and the shape is [batch_size, k_num_heads, k_seq_len, 1], indicating unidirectional attention. The value represents the starting row index of the left lower triangular mask in the dense mask. The value startend_row_indices[..., 0] indicates that elements in the lower left triangle of the attention score matrix starting from the startend_row_indices[..., 0]-th row downwards (inclusive) will be masked. - When `causal=True` and the shape is [batch_size, k_num_heads, k_seq_len, 2], indicating unidirectional attention. The values represent the starting and ending row indices of the left lower triangular mask in the dense mask. The values startend_row_indices[..., 0:2] in startend_row_indices indicate that elements in the lower left triangle of the attention score matrix starting from the startend_row_indices[..., 0]-th row downwards (inclusive) but above the startend_row_indices[..., 1]-th row (exclusive) will be masked. - When `causal=False` and the shape is [batch_size, k_num_heads, k_seq_len, 2], indicating bidirectional attention. The values represent the starting row index of the left lower triangular mask and the ending row index of the right upper triangular mask in the dense mask. The values startend_row_indices[..., 0:2] in startend_row_indices indicate that elements in the lower left triangle of the attention score matrix starting from the startend_row_indices[..., 0]-th row downwards (inclusive) will be masked, and elements in the upper right triangle starting from the startend_row_indices[..., 1]-th row upwards (exclusive) will be masked. - When `causal=False` and the shape is [batch_size, k_num_heads, k_seq_len, 4] , indicating bidirectional attention. The values represent the start and end row indices of the left lower triangular mask and the start and end row indices of the right upper triangular mask in the dense mask. The values startend_row_indices[..., 0:4] in startend_row_indices indicate that elements in the lower left triangle of the attention score matrix starting from the startend_row_indices[..., 0]-th row downwards (inclusive) but above the startend_row_indices[..., 1] row (exclusive) will be masked, and elements in the upper right triangle starting from the startend_row_indices[..., 2]-th row downwards (inclusive) but above the startend_row_indices[..., 3] row (exclusive) will be masked. dropout (float): The dropout ratio. Default is 0.0. causal (bool): Whether to enable causal mode. Default is False. window_size (int|tuple, optional): Indicates the window size of sliding window local attention. If causal mode is enabled, Query at position i will only attend to keys between [i - window_size, i] or [i - window_size[0], i]. If causal mode is disabled, Query at position i will only attend to keys between [i - window_size, i + window_size] or [i - window_size[0], i + window_size[1]]. return_softmax_lse (bool): Whether to return the log-sum-exp of the softmax. Default is False. return_seed_offset (bool): Whether to return the random seed offset. Default is False. fixed_seed_offset(Tensor, optional): With fixed seed, offset for dropout mask. rng_name (str): The name to select Generator. training (bool): Whether the module is in training mode. Default is True. name (str, optional): Name of the operation. Default is None. Normally, users do not need to set this property. For more information, refer to :ref:`api_guide_Name` . block_mask (tensor, optional): A 4-D integer mask tensor indicating whether each block in the attention matrix should be kept or masked. Must be used together with flashmask. The shape should be [batch_size, num_heads, blocklen_q, blocklen_k], where: blocklen_q = ceil(seqlen_q / 128), i.e., block_mask.shape[2] must be (seqlen_q + 127) // 128 blocklen_k = ceil(seqlen_k / 128), i.e., block_mask.shape[3] must be (seqlen_k + 127) // 128 block_mask.shape[1] (number of heads) must match the num_heads dimension of the flashmask Both seqlen_q and seqlen_k must be less than or equal to 128 * 1024 The dtype should be int32, and each element should be either 0 or 1. A value of 1 indicates that the corresponding block is kept (not masked), while 0 means the block is masked. Usage Notes: Only supported when blockdim_q = blockdim_k = 128 now. Only supported when headdim = 128 now. This argument must be provided together with flashmask. The mask will be applied at the block level: each [i, j] position in block_mask controls whether the corresponding [128 x 128] block in the attention matrix is masked. Any mismatch in expected shape or head dimension will raise an error. Returns Tensor. The computed attention result with the same shape as the input `query`. Warning: This API only supports inputs with dtype float16 and bfloat16. Hint: This API supports GQA. Examples: .. code-block:: python >>> # doctest: +SKIP('flash_attn need A100 compile') >>> import paddle >>> paddle.seed(2023) >>> q = paddle.rand((1, 10, 2, 32),dtype="bfloat16") # shape: [batch_size, seq_len, num_heads, head_dim] >>> k = paddle.rand((1, 10, 2, 32),dtype="bfloat16") # shape: [batch_size, seq_len, num_heads, head_dim] >>> v = paddle.rand((1, 10, 2, 32),dtype="bfloat16") # shape: [batch_size, seq_len, num_heads, head_dim] >>> startend_row_indices = paddle.to_tensor([8]*10 + [5]*10, dtype="int32").reshape([1, 2, 10, 1]) >>> output = paddle.nn.functional.flashmask_attention(q, k, v, startend_row_indices, causal=True) >>> print(output) Tensor(shape=[1, 10, 2, 32], dtype=bfloat16, place=Place(gpu:0), stop_gradient=True, [[[[0.82421875, 0.27539062, 0.80859375, 0.98046875, 0.00251770, 0.41992188, 0.17285156, 0.11767578, 0.42773438, 0.31250000, 0.34570312, 0.70312500, 0.29296875, 0.44531250, 0.51562500, 0.96093750, 0.85546875, 0.15625000, 0.34765625, 0.98437500, 0.96484375, 0.45312500, 0.33593750, 0.56640625, 0.07714844, 0.43750000, 0.83984375, 0.66796875, 0.93750000, 0.24804688, 0.51171875, 0.55468750], [0.54687500, 0.74609375, 0.43164062, 0.32421875, 0.10693359, 0.37304688, 0.53906250, 0.17187500, 0.57421875, 0.75000000, 0.13378906, 0.57031250, 0.19531250, 0.01403809, 0.29101562, 0.14257812, 0.07568359, 0.88671875, 0.75390625, 0.17089844, 0.87109375, 0.93359375, 0.89843750, 0.58203125, 0.75390625, 0.27539062, 0.67968750, 0.24804688, 0.57812500, 0.67578125, 0.92578125, 0.98046875]], [[0.59765625, 0.62890625, 0.62109375, 0.75781250, 0.03295898, 0.64062500, 0.27929688, 0.20800781, 0.72265625, 0.52343750, 0.53125000, 0.61718750, 0.57421875, 0.56640625, 0.65625000, 0.48242188, 0.68359375, 0.42968750, 0.26562500, 0.86718750, 0.83203125, 0.40820312, 0.38281250, 0.59765625, 0.43945312, 0.22851562, 0.86328125, 0.51562500, 0.89453125, 0.62500000, 0.50390625, 0.67968750], [0.34765625, 0.61328125, 0.58593750, 0.60156250, 0.43164062, 0.41601562, 0.71093750, 0.59765625, 0.53515625, 0.78125000, 0.13867188, 0.30664062, 0.48828125, 0.04394531, 0.24316406, 0.18847656, 0.10644531, 0.71093750, 0.69140625, 0.35937500, 0.44531250, 0.81640625, 0.44140625, 0.64062500, 0.81640625, 0.61328125, 0.72265625, 0.53125000, 0.49414062, 0.59765625, 0.54296875, 0.61328125]], [[0.65234375, 0.47656250, 0.71875000, 0.64843750, 0.23828125, 0.61328125, 0.29101562, 0.26562500, 0.54296875, 0.60937500, 0.67187500, 0.67578125, 0.64062500, 0.41406250, 0.47656250, 0.40820312, 0.66406250, 0.39453125, 0.39453125, 0.62109375, 0.58593750, 0.31054688, 0.31835938, 0.45703125, 0.52343750, 0.43164062, 0.64453125, 0.49804688, 0.82812500, 0.48242188, 0.38476562, 0.59375000], [0.44921875, 0.62109375, 0.50390625, 0.51562500, 0.51953125, 0.57812500, 0.78515625, 0.73437500, 0.60546875, 0.55078125, 0.30273438, 0.23339844, 0.60546875, 0.33007812, 0.23242188, 0.30468750, 0.34570312, 0.70703125, 0.72656250, 0.58593750, 0.40234375, 0.62109375, 0.62109375, 0.69531250, 0.66796875, 0.51562500, 0.45898438, 0.67968750, 0.48828125, 0.50000000, 0.54687500, 0.71875000]], [[0.67578125, 0.50000000, 0.58203125, 0.62109375, 0.43554688, 0.69531250, 0.30273438, 0.24023438, 0.57812500, 0.63671875, 0.51171875, 0.52734375, 0.60546875, 0.45507812, 0.42382812, 0.46093750, 0.55859375, 0.34960938, 0.39453125, 0.57031250, 0.55078125, 0.47265625, 0.24609375, 0.51953125, 0.46093750, 0.49218750, 0.49609375, 0.60156250, 0.76953125, 0.57421875, 0.40429688, 0.57031250], [0.45703125, 0.71093750, 0.58984375, 0.43164062, 0.54296875, 0.57031250, 0.72265625, 0.61328125, 0.64453125, 0.50781250, 0.28125000, 0.19531250, 0.60546875, 0.40625000, 0.18554688, 0.33203125, 0.40039062, 0.58593750, 0.79687500, 0.45507812, 0.32812500, 0.58203125, 0.70703125, 0.64453125, 0.53906250, 0.57421875, 0.48828125, 0.53515625, 0.49804688, 0.50000000, 0.48437500, 0.55468750]], [[0.64453125, 0.43164062, 0.54687500, 0.53125000, 0.42187500, 0.71484375, 0.30273438, 0.21484375, 0.50390625, 0.69531250, 0.58203125, 0.51562500, 0.61328125, 0.41992188, 0.40039062, 0.46679688, 0.58984375, 0.39062500, 0.41992188, 0.49023438, 0.47851562, 0.47070312, 0.30078125, 0.50390625, 0.47656250, 0.44921875, 0.43164062, 0.63671875, 0.78125000, 0.60156250, 0.48242188, 0.58203125], [0.52343750, 0.69921875, 0.58984375, 0.35156250, 0.49218750, 0.58593750, 0.71093750, 0.59375000, 0.66406250, 0.49414062, 0.24023438, 0.18554688, 0.66796875, 0.50000000, 0.23144531, 0.29882812, 0.49414062, 0.57031250, 0.70312500, 0.42773438, 0.35351562, 0.47460938, 0.73437500, 0.53125000, 0.47070312, 0.49609375, 0.50000000, 0.55078125, 0.50000000, 0.45898438, 0.45703125, 0.61328125]], [[0.63671875, 0.41210938, 0.52734375, 0.56640625, 0.44531250, 0.64843750, 0.37890625, 0.31250000, 0.56640625, 0.62890625, 0.53125000, 0.51562500, 0.54296875, 0.50781250, 0.35546875, 0.41601562, 0.55468750, 0.36914062, 0.35937500, 0.45117188, 0.46875000, 0.49609375, 0.28710938, 0.50000000, 0.49609375, 0.50000000, 0.51562500, 0.57031250, 0.77734375, 0.62109375, 0.43164062, 0.50781250], [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]], [[0.62109375, 0.44531250, 0.46875000, 0.61328125, 0.39062500, 0.60156250, 0.41015625, 0.28710938, 0.58984375, 0.67968750, 0.55859375, 0.48632812, 0.51562500, 0.42382812, 0.37695312, 0.46679688, 0.54687500, 0.44921875, 0.33789062, 0.36328125, 0.49023438, 0.44140625, 0.25000000, 0.45312500, 0.43945312, 0.45507812, 0.46679688, 0.57812500, 0.65625000, 0.64062500, 0.42382812, 0.57031250], [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]], [[0.62500000, 0.47070312, 0.51562500, 0.61328125, 0.36718750, 0.66406250, 0.37890625, 0.28320312, 0.65625000, 0.66015625, 0.48632812, 0.53906250, 0.46679688, 0.47851562, 0.43359375, 0.45703125, 0.47070312, 0.39843750, 0.32617188, 0.37304688, 0.49023438, 0.50390625, 0.27148438, 0.46679688, 0.37695312, 0.49023438, 0.47265625, 0.58593750, 0.64453125, 0.60156250, 0.38476562, 0.62109375], [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]], [[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ], [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]], [[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ], [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]]]]) >>> # doctest: -SKIP To convert FlashMask's `startend_row_indices` to `dense_mask`, use the code below: .. code-block:: python >>> import paddle >>> import numpy as np >>> def flashmask_to_densemask(startend_row_indices, dtype, causal=True): ... if startend_row_indices is None: ... return None ... bz, num_head, seq_len, bound_num = startend_row_indices.shape ... m = paddle.zeros((bz, num_head, seq_len, seq_len), dtype=dtype) ... has_end = (causal and bound_num == 2) or ((not causal) and bound_num == 4) ... for bi in range(bz): ... for hi in range(num_head): ... for j in range(seq_len): ... downstart = startend_row_indices[bi, hi, j, 0] ... if has_end: ... downend = startend_row_indices[bi, hi, j, 1] ... m[bi, hi, downstart:downend, j] = -np.inf ... else: ... m[bi, hi, downstart:, j] = -np.inf ... if causal: ... m[bi, hi, :j, j] = -np.inf ... else: ... if has_end: ... upstart = startend_row_indices[bi, hi, j, 2] ... upend = startend_row_indices[bi, hi, j, 3] ... m[bi, hi, upstart:upend, j] = -np.inf ... else: ... upend = startend_row_indices[bi, hi, j, 1] ... m[bi, hi, :upend, j] = -np.inf ... return m For `Causal Mask`, where `causal=True`, the values of `startend_row_indices` are as follows: .. code-block:: python [[[[1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]]]) >>> # doctest: +SKIP('Only example') >>> import paddle >>> startend_row_indices = paddle.to_tensor([8]*10, dtype="int32").reshape([1, 1, 10, 1]) >>> print(startend_row_indices) Tensor(shape=[1, 1, 10, 1], dtype=int32, place=Place(gpu:0), stop_gradient=True, [[[[8], [8], [8], [8], [8], [8], [8], [8], [8], [8]]]]) >>> # doctest: -SKIP For `Sliding Window Mask`, where `causal=True`, the values of `startend_row_indices` are as follows: .. code-block:: python [[[[1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [0, 1, 1, 1, 0, 0, 0, 0, 0, 0], [0, 0, 1, 1, 1, 0, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [0, 0, 0, 0, 1, 1, 1, 0, 0, 0], [0, 0, 0, 0, 0, 1, 1, 1, 0, 0], [0, 0, 0, 0, 0, 0, 1, 1, 1, 0], [0, 0, 0, 0, 0, 0, 0, 1, 1, 1]]]]) >>> # doctest: +SKIP('Only example') >>> import paddle >>> startend_row_indices = paddle.to_tensor([3, 4, 5, 6, 7, 8, 9, 10, 10, 10], dtype="int32").reshape([1, 1, 10, 1]) >>> print(startend_row_indices) Tensor(shape=[1, 1, 10, 1], dtype=int32, place=Place(gpu:0), stop_gradient=True, [[[[3 ], [4 ], [5 ], [6 ], [7 ], [8 ], [9 ], [10], [10], [10]]]]) >>> # doctest: -SKIP For `Causal Document Mask`, where `causal=True`, the values of `startend_row_indices` are as follows: .. code-block:: python [[[[1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 1, 0, 0, 0, 0, 0], [0, 0, 0, 0, 1, 1, 0, 0, 0, 0], [0, 0, 0, 0, 1, 1, 1, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 1, 0, 0], [0, 0, 0, 0, 0, 0, 0, 1, 1, 0], [0, 0, 0, 0, 0, 0, 0, 1, 1, 1]]]]) >>> # doctest: +SKIP('Only example') >>> import paddle >>> startend_row_indices = paddle.to_tensor([4, 4, 4, 4, 7, 7, 7, 10, 10, 10], dtype="int32").reshape([1, 1, 10, 1]) >>> print(startend_row_indices) Tensor(shape=[1, 1, 10, 1], dtype=int32, place=Place(gpu:0), stop_gradient=True, [[[[4 ], [4 ], [4 ], [4 ], [7 ], [7 ], [7 ], [10], [10], [10]]]]) >>> # doctest: -SKIP For `Document Mask`, where `causal=False`, the values of `startend_row_indices` are as follows: .. code-block:: python [[[[1, 1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 1, 1, 1, 0, 0, 0], [0, 0, 0, 0, 1, 1, 1, 0, 0, 0], [0, 0, 0, 0, 1, 1, 1, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 1, 1, 1], [0, 0, 0, 0, 0, 0, 0, 1, 1, 1], [0, 0, 0, 0, 0, 0, 0, 1, 1, 1]]]]) >>> # doctest: +SKIP('Only example') >>> import paddle >>> LTS = paddle.to_tensor([4, 4, 4, 4, 7, 7, 7, 10, 10, 10], dtype="int32").reshape([1, 1, 10, 1]) >>> UTE = paddle.to_tensor([0, 0, 0, 0, 4, 4, 4, 7, 7, 7], dtype="int32").reshape([1, 1, 10, 1]) >>> startend_row_indices = paddle.concat([LTS, UTE], axis=-1) >>> print(startend_row_indices) Tensor(shape=[1, 1, 10, 2], dtype=int32, place=Place(gpu:0), stop_gradient=True, [[[[4 , 0 ], [4 , 0 ], [4 , 0 ], [4 , 0 ], [7 , 4 ], [7 , 4 ], [7 , 4 ], [10, 7 ], [10, 7 ], [10, 7 ]]]]) >>> # doctest: -SKIP For `Share Question Mask`, where `causal=True`, the values of `startend_row_indices` are as follows: .. code-block:: python [[[[1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 0, 0, 0], [1, 1, 1, 1, 0, 0, 0, 1, 0, 0], [1, 1, 1, 1, 0, 0, 0, 1, 1, 0], [1, 1, 1, 1, 0, 0, 0, 1, 1, 1]]]]) >>> # doctest: +SKIP('Only example') >>> import paddle >>> startend_row_indices = paddle.to_tensor([10, 10, 10, 10, 7, 7, 7, 10, 10, 10], dtype="int32").reshape([1, 1, 10, 1]) >>> print(startend_row_indices) Tensor(shape=[1, 1, 10, 1], dtype=int32, place=Place(gpu:0), stop_gradient=True, [[[[10], [10], [10], [10], [7 ], [7 ], [7 ], [10], [10], [10]]]]) >>> # doctest: -SKIP For `Global + Sliding Window Mask`, where `causal=False`, the values of `startend_row_indices` are as follows: .. code-block:: python >>> # doctest: +SKIP('Only example') [[[[1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 1, 0, 0, 0, 0], [1, 1, 0, 0, 1, 1, 1, 0, 0, 0], [1, 1, 0, 0, 0, 1, 1, 1, 0, 0], [1, 1, 0, 0, 0, 0, 1, 1, 1, 0], [1, 1, 0, 0, 0, 0, 0, 1, 1, 1], [1, 1, 0, 0, 0, 0, 0, 0, 1, 1]]]]) >>> import paddle >>> LTS = paddle.to_tensor([10, 10, 4, 5, 6, 7, 8, 9, 10, 10], dtype="int32").reshape([1, 1, 10, 1]) >>> LTE = paddle.to_tensor([10, 10, 10, 10, 10, 10, 10, 10, 10, 10], dtype="int32").reshape([1, 1, 10, 1]) >>> UTS = paddle.to_tensor([0, 0, 0, 0, 2, 2, 2, 2, 2, 2], dtype="int32").reshape([1, 1, 10, 1]) >>> UTE = paddle.to_tensor([0, 0, 0, 0, 3, 4, 5, 6, 7, 8], dtype="int32").reshape([1, 1, 10, 1]) >>> startend_row_indices = paddle.concat([LTS, LTE, UTS, UTE], axis=-1) >>> print(startend_row_indices) Tensor(shape=[1, 1, 10, 4], dtype=int32, place=Place(gpu:0), stop_gradient=True, [[[[10, 10, 0 , 0 ], [10, 10, 0 , 0 ], [4 , 10, 0 , 0 ], [5 , 10, 0 , 0 ], [6 , 10, 2 , 3 ], [7 , 10, 2 , 4 ], [8 , 10, 2 , 5 ], [9 , 10, 2 , 6 ], [10, 10, 2 , 7 ], [10, 10, 2 , 8 ]]]]) >>> # doctest: -SKIP For `Causal Blockwise Mask`, where `causal=True`, the values of `startend_row_indices` are as follows: .. code-block:: python [[[[1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 1, 0, 0, 0, 0, 0], [0, 0, 0, 0, 1, 1, 0, 0, 0, 0], [0, 0, 0, 0, 1, 1, 1, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]]]) >>> # doctest: +SKIP('Only example') >>> import paddle >>> LTS = paddle.to_tensor([4, 4, 4, 4, 10, 10, 10, 10, 10, 10], dtype="int32").reshape([1, 1, 10, 1]) >>> LTE = paddle.to_tensor([7, 7, 7, 7, 10, 10, 10, 10, 10, 10], dtype="int32").reshape([1, 1, 10, 1]) >>> startend_row_indices = paddle.concat([LTS, LTE], axis=-1) >>> print(startend_row_indices) Tensor(shape=[1, 1, 10, 2], dtype=int32, place=Place(gpu:0), stop_gradient=True, [[[[4 , 7 ], [4 , 7 ], [4 , 7 ], [4 , 7 ], [10, 10], [10, 10], [10, 10], [10, 10], [10, 10], [10, 10]]]]) >>> # doctest: -SKIP For `Prefix LM Document Mask`, where `causal=False`, the values of `startend_row_indices` are as follows: .. code-block:: python [[[[1, 1, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 1, 0, 0, 0, 0, 0], [0, 0, 0, 1, 1, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1, 1, 0, 0, 0], [0, 0, 0, 0, 0, 1, 1, 0, 0, 0], [0, 0, 0, 0, 0, 1, 1, 1, 0, 0], [0, 0, 0, 0, 0, 1, 1, 1, 1, 0], [0, 0, 0, 0, 0, 1, 1, 1, 1, 1]]]]) >>> # doctest: +SKIP('Only example') >>> import paddle >>> LTS = paddle.to_tensor([3, 3, 3, 5, 5, 10, 10, 10, 10, 10], dtype="int32").reshape([1, 1, 10, 1]) >>> UTE = paddle.to_tensor([0, 0, 2, 3, 3, 5, 5, 7, 8, 9], dtype="int32").reshape([1, 1, 10, 1]) >>> startend_row_indices = paddle.concat([LTS, UTE], axis=-1) >>> print(startend_row_indices) Tensor(shape=[1, 1, 10, 2], dtype=int32, place=Place(gpu:0), stop_gradient=True, [[[[3 , 0 ], [3 , 0 ], [3 , 2 ], [5 , 3 ], [5 , 3 ], [10, 5 ], [10, 5 ], [10, 7 ], [10, 8 ], [10, 9 ]]]]) >>> # doctest: -SKIP For `Prefix LM Causal Mask`, where `causal=False`, the values of `startend_row_indices` are as follows: .. code-block:: python [[[[1, 1, 1, 1, 1, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]]]) >>> # doctest: +SKIP('Only example') >>> import paddle >>> LTS = paddle.to_tensor([10, 10, 10, 10, 10, 10, 10, 10, 10, 10], dtype="int32").reshape([1, 1, 10, 1]) >>> UTE = paddle.to_tensor([0, 0, 0, 0, 0, 5, 6, 7, 8, 9], dtype="int32").reshape([1, 1, 10, 1]) >>> startend_row_indices = paddle.concat([LTS, UTE], axis=-1) >>> print(startend_row_indices) Tensor(shape=[1, 1, 10, 2], dtype=int32, place=Place(gpu:0), stop_gradient=True, [[[[10, 0 ], [10, 0 ], [10, 0 ], [10, 0 ], [10, 0 ], [10, 5 ], [10, 6 ], [10, 7 ], [10, 8 ], [10, 9 ]]]]) For `QK-sparse Mask`, where `causal=True`, the values of `startend_row_indices` are as follows: .. code-block:: python [[[[1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]]]) >>> # doctest: +SKIP('Only example') >>> import paddle >>> LTS = paddle.to_tensor([10, 10, 2, 3, 4, 5, 6, 7, 10, 10], dtype="int32").reshape([1, 1, 10, 1]) >>> LTE = paddle.to_tensor([10, 10, 5, 5, 5, 5, 8, 8, 10, 10], dtype="int32").reshape([1, 1, 10, 1]) >>> startend_row_indices = paddle.concat([LTS, LTE], axis=-1) >>> print(startend_row_indices) Tensor(shape=[1, 1, 10, 2], dtype=int32, place=Place(gpu:0), stop_gradient=True, [[[[10, 10], [10, 10], [2 , 5 ], [3 , 5 ], [4 , 5 ], [5 , 5 ], [6 , 8 ], [7 , 8 ], [10, 10], [10, 10]]]]) >>> # doctest: -SKIP Nr"rz/can't use window_size with startend_row_indicesint32)r)maxrG)minrz= 90 now.rDz/flashmask_attention_v2 does not support dropoutz:flashmask_attention_v2 does not support return seed_offsetz;flashmask_attention_v2 does not support setting seed_offsetrxz8flashmask_attention_v2 does not support setting rng_namezAflashmask_attention_v2 does not support setting training to Falsez4flashmask_attention_v2 does not support setting namerIr}) isinstancer`rRr$arangercliprepeat_interleaveemptyrrdrrlenrrWrrrflashmask_attentionflashmask_attention_v2)r+r,r-rrZr0rrrrmrnr3rorprsqbszr_rresult_softmax_lseresult_seed_offsethas_endrrs rrrs@ k3 ' ' 5& 4K [^k!n#++ =,++  (#)=A"BQ$7!$;7$$$gq!Rm$$ !$*;$"$$$Q'' ! $*<Ar1 W#M#M#M /5}A"BQ$7!$;7000 Aqqq! ,06}Qk!n!4G000 Aqqq! ,$*;$!$$$Q'' !#// J0//#          L       $)V\999 dH\Hb d d:99'-..!333 W;O;U W W433$)!, ! <<< ERfRlmnRo E EvyvABwC E E=<<$)!, ! <<< CPdPjklPm C Ctwt}~uA C C=<<$)!, KN IaL1    g      !#v|333T*BRTT433#A&#)A,666tJL\]^L_ttfifopqfrtt766#A&*>*DQ*GGGGH_i_opq_rHHy|zCDEzFHHHGG#A&5;q>C+?C*GGGGCHGG#A&39Q<#+=#*EEEECFEE9Q<3&&&@'&&  #)"-22%+B/144 gG[GabdGegg$)"-22%+B/144 hH\HbceHfhh *,, , , "=!>??+ - %%F&%% F%'' ' 'JJ K ! + +-G,H I I*    %//1L0MNN+   A$$JJ.88+,(*J ?? ((p)((%%B&%%*$!     """1__c>>>A">>*  L  )%,,M-,,r>>>J">>  S  8<<F <<$ % BD 9 -$ ""MMMNN NeG(&''(&'' 7||qqzr* paddle.Tensorrcr|jr|js Jdtrtj|||}|St dit }|tj}|tj}|||d}d|i}| d|||S)aj The equation is: .. math:: result=reduce\_sum(softmax(\frac{ Q * K^T }{\sqrt{d}}), dim=-2) Warning: This API only supports inputs with dtype float16 and bfloat16. Args: query(Tensor): The query tensor in the Attention module. 4-D tensor with shape: [batch_size, seqlen_q, num_heads, head_dim]. The dtype can be float16 or bfloat16. key(Tensor): The key tensor in the Attention module. 4-D tensor with shape: [batch_size, seqlen_k, num_heads, head_dim]. The dtype can be float16 or bfloat16. softmax_lse(Tensor): The logsumexp of each row returned by _C_ops.flash_attn(). 3-D tensor with shape: [batch_size, num_heads, seqlen_q_rounded], where seqlen_q_rounded = ceil(seqlen_q/128). The dtype is float32. Returns: reduced_attention_scores(Tensor), The reduce sum of attention scores across seqlen_q. 4-D tensor with shape: [batch_size, num_heads, 1, seqlen_k]. The dtype is float32. Examples: .. code-block:: python >>> # doctest: +SKIP('reduce_attn_scores need A100 compile') >>> import paddle >>> import numpy as np >>> import paddle._C_ops as _C_ops >>> from paddle.nn.functional.flash_attention import ( >>> calc_reduced_attention_scores >>> ) >>> np.random.seed(2024) >>> q_shape = (5,1024,16,128) >>> k_shape = (5,2048,16,128) >>> dtype = 'float16' >>> query = np.random.random(q_shape) >>> key = np.random.random(k_shape) >>> q = paddle.to_tensor( >>> query, place=place, dtype=dtype, stop_gradient=True >>> ) >>> k = paddle.to_tensor( >>> key, place=place, dtype=dtype, stop_gradient=True >>> ) >>> _, _, softmax_lse, _ = _C_ops.flash_attn( >>> q, >>> k, >>> k, >>> (None,), #fixed_seed_offset >>> None, #attn_mask >>> 0.0, #dropout >>> False, #causal >>> False, #return_softmax >>> False, #is_test >>> "" #rng_name >>> ) >>> reduced_attn_scores = calc_reduced_attention_scores( >>> q, >>> k, >>> softmax_lse, >>> ) >>> # doctest: -SKIP z6calc_reduced_attention_scores() is for inference only.calc_reduced_attn_scores)r~rrreduced_scores)rrr)r) r&rrrr rrr$rr)r+r,rrrrVrrs rcalc_reduced_attention_scoresr-sL  3#4@ 48 3     @ @vxx @ @F>>v~NNN77GGG  "F .G  ' r*)FTT)rrrrrrrr)r rrr).....)r+rr,rr-rr(rr.r/r0rr1r2r3rr4r5rr6)r+rr,rr-rr(rr.r/r0rr1r=r3rr4r5rr>)r+rr,rr-rr(rr.r/r0rr1rr3rr4r5rrA)NrDFFTN)r[r`rra)...)r+rr,rr-rrZr/r0rr1r2rmrqrnrar3rrorrrpr5rr6)r+rr,rr-rrZr/r0rr1r=rmrqrnrar3rrorrrpr5rr>)r+rr,rr-rrZr/r0rr1rrmrqrnrar3rrorrrpr5rrA)rDFF)rrrZr/r0rr1r2rmrqrnrar3rrorrrr6)rrrZr/r0rr1r=rmrqrnrar3rrorrrr>)rrrZr/r0rr1rrmrqrnrar3rrorrrrA).......) r+rr,rr-rrrrrrr`rr`r4r/rZr/r0rr1r2rmrqrnrar3rrorrrr6) r+rr,rr-rrrrrrr`rr`r4r/rZr/r0rr1r=rmrqrnrar3rrorrrr>) r+rr,rr-rrrrrrr`rr`r4r/rZr/r0rr1rrmrqrnrar3rrorrrrA)rDFFNrxTN) NNNFNNNNrrDr"Nr)........)rrrrrrrr`rr`r4r/rZr/r0rr1r2rmrqrnrarrr3rrorrrr6)rrrrrrrr`rr`r4r/rZr/r0rr1r=rmrqrnrarrr3rrorrrr>)rrrrrrrr`rr`r4r/rZr/r0rr1rrmrqrnrarrr3rrorrrrA)rDFFNrxTTNr8)r+rr,rr-rrrqrZr/r0rrrrrrrrmrqrnrar3rrorrrpr5rrq)r+rr,rrrrr)& __future__rtypingrrrr$paddle.nn.functionalnn functionalrUrpaddle.base.frameworkrpaddle.base.layer_helperr paddle.base.wrapped_decoratorr paddle.nn.attention.sdpar r r collections.abcrrrr)r;rfrlrurrrrrrrr9r*rrs=#"""""3333333333 888888000000GGGGGG ))))))!%8 %(       $'          ' ' ' '  '"    34343434lB %( (+"%         $'  (+"%           '(+"% ' ' ' ' '  '(   \   \\\\\~ %(  (+        $'  (+           ' (+ ' ' ' ' '  '   U    UUUUUp %('* & $''*      & '*'''' '8     H4H4H4H4f    )++++l    )uuuup %('* $ $''*      $ '*'''' '2     }4}4}4}4H+/ ^  &*$$'+"& $!^ ^ ^ ^ ^ ^ B``````r*