f,jX$0 dZddlZddlmZddlmZmZddlZeje Z gdZ de e dzde e fdZed d e defd ZGd deZejdi d:dejdejdejdejdejdzde de dededzde e dzdedejdzdejdzde dzdeejejejffdZej d:dejdejdejdejdejdzde de dededzde e dzdedejdzdejdzde dzdeejejejffd Zddd!ddddd"dejdejdejdejdejdzde de d#edzdedzdee e fdedejdzdejdzde dzdejeejejfzfd$Zejd%d&h d:d&ejdejdejdejdejdejdzde de dededzde e dzdedejdzdejdzde dzdejf d'Zej d:d&ejdejdejdejdejdejdzde de dededzde e dzdedejdzdejdzde dzdejf d(Zddd!ddddd"d&ejdejdejdejdejdejdzde de d#edzdedzdee e fdedejdzdejdzde dzdejeejejfzf d)Zd*ed+eed,fd-eddfd.Zejd/i d;d0ejdejdejdejd&ejd1ejdejdejde de ded2ejdedzde e dzdeejejejffd3Zej d;d0ejdejdejdejd&ejd1ejdejdejde de ded2ejdedzde e dzdeejejejffd4Z d*ed0ejd5ejd6ejdeejdzd,ff d7Z!e"e!e8ej#$ej%j&j'dd9l(m)Z)m*Z*m+Z+m,Z,e*e,ej%j-j<e+e,ej%j-j<e)e,ej%j-j<dS)R  ;q>R#7#7f   K$;T 88            9   *0F1IvayY  @AAA/4y~/V/V         #(^)!n#!!0W0 0 , Y1& ELJ ; **rcVt| } tj|}|d}|d}tj||ftj|j}tjjrytj }|tj j j krA|ddz }tj|||ftj|j}tjdtj |j}|||fS)z Fake implementation for meta tensor computation and tracing. Based on the 3D varlen path from meta__flash_attention_forward: - query shape: (total, num_heads, head_dim) - logsumexp shape: (num_heads, total_q) rr1r;r:)rrC empty_likesizeemptyfloatr=versionhip_C_get_rocm_fa_preferred_backend_ROCmFABackendAOTritonrI)r"r#r$r%r&r'r(r)r*r r+r,r-r.rLtotal_q num_heads logsumexp preferred batch_sizerNs r_varlen_attn_fakerbs0)55K e $ $FjjmmG 1 I GEK I }H;;== /8 8 8!q))A-J Y.ek%,I D U\JJJI 9i ''r)r r ) return_auxr*r r+r,r-r.rcc|d}| |dn|d}| s||krtd|d|d| r||zdkrtd|d|d | d k}tjj|||||||||t | | | | | \}}}| |jr||fS|S) aCompute variable-length attention using Flash Attention. This function is similar to scaled_dot_product_attention but optimized for variable-length sequences using cumulative sequence position tensors. Args: query (Tensor): Query tensor; shape :math:`(T_q, H_q, D)` key (Tensor): Key tensor; shape :math:`(T_k, H_{kv}, D)`, or :math:`(\text{total\_pages}, \text{page\_size}, H_{kv}, D)` when ``block_table`` is provided. value (Tensor): Value tensor; shape :math:`(T_k, H_{kv}, D)`, or :math:`(\text{total\_pages}, \text{page\_size}, H_{kv}, D)` when ``block_table`` is provided. cu_seq_q (Tensor): Cumulative sequence positions for queries; shape :math:`(N+1,)` cu_seq_k (Tensor): Cumulative sequence positions for keys/values; shape :math:`(N+1,)` max_q (int): Maximum query sequence length in the batch. max_k (int): Maximum key/value sequence length in the batch. return_aux (Optional[AuxRequest]): If not None and ``return_aux.lse`` is True, also returns the logsumexp tensor. scale (float, optional): Scaling factor for attention scores window_size (tuple[int, int], optional): Window size for sliding window attention as (left, right). Use (-1, -1) for full attention (default), (-1, 0) for causal attention, or (W, 0) for causal attention with sliding window of size W. enable_gqa (bool): If set to True, enables Grouped Query Attention (GQA) and allows key/value to have fewer heads than query. Each KV head is shared by a group of :math:`H_q / H_{kv}` query heads, so :math:`H_q` must be divisible by :math:`H_{kv}`. Default is False. seqused_k (Tensor, optional): Number of valid KV tokens per batch element; shape :math:`(N,)`. When set, only the first ``seqused_k[i]`` tokens in the key/value sequence for batch element *i* participate in attention. Useful for KV-cache decoding where the cache slot is larger than the actual sequence. Inference-only (not supported in backward). block_table (Tensor, optional): Block table for paged KV cache; shape :math:`(N, \text{max\_pages\_per\_seq})`, dtype ``int32``. Requires ``seqused_k``. Inference-only (not supported in backward). When ``block_table`` is provided, ``key`` and ``value`` are a "pool" of pages of tokens of KV data and the pages belong to any sequence/order. The ``block_table`` is what maps each sequence's logical chunks back to physical pages in this pool. ``seqused_k[i]`` tells the kernel how many tokens in sequence *i* are actually valid, since the last page is typically only partially filled. num_splits (int, optional): Number of splits for split-KV. Set to ``1`` to disable split-KV which enables batch invariance. Split-KV parallelizes the key/value sequence dimension across multiple thread blocks and combines partial results. The split decision depends on ``max_k`` (the longest sequence in the batch), so different batch compositions can change the reduction order and produce different floating-point results for the same sequence. When this is disabled, bitwise identical outputs are guaranteed for a given sequence regardless of what other sequences are in the batch, at the cost of lower GPU utilization when there are few queries. When ``None`` (default), the kernel chooses automatically. Returns: output (Tensor): Output tensor from attention computation; shape :math:`(T_q, H_q, D)`. If ``return_aux`` is not None and ``return_aux.lse`` is True: lse (Tensor): Log-sum-exp of attention scores; shape :math:`(T_q, H_q)`. Shape legend: - :math:`N`: Batch size - :math:`T_q`: Total number of query tokens in the batch (sum of all query sequence lengths) - :math:`T_k`: Total number of key/value tokens in the batch (sum of all key/value sequence lengths) - :math:`H_q`: Number of query attention heads - :math:`H_{kv}`: Number of key/value attention heads (equal to :math:`H_q` unless GQA is enabled) - :math:`D`: Head dimension Example:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> batch_size, max_seq_len, embed_dim, num_heads = 2, 512, 1024, 16 >>> head_dim = embed_dim // num_heads >>> seq_lengths = [] >>> for _ in range(batch_size): ... length = torch.randint(1, max_seq_len // 64 + 1, (1,)).item() * 64 ... seq_lengths.append(min(length, max_seq_len)) >>> seq_lengths = torch.tensor(seq_lengths, device="cuda") >>> total_tokens = seq_lengths.sum().item() >>> >>> # Create packed query, key, value tensors >>> query = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> key = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> value = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> >>> # Build cumulative sequence tensor >>> cu_seq = torch.zeros(batch_size + 1, device="cuda", dtype=torch.int32) >>> cu_seq[1:] = seq_lengths.cumsum(0) >>> max_len = seq_lengths.max().item() >>> >>> # Call varlen_attn >>> output = varlen_attn( ... query, key, value, cu_seq, cu_seq, max_len, max_len ... ) r1Nr GExpect query and key/value to have the same number of heads but got Hq= and Hkv=&. Try setting enable_gqa=True for GQA.rMExpect number of query heads to be a multiple of kv heads for GQA but got Hq=.r r)rTrrCrD torch_attnrQlistr)r"r#r$r%r&r'r(rcr*r r+r,r-r. num_heads_q num_heads_kr)outrrOs rrrsHj**Q--K!,!8#((1+++chhqkkK  +44 4% 4 40; 4 4 4    kK/144 ?% ? ?0; ? ? ?   w&I)&33       [KCa *.Cx Jrztorch_attn::_varlen_attn_outroc<t| } |jot|jj}|rt dt dtj j ||||||||d|d| | d| d| | |}|S)z Private custom op for variable-length attention with pre-allocated output. Same as _varlen_attn but writes the attention output into the provided out tensor. z+cuDNN backend does not support out variant.z1Using Flash Attention backend for varlen_attn_outr3Frr1)r*r8r9r,r-r.) rr>rr=r?rBr@rArCrDrE+_flash_attention_forward_no_dropout_inplace)ror"r#r$r%r&r'r(r)r*r r+r,r-r.rJrMs r_varlen_attn_outrrRs,)55K G"3EL4F"G"GIJHIIIHH @AAA).LL        $Q%a.#MK( rc|d}|d}tj||ftj|j}tjjrytj}|tjj j krA|ddz }tj|||ftj|j}|S)F Fake implementation for meta tensor computation and tracing. rr1r;) rTrCrUrVr=rWrXrYrZr[r\)ror"r#r$r%r&r'r(r)r*r r+r,r-r.r]r^r_r`ras r_varlen_attn_out_fakerus*jjmmG 1 I GEK I }H;;== /8 8 8!q))A-J Y.ek%,I rc|d}| |dn|d}| s||krtd|d|d| r||zdkrtd|d|d | d k}tjj|||||||||| t | | | | |}| |jr||fS|S) zCompute variable-length attention using Flash Attention with a pre-allocated output tensor. Same as :func:`varlen_attn` but writes the attention output into the provided ``out`` tensor instead of allocating a new one. r1Nr rerfrgrrhrirj)rTrrCrDrkrrrlr)ror"r#r$r%r&r'r(rcr*r r+r,r-r.rmrnr)rs rrrsD0**Q--K!,!8#((1+++chhqkkK  +44 4% 4 40; 4 4 4    kK/144 ?% ? ?0; ? ? ?   w&I )  / /        [  C"*.Cx Jrctxinputs.rLc|\}}}}}}} } } } } }}}|\}}}|td|td|||||||||||_| |_| |_| |_| |_dS)Nz)seqused_k is an inference-only parameter.z+block_table is an inference-only parameter.)rBsave_for_backwardr'r(r)r*r )rwrxrLr"r#r$r%r&r'r(r)r*r r+r,r-r.rorrNs r_setup_contextr{s        CiFGGGHIII%eXxc9UUUCICICMCI!COOOrz!torch_attn::_varlen_attn_backwardgrad_outrrNcTt| } tjd|j}|jot |jj}|rytd| ddks | ddkrtdtj j |||||||||| d| | || \}}}n_td tj j |||||||||| d| | || | d| d \}}}|||fS) Nr)r=r0r r1r2r3r4r6)r*r8r9)rrCrUr=r>rr?r@rArBrDrE_cudnn_attention_backward_flash_attention_backward)r|r"r#r$rorr%r&r'r(r)rNr*r unusedrJdqdkdvs r_varlen_attn_backwardr se")55K [5< 0 0 0F G"3EL4F"G"GI+  6777 q>R  ;q>R#7#7f Y^==              >  B$ @AAAY^==              (^)!n#>  B& r2:rct| } tj|}tj|}tj|}|||fS)rt)rrCrS)r|r"r#r$rorr%r&r'r(r)rNr*r grad_querygrad_key grad_values r_varlen_attn_backward_fakerQsN()55K!%((J$$H!%((J x ++rgrad_lsegrad_rngc|j\}}}}}} } } |j} |j} |j}|j}|j}t jj ||||| | ||| | || ||\}}}d}|||gd|zRS)N )N) saved_tensorsr'r(r)r*r rCrDrkr)rwr|rrr"r#r$r%r&rorrNr'r(r)r*r rrr num_paramss r _backwardrnsBEAR>E3x3Y IE IE I IE/K%;;        JBB$J B 0'J. 0 00r) setup_context)_varlen_attn_backward_flop_varlen_attn_forward_flop_varlen_attn_out_flop flop_registry)FNNFNNN)NN).rlogging functoolsrtypingrrrC getLoggerrr@__all__rlintrrrrlibrary custom_opTensorrVtuplerQ register_fakerbrrrrurr{rrrregister_autograd_dynamodisallow_in_graphrDrErqtorch.utils.flop_counterrrrrrkrrrrs """""""" g!! : : :S D(8T#Y 1CD 3"EE$(%)'+!V+V+ <V+ V+ <V+l V+ lT! V+  V+ V+V+ 4<V+cT!V+V+|d"V+$V+d V+ 5<u| 34V+V+V+FEV+r$(%)'+!.(.( <.( .( <.(l .( lT! .(  .( .(.( 4<.(cT!.(.(|d".($.(d .( 5<u| 34.(.(.(.(t%)#+%)'+!VVV <V V <Vl V lT! V  V VT!V 4<VsCxVV|d"V$Vd V  \E%, 455!VVVVr7ugNN$(%)'+!22 2 <2 2 < 2 l 2 lT! 2 2 22 4<2cT!22|d"2$2d 2  \!222ON2j$(%)'+!"" " <" " < " l " lT! " " "" 4<"cT!""|d""$"d "  \!""" "^%)#+%)'+!!::: : <: : < : l : lT! : : :T!: 4<:sCx::|d":$: d !:" \E%, 455#::::z""U38_"c"d""""B<2NN$(AAlA <A A < A  A  AlAlA A AA|A 4<AcT!A 5<u| 34AAAONAH$$(,,l, <, , < ,  ,  ,l,l, , ,,|, 4<,cT!, 5<u| 34,,,%$,81 1 105 1HM 1 5<$  #$1111B yGGG  IN>4M ei"/07L ei"34