|-jddlZddlmZmZmZddlmZddlZddlm Z ddl m Z ddl m Z dd lmZmZmZmZdd lmZdd lmZdd lmZmZmZmZd e deeeeeeffdZ GddZ!GddZ"dS)N)floorgcdsqrt)Any)PreTrainedConfig)ContinuousBatchingConfig)is_flash_attention_requested) BlockManagerCacheAllocatorFullAttentionCacheAllocatorSlidingAttentionCacheAllocator)DistributedHelper)resolve_max_memory_percent) RequestState RequestStatusget_device_and_memory_breakdownloggerconfigreturnc& t|dd 5t|dddndfdt|jD i}t D]"\}}||g|gz||<#t d|D}g}|D]F\}}tdt||D]"}| ||||z#G fd |D}||fS) a Group layers depending on the attention mix, according to VLLM's hybrid allocator rules: - Layers in each group need to have the same type of attention - All groups have the same number of layers For a model with the following layer types: ["sliding", "full", "full", "sliding", "full", "full", "full", "full"] We would get four groups: [0, 3], [1, 2], [4,5] and [6,7]. layer_typesNsliding_windowsliding_attentionfull_attentioncg|]}Sr).0_ attn_types q/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/transformers/generation/continuous_batching/cache.py z-group_layers_by_attn_type..*sJJJQyJJJc,g|]}t|Sr)len)rindicess r"r#z-group_layers_by_attn_type..2sIIIs7||IIIr$rc,g|]}|dS)rr)rlgrs r"r#z-group_layers_by_attn_type..:s"===";r!u%===r$) getattrrangenum_hidden_layers enumerategetrvaluesitemsr&append) r layer_countsi layer_type group_size layer_groupsr' group_typesr!rs @@r"group_layers_by_attn_typer8sf&-66K+26;KT+R+R+^''dt JJJJ%0H*I*IJJJ L";//JJ :#/#3#3J#C#Cqc#I Z  II<3F3F3H3HIIIJJL+1133== Gq#g,, 33 = =A   A N(: ; < < < < =>=== ===K  $$r$ceZdZdZejfdededeje zde de e e fdej dd fd Zd ed edefd Zdede d eded zfdZde dd fdZdefdZde dededeeed zdeeedd f dZde dededejdd f dZdedede e effdZdejdejdedeejdeejdeejejff dZde de fd Zde d!eedefd"Zd#ed$edd fd%Zd&eed'eedd fd(Z d)e d*ee deeeeeffd+Z!d-d,Z"d S).PagedAttentionCacheu Manages the cache for a paged attention mechanism, inspired by VLLM's hybrid allocator. The cache relies on making groups of layers to reduce the complexity of cache management and fragmentation. The cache uses a three-level hierarchy: - Pages: The smallest unit of cache, a page has a size of [num_heads, head_size], which is the space needed to store the key or value states for one token and one layer. For a model with only full-attention layers, to store the KV cache of one token, we need `2 * num_layers` pages: key and values each take `num_layers` pages. Pages are grouped into blocks: - Blocks: A block is a collection of `block_size` pages, serving as the allocation unit to reduce management complexity and fragmentation. Cache is allocated and freed block by block, not page by page. One block is allocated to one layer group, which only has one attention type, like full-attention or sliding-attention. If all layers in the model have the same attention type, then all layers will be in the same group. There is more than one group if and only if the model has a mixed attention types, like layers with full-attention and layers with sliding-attention. - Cache tensors: The physical supports for the cache. There are as many cache tensors as there are layer in a layer group, and the shape of the cache tensor is `[num_blocks * block_size, num_heads, head_size]`. Grouping layers into groups is useful because when we allocate one block to a group N, the block allocated is the same for all layers in group N, equivalently it is allocated across all cache tensors. This allows us to efficiently allocate and free blocks, and to efficiently read and write key and value states. For instance, imagine we have 8 blocks of cache and a model with two layer groups: a full-attention group with 3 layers and a sliding-attention group with 3 layers. At creation time, the physical cache tensors look like this: cache_tensor_0: □ □ □ □ □ □ □ □ cache_tensor_1: □ □ □ □ □ □ □ □ cache_tensor_2: □ □ □ □ □ □ □ □ where □ means the blocks is not allocated to any layer group yet. We have 3 cache tensors because there are 3 layers per group. We allocate 1 block to each group, after allocation, the cache tensors look like this: cache_tensor_0: ✖ ◉ □ □ □ □ □ □ cache_tensor_1: ✖ ◉ □ □ □ □ □ □ cache_tensor_2: ✖ ◉ □ □ □ □ □ □ where ✖ means the block is allocated to the full-attention group, and ◉ means the block is allocated to the sliding-attention group. Now, if we continue to generate, and the sliding window has been reached, we only need to allocate a new block for the full-attention group, and the cache tensors look like this: cache_tensor_0: ✖ ◉ ✖ □ □ □ □ □ cache_tensor_1: ✖ ◉ ✖ □ □ □ □ □ cache_tensor_2: ✖ ◉ ✖ □ □ □ □ □ And after further generation, when we need a new block allocated: cache_tensor_0: ✖ ◉ ✖ ✖ □ □ □ □ cache_tensor_1: ✖ ◉ ✖ ✖ □ □ □ □ cache_tensor_2: ✖ ◉ ✖ ✖ □ □ □ □ This would not have been possible if all layers were in the same group: we would have had to allocate a new block for the sliding-attention group, although it is not needed. rcontinuous_batching_configdevicedistributed_helpertp_plandtyperNc  ||_||_||_t|dd}||n|j|_t|dd}||n|j|jz|_|j|_|jdkrtd|jt|\} } t| d} t| |_ i|_ i|_t| D]E\} } | | dkr|jnd}t| D]\}}| |f|j|<||j |<Fd}d D]}||vs d |z|vsd }n|j}|dkr;|r9|j|zdkrtd |jd |d|xj|zc_|j|jz}t%|jrd}n d| vrd}nd}|j|jz}d|jd|jzzf}d|z|j|zd|zzf}t)|||j | ||g|}|jt-|d||j|j|j|j\}}|dkrt5j||g|jt4j}||t=|dt=|d}}||_||_|j|jz|_ tCj"d|jd|jd|d|jd| |j#}||j$}||_#g|_%g|_&|dz|jz|j|jf|_'|j'ddz |_(|j(dz |_)tU| D]}t5j+|j'|j|j}t5j+|j'|j|j} t4j,-|t4j,-| |j%.||j&.| tCj"d|j'd|j%dj/d|j%d0|j1|_1g|_2d|_3d|_4d|_5t| D]\} }!|!dkr-tm| |j|j1}"|xj3dz c_3n\|!dkrDto| |j|j|j(|j)}"|xj4dz c_4|"j8|_5ntd|!|j2.|"|j1o| dgk|_9tu||j|dk |_;d|_<d|_=dS)!a/Initialize a paged attention cache for efficient memory usage. Also turns in prefix sharing if the model has only full attention layers. Args: config: Model configuration continuous_batching_config: Continuous batching configuration containing cache parameters device: Device for the cache tensors distributed_helper: TP-aware helper. Used to dispatch attention heads and ensure coherent cache size tp_plan: Tensor parallelism plan dtype: Data type of the cache num_key_value_headsNhead_dimrz%Block size must be positive, but got rr T)zlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzmodel.FzNumber of key value heads z+ must be divisible by tensor parallel size .)r; page_size num_groupsr5activation_peaksnum_attention_masks) cb_confighas_logit_processors) num_blocksmax_batch_tokensmax_memory_percent cache_dtyper<r?z7PagedAttentionCache initialized with self.num_blocks = z, self.block_size = z, page_size = z, self.max_batch_tokens = z num_attention_masks = )r?r<zself.cache_shape = z self.key_cache[0].shape = z self.key_cache[0].numel() = r)allow_block_sharingzInvalid group type: )tp_on)>rr?r<r*num_attention_headsrA hidden_sizerB block_size ValueErrorr8r&rFsliding_windowslayer_index_to_group_indicesr-rtp_sizer vocab_sizePagedAttentionMemoryHandlerrMr%infer_num_blocks_and_max_batch_tokensrKrLtorchtensorint64tp_all_reduce_minintitem num_pagesrinfomax_blocks_per_requestfallback_max_blocks_per_request key_cache value_cache cache_shapesentinel_index trash_indexr+empty_dynamomark_static_addressr1shapenumelrPgroup_cache_managersnum_full_attention_groupsnum_sliding_attention_groups%max_sliding_window_blocks_per_requestrr_max_blocks_per_requestuse_prefix_sharingr _block_manager_total_prefix_length_block_table_key)#selfrr;r<r=r>r?kv_headsrBr6r7r5r3grouprjlayerkv_is_tpkeyrXrErHq_bytes_per_token lm_head_peakattention_peakmemory_handlerrKrLsyncrdr new_layer_key_cachenew_layer_value_cache group_typecms# r"__init__zPagedAttentionCache.__init__ws (   6#8$??4<4HfNh 6:t44)1)=XX6CUY_YsCs 5? ?a  VT_VVWW W%>f$E$E! ka)) l++!,.)!,// = =HAu6A!nH[6[6[V22abN%e,, = =5<=q61%8.<$U++ = M  C7NNhn&?&? %, Q;;8;''1Q66 A1IAAv}AAA  $ $ 0 $ $MD$<< ' 4 4 $"#   K / /"#  "# #6F  V%6!6 6  M  !2 2Q] B  5'A!*N; 3     & 8 @ &1Kbf g g g g'5'['[1<7H9L (\( ( $ $ Q;;<-= >t{Z_ZefffD  0 0 6 6 6+.tAw||~~+>+>DGLLNN@S@S(J% 04?:  CDO C C$/ C C`i C C$ C C*= C C   "@XZ^Zgh".q1A5.2z"" ; ;A"'+d.>djY]Yd"e"e"e $)K0@ [_[f$g$g$g ! M - -.A B B B M - -.C D D D N ! !"5 6 6 6   # #$9 : : : : ht'hhT^A->-Dhh$.YZJ[JaJaJcJchhiii$>#Q :<!)*&,-)562&{33 1 1MAz---0DOY]Yqrrr..!3...2223t(=t?RTXTd11Q611=?=W:: !D !D!DEEE  % , ,R 0 0 0 0#'":"`{O_N`?`*:tgXYkZZZ)*!!%r$num_requested_blocksallocated_blocksc||jz}|jr3t|j|z d}|t |||jzz }||kS)aNReturns a boolean indicating if the allocation of (num_requested_blocks) blocks will be successful. The number of newly allocated blocks needed is predicted by the following rules: - for full attention groups: since there is no sliding window for full attention layers, one requested block is always equivalent to one newly allocated block for EACH full attention group - for sliding window groups: because of the sliding window, the number of blocks allocated to a request is capped. Using the number of already (allocated_blocks) we can compute the number of new blocks to actually allocate to the request, which can be lower than the number of requested blocks. That number is the same for all sliding window groups, as only one sliding window size is supported. r)rqrrmaxrsminget_num_free_blocks)ryrr needed_blocks blocks_lefts r"will_allocation_be_successfulz1PagedAttentionCache.will_allocation_be_successful'sj-t/MM  , hdHK[[]^__K S.BCCdFgg gM 8 8 : :::r$n_blocks request_idc|||sdSd}|jD]E}||||j}|t d|d|t ||}F|S)zAllocate cache blocks across all layer groups for a given request. Actual allocation is done by the cache managers, and this method only returns the maximum number of blocks actually allocated across all managers.NrzFailed to allocate z blocks for request )rrpallocate_blocksrvrUr)ryrrr max_allocatedrnum_allocated_blockss r"rz#PagedAttentionCache.allocate_blocks9s11( !L!L ^ ^ B  !5!5j+|!\!\ ] ] ] ]  !$'(A:$N$N ` ` L##B$7$7 KQ]$^$^____ " ! ` `r$ block_tablecvt|jD]#\}}||||||$dS)N)r-rpfill_block_table)ryrrrrr3rs r"rz$PagedAttentionCache.fill_block_tablefsRt899 W WEAr    K{ST~ V V V V W Wr$ci}|jdkr||z|d<|jdkr#|t||jjdz z|d<|S)zRetrieve the key sequence length for the given request_id across all layer types. Returns a dictionary of layer types to their corresponding key sequence lengths.rrr r)rqrrrrr)ryrr seqlens_ks r" get_seqlens_kz!PagedAttentionCache.get_seqlens_kls`  )A - -*5 *DI& '  ,q 0 0-9C T[MgjkMk>   ! ! # #q ( (   #4j A A A   #4l C C C|+ +-i8 Q     #4j A A A   #4l C C C$)$6wCS$T$T !&+&8!EU&V&V # #%(;;FFrJJTTUWXXD$)$6wCS$T$T ! ! 1 1$ C C C&+&8!EU&V&V # # 3 3D, G G G   #4j A A A   #4l C C C%&===r$flash_attn_with_kvcache_fnc|jgtj|j}d|vrd|_n0d|vrd|_n$t dtj||jS)zA function to get the name of the block table key for the given flash_attn_with_kvcache_fn. The function's signature is only inspected once. This is necessary because different version of flash have different names for the block table key.Nr page_tablezOflash_attn_with_kvcache_fn does not have a block_table or page_table argument: )rxinspect signature parameterskeysrU)ryr kwarg_namess r"get_block_table_keyz'PagedAttentionCache.get_block_table_keys  (!+,FGGRWWYYK ++(5%%,,(4%% VfmfwySgTgTVV$$r$ prompt_idscBd}g}tt||jzD]}|||jz|dz|jz}|j||d}|jj|}|0|||j||r?tj d|dt|d|j d}||j |<t||jz} |xj | z c_ | S)aSearches for a prefix match in the cache for the given (prompts_ids). If one is found, we reference the matching blocks in the (request_id), increase the reference count of the blocks and return the number of blocks that match. If no prefix match is found, we return 0.Nr r)group_idzFound prefix match for request z with z blocks)r+r&rTrv compute_hash _hash_to_idr.r1increase_ref_countrdebugrprrw) ryrr current_hashrbtokensblock_idr prefix_lengths r"search_prefix_matchz'PagedAttentionCache.search_prefix_matchs? s:$/9::  ADO 3q1u6O OPF.;;L&[\;]]L*6::<HHH# ''111#66x@@@@  : Lk:kkSQaMbMbkkk l l l*1-B)9BN: &,--?  !!]2!!r$statenum_complete_blocksc|dks|jtjkrdS|jD]C}|jr:|j||j|j|j |j zDdS)aMarks the blocks allocated to a request (state) as complete if they are shareable and they have been computed in the forward pass. A complete block is a block where the KV cache has been fully computed: if the block has enough space to hold the cache for N tokens, the block is marked as complete when the cache data is present for the N tokens. If block sharing is off, this is a no-op.rN)rrr) statusrFINISHEDrpuses_block_sharingrv!mark_shareable_blocks_as_completerrinitial_tokensgenerated_tokens)ryrrrs r"rz5PagedAttentionCache.mark_shareable_blocks_as_completes ! # #u|}7M'M'M4+  B$ #EE(;%'^E4D%E % 4u7M MF  r$list_source_blockslist_forked_blocksctj||jtj}tj||jtj}t |j|jD]i\}}|d|j|j |j }|d|j|j |j }||||<||||<jdS)z;Copy the cache from the source blocks to the forked blocks.rOrN) r\r]r<int32rrfrgviewrTrArB)ryrr source_blocks forked_blocksrfrgs r" copy_cachezPagedAttentionCache.copy_caches %7 SXS^___  %7 SXS^___ &)$.$:J&K&K D D "I{!r4?D#6#6#8#8 9 9 9 9) ) )J   Z ( ( ( ( ) )r$)rN)#__name__ __module__ __qualname____doc__r\float16rr r<strrdictrr?rr`boolrrrrlistrTensorrrtuplerrrrrrrrrr$r"r:r:>s66~#]n%n% n%%=n% s" n% . n% c3h n%{n% n%n%n%n%`;#;Y\;ae;;;;$   PS X[^bXb    L;>l;> ;> & ;> %,' ;> u|U\) *;>;>;>;>z%c%c%%%% ctCyS4|Z]bf"DT#YDDQTIDZ^DDDD 1c 1DQTI 1Z_`deh`ikopskt`tZu 1 1 1 1))))))r$r:ceZdZdZejZejZdZ dZ de de de de de ee e fd e d d fd Zedded e fdZdee e fdejd ee e e e ffdZedededed efdZdee e fde de d zde d zdejd ee e ff dZd d dejfde d zde d zdedejd ee e ff dZde de dejd e fdZd S)rZuDetermines the optimal number of pages (N) and max batch tokens (M) for the paged attention cache, given available GPU memory. The relation between N and number of blocks is: num_blocks = N // block_size. The memory footprint is a polynomial in N and M, where each term maps to a tensor allocated in ``ContinuousBatchingIOs._setup_static_tensors`` or ``PagedAttentionCache.__init__``: memory(N, M) = coeff_n · N + coeff_m · M + coeff_nm · N·M + coeff_mm · M² See ``_equation_coefficients`` for the breakdown. All three solving modes (auto, fixed-N, fixed-M) reduce to solving this equation, which is at most quadratic in one variable. iir;rErFr5rGrHrNc|j|_||_||_||_||_||_|j|_|j |j|_|jrdnd|_ |j rdnd|_ dS)uNInitialize the memory handler. `activation_peaks` is a list of `(Δcn, Δcm)` pairs giving the activation memory contributions proportional to N (pages) and M (batch tokens) for each peak. Memory must satisfy the constraint at every peak, so we solve each polynomial independently and take the most restrictive result.NrDr ) rTrErFr5rGrHrdrereturn_logprobsnum_output_rowsuse_async_batching io_multiplier)ryr;rErFr5rGrHs r"rz$PagedAttentionMemoryHandler.__init__$s5?"$$ 0#6 &@&W#  & .*D*dD '$>$NUqqTU"<"OVQQUVr$?rMcvt\}}}}|t||z }t||z}|S)z^Calculate available GPU memory for cache allocation, accounting for already allocated tensors.)rrr`)rMr totalreserved allocatedavailable_memorys r"get_available_memoryz0PagedAttentionMemoryHandler.get_available_memory>sI)H(I(I%5(I 3y(#;#;;/2DDEEr$peakrNc|jj}|jj}|j}|j}|\}}d|jz|jz|z||jzdzz||zz} ||z|dz|zz||jz|zz||jz|jz|zz||jzdzz||jzdzz} ||j z|z} ||j z|z} | | | | fS)uReturns `(coeff_n, coeff_m, coeff_nm, coeff_mm)` for the memory polynomial of a single activation peak. `peak = (Δcn, Δcm)` is the peak-specific activation contribution; the rest of the coefficients are shared across peaks. Each addend is annotated with the tensor it corresponds to in `ContinuousBatchingIOs._setup_static_tensors` (or the forward pass, for activation terms). rD) _input_dtypeitemsize_activation_dtyperr5rErFrrdrH) ryrrNr3ackdelta_ndelta_mcoeff_ncoeff_mcoeff_nmcoeff_mms r"_equation_coefficientsz2PagedAttentionMemoryHandler._equation_coefficientsHs)   &  " +      $. 01 4$/!A% &k   aK!eai $&&* +$/!)*,-. . $/!A%  & $/!A%  & t//!3t//!3833r$rrrc|dkr| |z S|dzd|z|zz }|dkrtd|d| t|zd|zz }|dkrtd|d|S)uQLargest positive root of a·x² + b·x + c = 0. Falls back to linear when a == 0.rrDz!No real solution (discriminant = )zNo positive solution (root = )rUr)rrr discriminantroots r"_solve_quadraticz,PagedAttentionMemoryHandler._solve_quadraticns 6626M!ta!eai' !  PPPPQQ QT,'''AE2 !88DTDDDEE E r$ availablerKrLcv|||\}}}} ||}d} ||| z| | dzzz||| zz| } t| | z}||jkr |j}d}n*t t | |jz|j}|G|} t ||| zz | | dzzz ||| zzz } t | |jz|j}nQ|O||jz} || ||| zz|| z|z } t t | |j}||fS)zSolve for `(num_blocks, max_batch_tokens)` against one activation peak's memory polynomial. Clamps to upper bounds. Either input may be None; whichever is None is solved for.Ng{Gz?rD)rr%r`_upper_bound_max_batch_tokensrrrT_upper_bound_num_blocks)ryrr&rKrLrNcnrcnmcmmmrbMNs r"_solve_for_peakz+PagedAttentionMemoryHandler._solve_for_peak{sm 66t[IIBS  "2":A--cAgad .BBaKR[Q[\\I"9q=11 $"DDD#'#E !  y!1!1T_!DdFbcc   Ay261C!Q$J>2a<PQQIY$/94;WXXJJ  %T_,A%%c2a<a)9KLLA"588T-OPP +++r$g?c||}tjd|td}td}|jD]>}||||||\} } t || }t || }?||}}||||} | |krtd| d|||fS)uSolve for the missing variable(s) in the memory polynomial (see ``_equation_coefficients``). There is one polynomial per activation peak; we solve each independently and take the most restrictive (smallest) result. When both `N` and `M` are unknown, assumes `M = m·N` (m = 0.01, i.e. one batch fills ~1 % of the cache) and solves the resulting quadratic in N. zCache memory: infzMemory footprint z is more than available memory ) rrrcfloatrGr0rcompute_memory_footprint MemoryError) ryrKrLrMrNr&acc_num_blocksacc_max_batch_tokensrrm_batch_tokensmemory_footprints r"r[zAPagedAttentionMemoryHandler.infer_num_blocks_and_max_batch_tokenss--.@AA  0Y00111u$U||) M MD'+';';D)ZYikv'w'w $Hn ::N#&';^#L#L '57K$ 88EUWbcc i ' 'n2Bnnclnnoo o+++r$c||jz}|}d}|jD]J}|||\}} } } ||z| |zz| |z|zz| |z|zz} t|| }K|S)zaEvaluate the memory polynomial at concrete (N, M) values, taking the max across activation peaks.r)rTrGrr) ryrKrLrNr/r.max_memory_footprintrr*rr+r,r9s r"r4z4PagedAttentionMemoryHandler.compute_memory_footprints  (  ) O OD#::4MM BC!AvQq1rGs!!!!!!!!!! 333333FFFFFF999999tttttttttttt******666666ZZZZZZZZZZZZ%&6%5d3iRVWZR[A[;\%%%%BP)P)P)P)P)P)P)P)hu$u$u$u$u$u$u$u$u$u$r$