|-jx% ddlZddlmZddlmZddlmZddlmZm Z ddl m Z ddl Z ddl mZddlmZd d lmZmZmZmZGd d ZeGd dZdedefdZdedededefdZd,dedededefdZdedededefdZ d-de jdeedeed eddf d!Z d"ed#ed$ed%ed&e deef d'Z!d(ej"deefd)Z#d*Z$ed+Z%dS).N) OrderedDict)contextmanager) dataclass)ceillog2)Any)PretrainedConfig)is_torch_greater_or_equal)FutureRequestState RequestState RequestStatusloggerceZdZdZdeddfdZddZdeedfdej j dzfd Z dd e ddfd Z deedfd ej j ddfdZdS)CudaGraphBufferz>A fixed-size dict for CUDA graphs with LRU eviction when full.max_sizereturnNcj|dkrtd|||_t|_dS)Nrz#max_size must be positive, but got ) ValueErrorrr_storage)selfrs q/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/transformers/generation/continuous_batching/utils.py__init__zCudaGraphBuffer.__init__ s8 q==M8MMNN N  LWMM c\|j}d|_|d||_dS)Nr T)silent)rplan_for_new_graph)roriginal_max_sizes r__del__zCudaGraphBuffer.__del__&s3 M  t,,,) rkey.cr|j|}||j||SN)rget move_to_endrr graphs r get_graphzCudaGraphBuffer.get_graph,s8 !!#&&   M % %c * * * rFrct|j|jkrj|jd\}}|st jd||t|j|jkhdSdS)NF)lastz!Evicting graph for evicted_key = )lenrrpopitemrinforeset)rr evicted_key evicted_graphs rrz"CudaGraphBuffer.plan_for_new_graph2s$-  DM11)-)>)>E)>)J)J &K D B+BBCCC    ! ! ! $-  DM111111rr&cB|||j|<dSr")rrr%s r set_graphzCudaGraphBuffer.set_graph9s& !!!" cr)rN)F)__name__ __module__ __qualname____doc__intrrtupletorchcuda CUDAGraphr'boolrr1rrrrsHHZZZZZZ **** U38_1E1L """$""""#U38_#UZ5I#d######rrc0eZdZUdZdZeed<dZeed<dS) WorkloadHintszRA tiny dataclass containing hints to help choose good continuous batching defaultsrmax_prompt_lengthmax_generated_lengthN)r2r3r4r5r?r6__annotations__r@r<rrr>r>?s:\\s !#!!!!!rr>configrc|jdvS)z:Checks if attention mask is needed for the given (config).)z paged|eagerz paged|sdpa)_attn_implementation)rBs rattn_mask_is_neededrEGs  &*G GGrsize interval_size max_valuecl|dkr|S|dkrt||z |zn|}t||S)zQReturn the smallest multiple of (interval_size) >= (size), capped at (max_value).r)rmin)rFrGrHpaddeds rpad_to_intervalrLLsG;?!88T$& ' '- 7 7F vy ! !!rvalue min_valuect|td|}dttt|z}t ||S)zReturn the smallest power of 2 >= (value), capped at (max_value). If a minimum value is provided, the value is at least padded to that value.r )maxr6rrrJ)rMrHrNrKs r pad_to_pow2rRTsM s1i(( ) )E #d4;;''(( (F vy ! !!rx divide_byalign_tocdtt||z }||zr ||||zz z }|Sr")r6r)rSrTrUs raligned_dividerW\s? DY    A8|' XX && Hrattention_maskcumulative_seqlens_qcumulative_seqlens_ksliding_windowcjtj|jj}t t |dz D]}||dz||z }||dz||z }||kr|dkr ||z dz}nd}t ||||dz} t ||||dz} tj|d| | fj||j|j } tj | |} |dkr!||z |z } | tj | | z } | |d| | f<dS)uBuilds an attention mask inplace using the cumulative seqlens of the query and key. If given a sliding window, it will also apply a sliding window mask on top. The attention mask is not boolean, it uses zeroes and -inf (or its equivalent) so it's more of an attention score bias tensor. The attention mask is a block-diagonal matrix, with each block an attention mask for a single query-key pair. Each of those block is built from a causal mask and, if there is a sliding window, a sliding window mask. An example is represented below, with seqlen_k = 8, seqlen_q = 4 and sliding_window = 6: CAUSAL MASK: █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ █ █ █ █ █ █ █ ░ █ █ █ █ █ █ █ █ SLIDING WINDOW MASK: ┌──────────────────────── seqlen_k - seqlen_q - sliding_window = 8 - 4 - 6 = -2 offset to the left <─┴─> ░ █ | █ █ █ █ █ █ █ █ ░ ░ | █ █ █ █ █ █ █ █ ░ ░ | ░ █ █ █ █ █ █ █ ░ ░ | ░ ░ █ █ █ █ █ █ ATTENTION MASK (sum of causal and sliding window masks): █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ Another example with seqlen_k = 5, seqlen_q = 3 and sliding_window = 2: CAUSAL MASK: █ █ █ ░ ░ █ █ █ █ ░ █ █ █ █ █ SLIDING WINDOW MASK: ┌──────────────────────── seqlen_k - seqlen_q - sliding_window = 5 - 3 - 2 = 0 offset to the left <┴> | ░ █ █ █ █ | ░ ░ █ █ █ | ░ ░ ░ █ █ ATTENTION MASK (sum of causal and sliding window masks): ░ █ █ ░ ░ ░ ░ █ █ ░ ░ ░ ░ █ █ r .)dtypedevice)diagonalN) r8finfor]rJranger*slicefullshaper^triutril)rXrYrZr[rNiseqlen_qseqlen_kcausal_diagonal query_range key_range minus_infmaskedsliding_diagonals rbuild_attention_maskrpcsxt N0115I 3+,,q0 1 1=='A.1Ea1HH'A.1Ea1HH h  8q==&1A5OOO035I!a%5PQQ .q13GA3NOO J 3 Y6 7 =  &!(    I@@@ A  '(2^C  ej5EFFF FF6<sK233-==rnumstatus num_q_tokens max_kv_readcachec rfdt|D}||z}t||jz }g}|D]y} t| dg|zd} | _dg|z| _|| _||| jd} | |cS| t| dd|z|S)zPA utility function to create a list of FutureRequestStates for the warmup of CB.c,g|]}djd|dS) __warmup____)name).0rgrrs r z/create_warmup_future_states..s/GGGq2v{22Q222GGGrrr ) request_idinitial_tokensmax_new_tokensNT) has_new_tokencomplete_blocks query_length) rar block_sizer _statustokens_to_processposition_offsetallocate_blocksr~appendr ) rqrrrsrtru request_ids total_tokens blocks_needed future_statesreq_idstate allocateds ` rcreate_warmup_future_statesrsHGGGE#JJGGGK+-L(8899MM   s\?Qbcddd #$# "4 +))-9I1MM    uD!Zf g g g     r request_queuecg}|sV |}|||n#tj$rYnwxYw|V|S)z3Drains a queue and returns a list of RequestStates.)empty get_nowaitrqueueEmpty)r new_statesrs r drain_queuers%'J!!## !,,..E !!%((({    E  !!## s+AAActdr)tj}|j}||fSd}tj}||fS)zReturns a tuple of (mem_pool, graph_pool_id) for CUDA graphs. Since the MemPool object is only available in torch 2.5+, we only return a graph_pool_id for older versions.z2.5.0N)r r8r9MemPoolidgraph_pool_handle)mem_pool graph_pool_ids rget_cuda_poolsrs\!))':%%'' && 4466 &&rc#K|>tj|5dVddddS#1swxYwYdSdVdS)zA context manager to use a CUDA mem pool. If the mem pool is None, it is a no-op. No type hint because it would make torch 2.4 or below crash.N)r8r9 use_mem_pool)rs r mem_pool_ctxrs Z $ $X . .   EEE                   s 6::)r)r )&r collectionsr contextlibr dataclassesrmathrrtypingrr8 transformers.configuration_utilsr transformers.utilsr requestsr r rrrr>r;rEr6rLrRrWTensorlistrprQueuerrrr<rrrs ######%%%%%%!!!!!! ======888888MMMMMMMMMMMM########D """"""" "H 0HTHHHH "#"c"c"c""""""s"s"s"3"""" c c S S     Q=Q=LQ=s)Q=s)Q= Q=  Q=Q=Q=Q=h        : u{ tL/A     ' ' 'r