|-jWddlZddlmZmZddlmZddlmZddlm Z m Z m Z m Z GddeZ Gd d e ZGd d e Zeed ZdS)N)ABCabstractmethod)deque)PagedAttentionCache)FutureRequestState RequestState RequestStatusloggerc eZdZdZdefdZd dZdefdZe d e d e de e e dzee e ffd Zdefd Zd eddfdZde eeffdZd ede e fdZd efdZde efdZd edefdZdede defdZdedeede e fdZdede e d e deeddf dZ d!de ed e d e deedede e e eee e ff dZde efdZdeeddfdZdS)" ScheduleraA Abstract base class for scheduling requests in the continuous batch processor. Schedulers manage the lifecycle of requests from when they are added to the waiting queue to when they are scheduled for processing. Different schedulers implement different strategies for prioritizing and batching requests. cachec||_tj|_|jjrdn|jjj|_|jj|jj z|_ | dS)N) r threadingLock_cancellation_locknum_full_attention_groupsconfigsliding_windowread_cache_limitmax_blocks_per_request block_sizemax_decode_fast_path_lengthreset)selfrs u/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/transformers/generation/continuous_batching/scheduler.py__init__zScheduler.__init__sa "+."2"2(, (L rRVR\RcRr+/:+LtzOd+d( returnNci|_i|_t|_t |_g|_d|_dS)z0Reset scheduler state for a new generation loop.FN)active_requestswaiting_requestsrwaiting_requests_orderset_requests_to_cancel_requests_to_forkblock_new_requestsrs rrzScheduler.reset&s@8:9;27''#-0UU 57"'rstatecb||j|j<|j|jdS)z#Adds a request to the waiting list.N)r" request_idr#append)rr)s radd_waiting_requestzScheduler.add_waiting_request/s227e./ #**5+;<<<<$U%56266u7GHHH,7 &&.DJ$U%56266u7GHHH|}777*/*H'13. -5 87|}4449>$U%56,7 266u7GHHH-;LMM-JE *&4]l]&CE # # #r candidates safety_marginc g}d}|jjdk}||jjz} ||} } |D]7} |j} | | k}|r4|r2| jt jkrtjd| d| n| | |}tt||}|dko| j |j k}| }|jt||j}|r|s||kr|| |}|s8d}| dkr.| j|jvr tjd| jd n,|| |||t| j}||dko| j |j kz}||z}||z}|jjr6| |jjz}||z}||jjz}nd}| j }|t3| |||| j}|j|ddu}|r|||dks|dkr|sn9| |z }| |z }|||||fS) a>Schedules candidate requests for the current batch. This method contains the common logic shared by all schedulers: it checks token and cache budgets, allocates cache blocks if needed, updates request states, and tracks which waiting requests should be removed from the waiting queue. FrzJOutside safety margin, breaking out of scheduling loop. num_free_blocks = z safety_margins = rNTzBreaking mid-loop for request z because the cache is full)rr num_blocksget_num_free_blocksr[r rcr inforgrar5rbrrQrrWr+r"rjrdallow_block_sharingrr_r,rr9rE)rrlr.r/rXrmscheduled_requestsone_allocation_faileddecode_fast_pathsafety_marginsoriginal_token_budgetoriginal_cache_budgetr)num_free_blocksoutside_safety_marginrf request_lenis_decode_eligibleread_cache_neededallocation_successfultokens_in_current_blocktokens_after_forwardcomplete_blocks has_new_tokenreq_id was_waiting num_q_tokens max_kv_reads r_process_candidateszScheduler._process_candidatess.  %:>7C\4F F E"j<<>>O$3n$D !$ );  P]Pf@f@f wwwdrww"77?abbNc.11<@@K "-!1!ne6KdNn6n  % 1 1 3 3 $0$'(94;P$Q$Q!$ ); PaAaAa%)$C$CE;$W$W !) (,%#a''E,<@U,U,UK mAQ m m mnnnE  " "5.,Hj k k ke566K  q 0 mU5JTMm5m m  K 'L - -Lz- $*/*;*;*=*= @U*U''>'L$"6$*:O"O"#!& >>M  % %&8 `k&l&l m m m%F/33FDAAMK ?266v>>>q  \Q%6%6?O%6,|; +l: !#8:JLZeeerc`t}t}|jD]-}|j|}|jr|n||.g}|s|rV|r'|||r'|||T|V|S)zReturns waiting requests in priority order. Since CPU-offloaded requests are cheaper to restore than fresh requests, they get priority, but we interleave them with fresh request to not saturate new batches with only offloaded requests.)rr#r"r]r,popleft)r offloadedfreshrr)ordereds r_get_waiting_candidatesz!Scheduler._get_waiting_candidates%s*/ %*WW1 K KF)&1E0 ;YYe C CE J J J J&( 05 0 4y0022333 0u}}///  05 0 rcRtfd|jD|_dS)z8Removes processed requests from the waiting queue order.cg|]}|v| Sr1r1).0rrXs r z4Scheduler._cleanup_waiting_queue..9s$ r r rOqAqAqVAqAqAqrN)rr#)rrXs `r_cleanup_waiting_queuez Scheduler._cleanup_waiting_queue6s5&+ r r r r$"= r r r' ' ###r)rN)rk) __name__ __module__ __qualname____doc__rrrr r-rinttuplelistrr4r2r6strr;r@rCrFrLrNrWr$rgrjfloatrrrr1rrr r s< 1((((===== mm/2m t&'$.c3> ?mmm^mMdMMMM333333@eC,=&>@@@@CDI 535555  $|*<    " s t     sW[$<]`ad]ejnorjs2$D$DS $D $D -0H $D  $D$D$D$DX # ^f^f&^f^f ^f -0H ^f  ^f t&'tS#= >^f^f^f^f@l);" S VZ      rr c heZdZdZd dedeffd Zdededee e d ze eeffd Z xZ S) FIFOScheduleraKThis scheduler processes requests in the order they arrive, meaning decoding requests has priority over prefilling requests. Additionally, it includes a safety margin mechanism to prevent cache exhaustion. By default, when 80% of the cache is full, new requests will not be scheduled to prioritize decoding active requests.皙?rrmcXt|||_dS)a[Initializes the FIFO scheduler. The safety margin is the percentage of free blocks under which we stop scheduling new prefill requests, so safety_margin = 0.1 means that when there is less than 10% of free blocks, or equivalently when more than 90% of blocks are already allocated, we stop scheduling new prefill requests. N)superrrm)rrrm __class__s rrzFIFOScheduler.__init__Cs+ *rr.r/rNcg}g}|jD]W}|jtjkr||-|jtjkr||X|js'|| ||z}t}| |||||j \}} } } } | ||s| rd| ddfS|| | | fS)Nrmr)r!valuesr[r rcr,r`r'extendrr$rrmr rr.r/priority_statessecond_priority_statesr)rlrXrsrtrurrs rr2zFIFOScheduler.schedule_batchKsC/157)0022 5 5E|}555&&u----!999&--e444& J " ) )$*F*F*H*H I I I$'== -0UU*  $ $2"0 %   _13C\S^ ##$FGGG" 0&; 0)1a/ /!#3\;NNr)r)rrrrrrrrrrrr4r2 __classcell__)rs@rrr>sqq++1+%++++++#O#O/2#O t&'$.c3> ?#O#O#O#O#O#O#O#Orrc LeZdZdZdededeeedzeeeffdZ dS)PrefillFirstSchedulerzScheduler that prioritizes split prefill requests over decoding requests. This scheduler ensures that split prefill requests (which are continuations of partially processed prompts) are completed before processing new decoding requests.r.r/rNcg}g}|jD]W}|jtjkr||-|jtjkr||X|js'|| ||z}t}| ||||d\}} } } } | ||s| rd| ddfS|| | | fS)Nrkrr) r!rr[r r`r,rcr'rrr$rrrs rr2z$PrefillFirstScheduler.schedule_batchxs@/157)0022 5 5E|}777&&u----!777&--e444& J " ) )$*F*F*H*H I I I$'== -0UU*  $ $2! %   _13C\S^ ##$FGGG" 0&; 0)1a/ /!#3\;NNr) rrrrrrrrr4r2r1rrrrssl$O$O/2$O t&'$.c3> ?$O$O$O$O$O$Orr)fifo prefill_first)rabcrr collectionsrrrrequestsrr r r r rrSCHEDULER_MAPPINGr1rrrs'########&&&&&&MMMMMMMMMMMMd d d d d d d d P 0O0O0O0O0OI0O0O0Oj)O)O)O)O)OI)O)O)OZ *r