|-jy dddlZddlmZmZddlmZejeZeZ dej dej j dej fdZ ddej j dej d ej d ej d ej dzd ededzdedzdedzdedzdej dzdeej dffdZdS)N)_flash_attention_forward!flash_attn_supports_top_left_mask)loggingquerymodulereturnc,|jtjkr~tjdrtjdSt |jdr |jjStd|Dj jSdS)ziIf the query is in float32, return a target dtype compatible with flash attention. Return None otherwise.cuda _is_quantizedc3XK|]%}t|tjj!|V&dS)N) isinstancetorchnnLinear).0layers i/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/transformers/integrations/flash_attention.py z#get_target_dtype..s8bb%z%QVQYQ`?a?abbbbbbbN) dtyperfloat32is_autocast_enabledget_autocast_dtypehasattrconfignextmodulesweight)rrs rget_target_dtyper s {em##  $V , , p+F33 3 V]O 4 4 p=& &bb6>>+;+;bbbbbio o 4rkeyvalueattention_maskdropoutscalingsliding_windowsoftcap is_causals_auxc  T| ddrtd|jd} t d|jDrt d|dd}|dd}|dd}t||} | | n|j} t||||f| | ||||t| |j j t|dr|jnd| | |jndd | }|dfS) Noutput_attentionsFzFlash Attention does not support `output_attentions=True`. Please set your attention to `eager` if you want any of these features.rc3"K|] }|dkV dS)rN)rdims rrz*flash_attention_forward..0s& + +3!8 + + + + + +rzTensor query has shape with a zero dimension. FlashAttention does not support inputs with dim=0. Please check your input shapes or use SDPA instead. layer_idx) query_lengthr)r% softmax_scaler'r(use_top_left_mask target_dtypeattn_implementationr1r*)getlogger warning_onceshapeany ValueError transposer r)r_use_top_left_maskr_attn_implementationrr1tor)rrr"r#r$r%r&r'r(r)r*kwargsseq_lenr5 attn_outputs rflash_attention_forwardrDswzz%u--  W   k!nG + +u{ + + +++  B   OOAq ! !E --1  C OOAq ! !E$E622L'2 8HI*     %,!"M>&-fk&B&BL&""  HHU[ ! ! !%( )K.  r)r!NNNNN)rmodeling_flash_attention_utilsrrutilsr get_logger__name__r8r>TensorrModulerr floatintbooltuplerDr.rrrOs~ hhhhhhhh  H % %6688 EL %(/ ek    & !% !!%CC HOC <C C < C L4' C  CT\C$JCT\Cd{C <$ C 5< CCCCCCr