|-jKvUddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl mZmZddlmZd d lmZejeZ dd lmZmZmZmZmZdd lmZdd lmZ  ddlm!Z!n #e"$rdZ!YnwxYw ddlm#Z$dZ%n #e"$rdZ%YnwxYwej&'dd(Z)dZ*e)evZ+de,fdZde,fdZ#deddideddiej-eddiej-eddiej-ed d!iej-ed"d!iej-eddid#ded$d%iej.ed&d'ej-ed&d'iej-ed(d'iej-ed&d'iej-ed&d)id*dej-ej/zed+d,d -iidej-ej/zed+d.d -iidej-ej/zed+d/d -iidej-ej/zed+d0d -iidej-ej/zed+d1d -iidej-ej/zed+d2d -iid3 Z0e1e,e1ee,zee1eefzffe2d4<e!>ej-e!d5d67iej.e!d5d67ej-e!d5d67id8e0d9<d:Z3d^d;Z4n+#e"$r#dZ*dZ+d<Zd=Z#Gd>d?Zd@ZdAZdBZ4YnwxYwdCd dDdEd dDdEd dDdFd dDdGd dDdHdIdJdKZ5e1e,e1e,e,ffe2dL<iZ6e1e,e dzfe2dM<dNe,dzdOe7fdPZ8 d_dNe,dQedzdRe7dOe dzfdSZ9e6fdTe,dUe1e,e dzffdVZ: d`dTe,dWe,dzdXe;e,zdzdRe7dOe f dYZdZeed\Z?gd]Z@dS)aN)Callable)contextmanager) ModuleType)version)ENV_VARS_TRUE_VALUESlogging)is_kernels_available)flash_attention_forward)DeviceLayerRepositoryModeregister_kernel_mappingreplace_kernel_forward_from_hub) get_kernel)use_kernel_forward_from_hub)FuncRepository)use_kernel_func_from_hubTFUSE_HUB_KERNELSYES layer_namecxtrt|StdtdS)NFkernels hub usage is disabled through the environment USE_HUB_KERNELS=c|SNclss e/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/transformers/integrations/hub_kernels.pyz-use_kernel_forward_from_hub..Fss)_kernels_enabled$_kernels_use_kernel_forward_from_hublogger warning_once_TRANSFORMERS_USE_HUB_KERNELS)rs r rr?sF  #7 CC C   xYvxx   #? "r" func_namectrtrt|Ststdn"tdt dS)Nzwuse_kernel_func_from_hub is not available in the installed kernels version. Please upgrade kernels to use this feature.rc|Srrfuncs r r!z*use_kernel_func_from_hub..Usr")r#_has_use_kernel_func_from_hub!_kernels_use_kernel_func_from_hubr%r&r')r(s r rrHs|  % = %4Y?? ?0 ##B ##|]z||%$ $r"cudaz!kernels-community/deformable-detrMultiScaleDeformableAttention)repo_idrzkernels-community/moe Llama4TextMoezkernels-community/liger_kernels LigerRMSNormzkernels-community/rmsnormRMSNormzkernels-community/mlx_rmsnorm)r/rocmxpumpsnpuzmedmekk/triton-llama-mlpTritonLlamaMLPzkernels-community/megablocksMegaBlocksMoeMLPzahadnagy/megablocksCPUMegaBlocksMoeMLP)r/r5r6cpuzkernels-community/activationFastGELU)r1rr QuickGELUNewGELUSiluGeluGeluTanh) r0r2r4MLPr:r=r>r?SiLUGeLUrB_KERNEL_MAPPINGzkernels-community/rotaryapply_rotary_transformers)r1r()r6r/rotary_pos_embcf|vp,tfd|DS)Nc3bK|])}t|tot|V*dSr) isinstancedicthas_key).0vkeys r zhas_key..s:[[1z!T22Fwq#[[[[[[r")anyvalues)drPs `r rMrMs<ax[3[[[[PQPXPXPZPZ[[[[[[r"c|t}t|drtdstdt |dS)Nr6z0.10.2) MIN_VERSIONzfkernels uses an incompatible version. Please install the latest version with `pip install -U kernels`.)rFrMr ImportErrorr)mappings r $register_kernel_mapping_transformersrYs\ ?%G 7E " " +?H+U+U+U x  (((((r"c d}|S)Nc|Srrrs r decoratorz.use_kernel_forward_from_hub..decoratorsJr"rargskwargsr\s r rrs   r"c d}|S)Nc|Srrr+s r r\z+use_kernel_func_from_hub..decoratorsKr"rr]s r rrs   r"ceZdZdZdS)rc td)NzNLayerRepository requires `kernels` to be installed. Run `pip install kernels`. RuntimeError)selfr^r_s r __init__zLayerRepository.__init__ sopp pr"N)__name__ __module__ __qualname__rgrr"r rr s( q q q q qr"rc td)Nz^replace_kernel_forward_from_hub requires `kernels` to be installed. Run `pip install kernels`.rdr^r_s r rrs l   r"c td)NzVregister_kernel_mapping requires `kernels` to be installed. Run `pip install kernels`.rdrls r rrsstttr"c td)Nzcregister_kernel_mapping_transformers requires `kernels` to be installed. Run `pip install kernels`.rdrls r rYrYs q   r"zkernels-community/causal-conv1d)r1rzkernels-community/mamba-ssmz!kernels-community/finegrained-fp8zkernels-community/deep-gemmzkernels-community/sonic-moez ep-support)r1revision)z causal-conv1dz mamba-ssmzfalcon_mamba-ssmzfinegrained-fp8z deep-gemmz sonic-moe_HUB_KERNEL_MAPPING_KERNEL_MODULE_MAPPINGattn_implementationreturnc8|duotjd|duS)zJCheck whether `attn_implementation` matches a kernel pattern from the hub.Nz'^[^/:]+/[^/:]+(?:@[^/:]+)?(?::[^/:]+)?$)research)rrs r is_kernelrw*s. 4' c I@BU V V^b br"attention_wrapperallow_all_kernelscddlm}ddlm}d|vr|ddn|}t |sdSt stdd|vr-|d\}}|}n|}d}|}| d \}}} |}| r| nd} t|| | } n(#t$r} td |d | d d} ~ wwxYwt| dr |t}|} n|t| |} ||| |||d| S)a Load and register the kernel associated to `attn_implementation`. Args: attn_implementation: A string, usually a kernel repo like "kernels-community/flash-mla". attn_wrapper: a callable for the wrapper around the attention implementation. In `transformers` we have a wrapper around the `flash_attn_var_len` call, and the same goes for `sdpa` and `eager`. They just prepare the arguments properly. This is mostly used for continious batching, where we want the `paged` wrapper, which calls the paged cache. allow_all_kernels (`bool`, optional): Whether to load kernels from unverified hub repos, if it is a custom kernel outside of the `kernels-community` hub repository. r)ALL_MASK_ATTENTION_FUNCTIONS)ALL_ATTENTION_FUNCTIONS|r N`kernels` is either not installed or uses an incompatible version. Please install the latest version with `pip install -U kernels`.:@)roryz-An error occurred while trying to load from 'z': .flash_attn_varlen_funcflash_attention_2) masking_utilsr{modeling_utilsr|splitrw_kernels_availablerWstrip partitionr Exception ValueErrorhasattrr getattrregister) rrrxryr{r|actual_attn_namer1 kernel_name_revkernelekernel_functions r load_and_register_attn_kernelr2s =<<<<<888888x-H-HHHkHgZdeeeekHgNNNNr" module_namescBttrgfd}|S)aO This decorator attaches the target function within the module as a plain attribute (not as a submodule). Keep in mind that this registration is only meant for `kernelize` to recognize its target modules (i.e. function exchanged for a weightless `nn.Module` with the same forward) to then exchange to the kernel variation (in-place) if the conditions are met. We cache each of these function-based registrations: After proper registration and exchange it is removed from the module's `_modules` dict as it does not really act as `nn.Module` but a base function. c2|jfd}||_|S)Nc|g|Ri||jdi}D]N}t|ddp!t|ddpt|dd}|td||||<OdS)N_hidden_kernelsrhkernel_layer_namer(z)Could not infer kernel function name for )__dict__ setdefaultrr)rfr^r_hidden_kernelsfnnamer orig_inits r new_initz8use_kernelized_func..decorator..new_inits Id ,T , , ,V , , ,"]556GLLN" * *B D116r#6==6r;55 <$%WQS%W%WXXX(*t$$ * *r")rg)rrrrs @r r\z&use_kernelized_func..decorators7L  * * * * * *"   r")rKr)rr\s` r use_kernelized_funcrs?,))&$~ . r"c#*K dadVdadS#dawxYw)z Context manager used to adjust the value of the global `ALLOW_HUB_KERNELS`. This is needed, as this argument cannot be forwarded directly to the `__init__` of the models, where we set the attention implementation. TNF)ALLOW_ALL_KERNELSrr"r allow_all_hub_kernelsrs:"  "E!!!!s) rrrrrYrrrrr)NF)NNF)Aimportlib.metadatarosrucollections.abcr contextlibrtypesr packagingrrutilsrr utils.import_utilsr flash_attentionr get_loggerrhr%rr rrrrrrrr$rrWrr.r-environrupperr'rr#str INFERENCETRAINING TORCH_COMPILErFrL__annotations__rMrYrprqboolrwrrintlistrrr__all__rr"r rs,  $$$$$$%%%%%%,,,,,,11111111555555444444  H % %}  ******* .YYYYYY(,%% ...(-%%%.%'JNN3De$L$L$R$R$T$T!48LL##### %C % % % %" OO;:*  OO/* =-!!!=-!!!7(!!!;(!!!=-!!!5   D OO2+  :1   :1!!! 11!!!:1!!!:4!!!/  > !33__:)666  !33__:*666  !33__:(666  !33__:vWX666  !33__:vWX666  !33__:z[\666 wBeBeOT#tFSL/DI^D_2_$_``aBBBJ!6B]!!!  ~~6B]   6B]!!!  - - () \\\)))))! ! !   qqqqqqqq   uuu     =! J"CqQQ:qII$AaPP#FSTUU:qII: UU 22T#tCH~-.8:S*t"3349993:$di:::19D:\`:$::::zPf- - #- S*t:K5K0L- - - - d $# OOODjO3Y O O  OOOO:$d8nx&?$$$$P " " "   s[ K&A-,K-A74K6A77K;BKB K BIK%LL