|-jitddlmZddlmZddlmZddlmZddlm Z ddl m Z m Z m Z mZdd lmZe r8dd lZeje Z eje Z ejeZ d7d8dZd9dZd:dZd:dZdZdZe rVejd ed!d"#ejd eej d ee$d;d%Z!d:d&Z" d7de&d-d d d-d.d?d6Z(d S)@) annotations)Callable)wraps)logging)GeneralInterface)is_torch_availableis_torch_greater_or_equalis_torch_less_or_equalis_torchdynamo_compiling)sonicmoe_experts_forwardNFinput torch.Tensorweightbiastorch.Tensor | None is_transposedboolreturnc&|r>i 3 344<> ) ) )+v$))A,,+v+vflfqfqrsftft+v+v ) ) ) ::a==FKKNN * * *UAUUV[[QR^^UU + * * :%+u{3 3 3 35h\`\f5h5h 3 3 3 ;uzz!}}fkk!nnU\QVQ\ ] ] ]]r!cd||d|d|d|_dS)zjSaves input and weight for backward; offs is stored directly as it is a non-differentiable integer tensor.rr rN)save_for_backwardrJ)ctxinputsrSs r"_grouped_mm_fallback_setup_contextres/&)VAY///ayCHHHr!c|j\}}tj|}tj|}d}t|jD]r\}}||kr tj|||||j|||tj|||j||||||}s||dfS)zuBackward pass for `_grouped_mm_fallback`. Computes grad_input and grad_weight per expert group; offs has no gradient.rrNN) saved_tensorsr zeros_likerPrJrQrRT) rc grad_outputrr grad_input grad_weightrTrUrVs r_grouped_mm_fallback_backwardrms%ME6!%((J"6**K ECHOO--..3 C<<  U3Y'*U3Y:OPPPP uSy!#[s%;QPPPP {D ((r!z!transformers::grouped_mm_fallbackz4(Tensor input, Tensor weight, Tensor offs) -> Tensor) mutates_argsschema) setup_contextctr|jtjksW|jjdkrIt ddr8|dzdks|dzdkrdS|jjdkrttj j d r(tj |jd kSttd ratd dr(tj |jd kStj |jd kSdSttj j d pttd S)a Check if torch.nn.functional.grouped_mm or torch._grouped_mm can be used based on availability and compatibility with torch.compile. Args: input (`torch.Tensor`): Input tensor of shape (S, input_dim). weight (`torch.Tensor`): Weight tensor of shape (num_experts, input_dim, output_dim). offs (`torch.Tensor`): Offsets tensor indicating the boundaries of each group in the input tensor. Returns: `bool`: True if grouped_mm can be used, False otherwise. cpuz2.10.0T) accept_devrFcuda grouped_mm)r _grouped_mmz2.9) r)r r=rbfloat16rMtyper data_ptrhasattrnn functionalrvget_device_capabilityr r_s r_can_use_grouped_mmrsT !"" v|u~'E'E e## "8 = = = $__   #q ( (ENN,<,r?r@rBrC expert_ids_gpermselected_hidden_states_gsample_weights_g histc_inputtokens_per_expertoffsets sentinel_maskrDrErFrGinv_permrHs rgrouped_mm_experts_forwardrzs   !F  $$I##A&&J##B''J#**2..N$$R((JJ//L$,TY->?%d+ +1+*G*G,$$&&&\M]M]M_M_K Kd6FASWScfgSghhhl,!5;GGGG""T%55@@DDMD,q0111 }U,BF-Y$0>>UY<=A]T$+L99PT))-=== "2G/aeasH  })##H--;;x((~;?=Rd),77dO"G/QUQcH .88<<>>HTN)L '++J :NNRRWXRYY  ! !-"5 6 66r!c0eZdZdZeeedZd fd ZxZ S) ExpertsInterfacez;Interface for registering custom experts forward functions.) batched_mmrwsonicmoeexperts_implementationstrdefaultrrc|tdn|dkr||vrtd|dt||S)zfReturn the requested `experts_implementation`. Also strictly check its validity, and raise if invalid.Na You tried to access the `ExpertsInterface` with a `config._experts_implementation` set to `None`. This is expected if you use an Expert Module as a standalone Module. If this is not the case, something went wrong with the dispatch of `config._experts_implementation`eager`zL` is not a valid experts implementation registered in the `ExpertsInterface`)logger warning_onceKeyErrorsuperget)r"rr __class__s r get_interfacezExpertsInterface.get_interfaces ! )   N     $w . .3IQU3U3Ux*xxx ww{{17;;;r!)rrrrrr) __name__ __module__ __qualname____doc__rIrr_global_mappingr __classcell__)rs@rrrsZEE10,O < < < < < < < < < dfd }| ||S|S)aDecorator to modify experts class to support different experts implementations. Args: experts_class (`type[torch.nn.Module]`, *optional*): The experts class to modify. If not provided, returns a decorator that can be applied to the class. experts_interface (`ExpertsInterface`, *optional*, defaults to `ALL_EXPERTS_FUNCTIONS`): The experts interface to use for dispatching the forward method. is_concatenated (`bool`, *optional*, defaults to `True`): Whether the expert weights are stored in concatenated layout [gate;up] or interleaved layout [gate0, up0, gate1, up1, ...]. is_transposed (`bool`, *optional*, defaults to `False`): Whether the expert weights are stored in transposed format. has_bias (`bool`, *optional*, defaults to `False`): Whether the expert layers include bias terms or not. has_gate (`bool`, *optional*, defaults to `True`): Whether the experts use a gating mechanism or not. Whether it has gate_up_proj weights or just up_proj weights. Returns: `type[torch.nn.Module]`: The modified experts class. rrrc|j|jt fd}tfd}t|ds t|_||_||_|S)Nch||g|Ri|||_|_|_|_|_dSN)configr0r2rr) r"rargskwargsr2r0rr original_inits r__init__z=use_experts_implementation..wrapper..__init__.sP M$ 8 8 8 8 8 8 8 DK$DM$DM!.D #2D r!c\|jj}||g|Ri|Sr)rr_experts_implementation)r"rrexperts_forwardroriginal_forwards rforwardz.wrapper..forward7s>/==dk>acsttO"?49$999&99 9r!r6)rrrr~rr6) rrrrrrr2r0rrs @@rwrapperz+use_experts_implementation..wrapper*s%. (0 }   3 3 3 3 3 3 3 3   3   : : : : : !  :}m44 <(;M %!)  ' r!N)rrrrrn)rrrrr2r0rs ````` ruse_experts_implementationr sV>2 w}%%% Nr!)NF) rrrrrrrrrr) r"r#r$rr%rr&rrr)rrrrrJrrr)rrrrrJrrr) rrrrrJrrrrrrr)rrrrr)rrrrrrrrr2rr0rrr)) __future__rcollections.abcr functoolsrutilsr utils.genericrutils.import_utilsr r r r rrr_dynamoassume_constant_result get_loggerrrr rIrWr`rermlibrary custom_op register_fakeregister_autogradrryrrrALL_EXPERTS_FUNCTIONSrrrnr!rrs#"""""$$$$$$,,,,,, /.....ZLLL !& D DE^ _ _"]AABXYY  H % %\!% D<7<7<7<7D4 ^ ^ ^ ^ )))&  M+E   M CE^___ M##+%8$*W*W*W*WZPPPPF!% #####Le7e7e7e7P<<<<<'<<<0)(** " " " "37;+@ ;;;;;;;;r!