}-jf)ddlmZddlmZmZmZmZddlmZddl m Z erddl Z er ddl m Z dd lmZejeZGd d eZdS) ) TYPE_CHECKING)is_accelerate_availableis_torch_availableis_torch_xpu_availablelogging) HfQuantizer)get_module_from_nameN)PreTrainedModel)FineGrainedFP8ConfigceZdZUdZdZded<fdZdZddd ed e fd Z ddd ed d d e ffd Z ddZ dZdZed e fdZed e fdZdZdZdZxZS)FineGrainedFP8HfQuantizerz FP8 quantization implementation supporting both standard and MoE models. Supports both e4m3fn formats based on platform. Fr quantization_configc <tj|fi|dS)N)super__init__)selfrkwargs __class__s q/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/transformers/quantizers/quantizer_finegrained_fp8.pyrz"FineGrainedFP8HfQuantizer.__init__s),7777777crtstd|jjrdStjsLts>|jr(t dd|j_dStdtjrdtj }|\}}|dks |dkr5|dkr/t d|d|d d|j_dS| d }|t d dSt|trU|js)t!|d krd |vsd|vrt%ddSdS)NzMLoading an FP8 quantized model requires accelerate (`pip install accelerate`)zUsing FP8 quantized models requires a GPU or XPU, we will default to dequantizing the model to bf16 since no GPU or XPU is availableTzANo GPU or XPU found. A GPU or XPU is needed for FP8 quantization. ziFP8 quantized models is only supported on GPUs with compute capability >= 8.9 (e.g 4090/H100), actual = `.z`. We will default to dequantizing the model to bf16. Feel free to use a different quantization method like bitsandbytes or torchao device_mapzYou have loaded an FP8 model on CPU and have a CUDA or XPU device available, make sure to set your model on a GPU or XPU device in order to run your model. To remove this warning, pass device_map = 'cuda' or 'xpu'. r cpudiskzYou are attempting to load an FP8 model with a device_map that contains a cpu/disk device.This is not supported when the model is quantized on the fly. Please use a quantized checkpoint or remove the cpu/disk device from the device_map.)r ImportErrorr dequantizetorchcuda is_availabler pre_quantizedlogger warning_once RuntimeErrorget_device_capabilityget isinstancedictlenvalues ValueError)rargsrcompute_capabilitymajorminorrs rvalidate_environmentz.FineGrainedFP8HfQuantizer.validate_environments &(( omnn n  # .  Fz&&(( h1G1I1I h! h##[7;(3"#fggg : " " $ $ !&!A!A!C!C -LE5 uzzeaii##[#([[+0[[[ 7;(3ZZ --     6       D ) ) &  OOa''Z..0000Z..0000 k   10rmodelr param_namereturnc ddlm}m}t||\}}t |||fr|js|dkrdSdSdS)Nr) FP8Experts FP8LinearbiasFT)integrations.finegrained_fp8r9r:r r+r%)rr5r6rr9r:module tensor_names rparam_needs_quantizationz2FineGrainedFP8HfQuantizer.param_needs_quantizationOsiHHHHHHHH25*EE fy*5 6 6 ! [F%:%:uturparamz torch.Tensorcz|||rdSt|||S)z4Return the element size (in bytes) for `param_name`.r )r?rparam_element_size)rr5r6r@rs rrBz,FineGrainedFP8HfQuantizer.param_element_sizeZs<  ( ( ; ; 1ww))%UCCCrc ddlm}|||jj|j|_|||j|j|j}dS)Nr)replace_with_fp8_linear)modules_to_not_convertrr%)r<rDget_modules_to_not_convertrrE_keep_in_fp32_modulesr%)rr5rrDs r$_process_model_before_weight_loadingz>FineGrainedFP8HfQuantizer._process_model_before_weight_loadingasq KJJJJJ&*&E&E 4+BED_' ' #(' #'#> $ 8,    rcRd|jjvrddddddddddddddd}||_|S)NQwen3colwiserowwise)z layers.*.self_attn.q_proj.weightz*layers.*.self_attn.q_proj.weight_scale_invz layers.*.self_attn.k_proj.weightz*layers.*.self_attn.k_proj.weight_scale_invz layers.*.self_attn.v_proj.weightz*layers.*.self_attn.v_proj.weight_scale_invz layers.*.self_attn.o_proj.weightz*layers.*.self_attn.o_proj.weight_scale_invzlayers.*.mlp.gate_proj.weightz'layers.*.mlp.gate_proj.weight_scale_invzlayers.*.mlp.up_proj.weightz%layers.*.mlp.up_proj.weight_scale_invzlayers.*.mlp.down_proj.weightz'layers.*.mlp.down_proj.weight_scale_inv)r__name__base_model_tp_plan)rconfig text_plans rupdate_tp_planz(FineGrainedFP8HfQuantizer.update_tp_planssX f&/ / /4=>G4=>G4=>G4=>G1:;D/89B1:;DI")2F % rcdSNTrs ris_serializablez)FineGrainedFP8HfQuantizer.is_serializablestrcdS)NFrTrUs r is_trainablez&FineGrainedFP8HfQuantizer.is_trainablesurcdSrSrTrUs ris_compileablez(FineGrainedFP8HfQuantizer.is_compileablestrc$ddlm}||S)Nr) Fp8Quantize)r<r\)rr\s rget_quantize_opsz*FineGrainedFP8HfQuantizer.get_quantize_opss$>>>>>>{4   rczddlm}ddlm}|jr'|jjr|gdd||ggSgS)Nr)WeightConverter Fp8Dequantize)zweight$weight_scale_invactivation_scaleweightsource_patternstarget_patterns operations)core_model_loadingr_r<rar%rr!)rr_ras rget_weight_conversionsz0FineGrainedFP8HfQuantizer.get_weight_conversionss888888@@@@@@   $":"E  $W$W$W$, - d 3 34  rcx|jr |jjs||zSddlm}m}ddlm}|dd}|gt|z}g}|D]}t||s| |(d|j D}|red|D} d |D} d |j D} | | z| z} ||gt|j z} || |j| }| ||||S) uWhen loading with ``dequantize=True``, attach an :class:`Fp8Dequantize` op to every existing :class:`WeightConverter` so that per-block scales are folded into the weight *before* any later merge/concat ops collapse the per-expert structure. For each model-supplied converter that has a ``.weight`` source, we: 1. anchor the existing weight patterns with ``$`` so they don't accidentally also match the ``.weight_scale_inv`` keys (the regex is searched, so the unanchored prefix would match both, sending scales to the wrong bucket); 2. add anchored ``*.weight_scale_inv`` sources next to each weight pattern so the loader collects scale tensors alongside the weight tensors into the *same* converter bucket (both keys rewrite to the same target); 3. prepend a fresh :class:`Fp8Dequantize` op so dequant runs first, before any merge/concat collapses the per-expert structure. The generic ``weight$ + weight_scale_inv → weight`` converter from :meth:`get_weight_conversions` is still appended at the end as a fallback for plain ``nn.Linear`` weights with no model-specific converter. r)r_WeightRenamingr`z ^(.+)\.scale$z\1.weight_scale_inv)rfrgc<g|]}|d|S.weightendswith.0ps r zGFineGrainedFP8HfQuantizer.update_weight_conversions..s)WWWAIAVAVWaWWWrcg|]}|dzS)$rTrrs rruzGFineGrainedFP8HfQuantizer.update_weight_conversions..s"C"C"Cq1s7"C"C"CrcDg|]}|dtd dzS)Nroz.weight_scale_inv$)r-rrs rruzGFineGrainedFP8HfQuantizer.update_weight_conversions..s2 e e eQR#4c)nn_#4!58L!L e e erc<g|]}|d|Srnrprrs rruzGFineGrainedFP8HfQuantizer.update_weight_conversions..s)VVVq 9@U@UVVVVrre)r%rr!rjrir_rlr<ralistr+appendrfrh_original_target_patternsextend)rweight_conversionsr_rlra scale_renameupdatedconvweight_sourcesanchored_weight scale_sourcesother new_sourcesnew_opss rupdate_weight_conversionsz3FineGrainedFP8HfQuantizer.update_weight_conversionss&" Ft'?'J F%(C(C(E(EE EHHHHHHHH@@@@@@&~6FXnooo *^d3E.F.FF& ! !DdO44 t$$$WW)=WWWN "C"CN"C"C"C e eVd e e e VVD$8VVV- =E (=../$t2G2GG&$/$($B& NN4 t2244555r)r5r )rM __module__ __qualname____doc__requires_calibration__annotations__rr4strboolr?floatrBrHrQrVpropertyrXrZr]rjr __classcell__)rs@rrrs !////88888///b .? S _c    D(9DsDSaDfkDDDDDD      $.dXX!!!  8888888rr)typingrutilsrrrrbaser quantizers_utilsr r"modeling_utilsr utils.quantization_configr get_loggerrMr&rrTrrrs ````````````222222LLLA000000@@@@@@  H % %PPPPP PPPPPr