ёi8%SSKJr SSKJrJr SSKrSSKJr SSKJr SSK J r SSK J r SSK JrJr \(a&SS KJr SS KJr SS KJr \S rS \S'\SrS \S'SrSSSjjrSSSjjrSSSjjrSSSjjrSSjrg) ) annotations) TYPE_CHECKINGLiteralN)_C_ops) check_dtype)is_compiled_with_cuda)get_device_capability) LayerHelperin_dynamic_or_pir_mode) TypeAlias)Tensor) DTypeLike)weight_only_int8weight_only_int4zllm.int8r _Algo)@ _GroupSizec[5(ak[RR5nUbUS:wd[R"5(a[ 5up[ US-U-5nU$[S5eg)NFalse zmPaddle is not compiled with CUDA, we cannot get SMVersion from device, please try to compile Paddle with CUDAr)rpaddleversioncudais_compiled_with_rocmr int ValueError) cuda_versionmajorminorarchs `/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/nn/quant/quantized_linear.py_get_arch_infor$+sp~~**,  $)@  ) ) + +02LEurzE)*DK  c Uc [5n[5(a;US:Xd5US:Xd/US:Xd)US:Xd#US:XdUS:XdUS:XdUS:Xd S US 35eUS :XdUS :XdUS :Xd SUS 35e[5(a[R"XX#5$Sn[ U40[ 5D6nURS5nURS5nURUSU0XgS.XUS.S9 Xg4$)a" Quantization function for weight_only and llm.int8's weight. Args: x (Tensor): The input Tensor to be quantized, the data type is float16 or bfloat16. algo (str): The algo that is x will be apply, must be one of 'weight_only_int8', 'weight_only_int4', 'llm.int8', 'w4a8' and 'w4afp8, default: 'weight_only_int8'. arch (int): The compute arch for target device. For example, A100 is 80, v100 is 70, if you do not assign arch, we will get arch from your device, default: None. group_size (int): The group size for weight quantization. -1 stands for default per-channel mode. Currently only support 64 or 128. Returns: out (Tensor): The Tensor which is the quantitative results, the data type is int8, the shape is transposition of x. scale (Tensor): The scale Tensor which is the scale of pre-channel, the data type is float32. Examples: .. code-block:: pycon >>> # doctest: +SKIP('No testing required') >>> import paddle >>> from paddle.nn.quant import weight_quantize >>> paddle.seed(2023) >>> x = paddle.rand(shape=[64, 32], dtype=paddle.float16) >>> out, scale = weight_quantize(x, algo='weight_only_int8') >>> print(out.shape) paddle.Size([32, 64]) >>> print(scale.shape) paddle.Size([32]) FKPVYZ\dKCurrently weight_quantize only support SM70/75/80/86/89/90/92/100. but got  rrr5Currently group_size only support -1/64/128. but got weight_quantizeint8floatx)outscale)algor" group_sizetypeinputsoutputsattrs) r$rr rr2r locals"create_variable_for_type_inference append_op)r5r8r"r9r;helperr6r7s r#r2r2>s.D | BJrzrzrzrzrzrzs{ ZZ^Y__` a    zR/:3D ? |1M D%%at@@ T.VX.77?99'B80ZH  |r%c$US:XdUS:XdUS:Xd SUS35e[5(a[R"XX$5$Sn[U40[ 5D6nUR nUR U5nURUXS.SU0UUS .S 9 U$) aV Dequantization function for weight_only and llm.int8's weight. Args: x (Tensor): The input Tensor to be dequantized, the data type is int8. scale (Tensor): The scale Tensor which is the output of weight_quantize, the data type is float32. algo (str): The algo that is x will be apply, must be one of 'weight_only_int8', 'weight_only_int4' and 'llm.int8', default: 'weight_only_int8'. out_dtype (str|np.dtype): [Deprecated][Not used] The output Tensor's data type, must be one of 'float16' and 'bfloat16', default: 'float16'. Returns: out (Tensor): The Tensor which is the dequantitative results, the data type is float16 or bfloat16, the shape is transposition of x. Examples: .. code-block:: python >>> # doctest: +SKIP('No testing required') >>> import paddle >>> from paddle.nn.quant import weight_quantize, weight_dequantize >>> paddle.seed(2023) >>> x = paddle.rand(shape=[64, 32], dtype=paddle.float16) >>> out, scale = weight_quantize(x, algo='weight_only_int8') >>> x_dequant = weight_dequantize(out, scale) rrrr1r0weight_dequantize)r5r7r6)r8r9r:)r rrDr r?dtyper@rA)r5r7r8 out_dtyper9r;rBr6s r#rDrDs@  zR/:3D ? |1M D''$CC"T.VX.KK 77 B+CL(   r%c  Uc [5n[5(a;US:Xd5US:Xd/US:Xd)US:Xd#US:XdUS:XdUS:XdUS:Xd S US 35eUS :XdUS :XdUS :Xd SUS 35e[5(a[R"XX#XEU5nU$[ USSS/S5 Sn[ U40[5D6n URn U/U/U/S.n UbU/U S'UUUS.n U RU 5nU RUU SU0U S9 U$)ab Applies matrix multiplication of two tensors and then bias addition if provided. This method requires CUDA version >= 11.2. Args: x (Tensor): The first input Tensor to be multiplied, the data type is float16 or bfloat16. weight (Tensor): The second input Tensor to be multiplied. Its rank must be 2. bias (Tensor|None): The input bias Tensor. If it is None, no bias addition would be performed. Otherwise, The bias is added to the matrix multiplication result. weight_scale (Tensor|None): The input scale Tensor Provided to weight for dequantization. Its rank must be 1. weight_dtype(str): The dtype of weight Tensor, must be one of 'int8', 'int4', Defaulted to 'int8'. arch (int): The compute arch for target device. For example, A100 is 80, v100 is 70, if you do not assign arch, we will get arch from your device, default: None. group_size (int): The group size for weight quantization. -1 stands for default per-channel mode. Currently only support 64 or 128. Returns: Tensor: the output Tensor, the data type is the same as that of x. Examples: .. code-block:: pycon >>> # doctest: +SKIP('No testing required') >>> import paddle >>> from paddle.nn.quant import weight_only_linear >>> x = paddle.cast(paddle.randn([1, 2, 64]), dtype='float16') >>> weight = paddle.cast(paddle.randint(0, 127, [32, 64]), dtype='int8') >>> scale = paddle.randn([32], dtype='float32') >>> bias = paddle.cast(paddle.randn([32]), dtype='float16') >>> if paddle.device.cuda.get_device_capability()[0] >= 8: ... out = weight_only_linear( ... x, ... weight, ... bias=bias, ... weight_scale=scale, ... weight_dtype='int8', ... ) ... print(out.shape) paddle.Size([1, 2, 32]) r'r(r)r*r+r,r-r.r/r0rrrzLCurrently weight_quantize only support group size of -1, 64 or 128. but got weight_dtyper3int4weight_only_linearr5weight weight_scalebias)rHr"r9r6r:) r$rr rrJrr r?rEr@rA) r5rLrNrMrHr"r9r6r;rBrEr<r>s r#rJrJs^ | BJrzrzrzrzrzrzs{ ZZ^Y__` a    zR/:3D VWaVbbcd D'' tCL   r%c[5(a[R"XX#U5nU$Sn[U40[ 5D6nUR nU/U/U/S.n U(aU/U S'SU0n UR U5nURUU SU0U S9 U$)a Applies matrix multiplication of two tensors and then bias addition if provided. This method requires CUDA version >= 11.2. Args: x (Tensor): the first input Tensor to be multiplied, the data type is float16 or bfloat16. weight (Tensor): the second input Tensor to be multiplied. Its rank must be 2. bias (Tensor|None): the input bias Tensor. If it is None, no bias addition would be performed. Otherwise, the bias is added to the matrix multiplication result. weight_scale (Tensor|None): the input scale Tensor Provided to weight for dequantization. Its rank must be 1. threshold(float): The min value of outlier in activation, outlier's channel will be apply multiply with x.dtype. Returns: Tensor: the output Tensor, the data type is the same as that of x. Examples: .. code-block:: pycon >>> # doctest: +SKIP('No testing required') >>> import paddle >>> from paddle.nn.quant import llm_int8_linear >>> x = paddle.cast(paddle.randn([1, 2, 64]), dtype='float16') >>> weight = paddle.cast(paddle.randint(0, 127, [32, 64]), dtype='int8') >>> scale = paddle.randn([32], dtype='float32') >>> bias = paddle.cast(paddle.randn([32]), dtype='float16') >>> if paddle.device.cuda.get_device_capability()[0] >= 8: ... out = llm_int8_linear(x, weight, bias=bias, weight_scale=scale, threshold=6.0) ... print(out.shape) paddle.Size([1, 2, 32]) llm_int8_linearrKrN thresholdr6r:)r rrPr r?rEr@rA) r5rLrNrMrQr6r;rBrEr<r>s r#rPrP%sL$$QIN  T.VX.h)N  "VF6Ni(77>CL   r%c[5(a[R"X5$Sn[U40[ 5D6nUR UR 5nURUU/U/S.SU0S9 U$)a Apply pre-quant per channel scale on activations Args: x (Tensor): Input tensor representing the activations, the data type can be float16 or bfloat16. scales(Tensor): Per-channel scale factors for pre-quantization. Data type should be compatible with x. Returns: out (Tensor): The Tensor which is the pre-quant results, the data type is compatible with x. Examples: .. code-block:: python >>> # doctest: +SKIP('No testing required') >>> import paddle >>> from paddle.nn.quant import apply_per_channel_scale >>> paddle.seed(2023) >>> x = paddle.rand(shape=[64, 32], dtype=paddle.float16) >>> scales = paddle.rand(shape=[32], dtype=paddle.float16) >>> out = apply_per_channel_scale(x, scales) apply_per_channel_scale)r5scalesr6)r;r<r=)r rrSr r?r@rErA)r5rTr;rBr6s r#rSrSgsx0--a88(T.VX.77@1CL   r%)rNr) r5r r8rr" int | Noner9rreturnztuple[Tensor, Tensor])rfloat16r) r5r r7r r8rrFrr9rrVr )NNr3Nr)r5r rLr rN Tensor | NonerMrXrHrr"rUr9rrVr )NNg@) r5r rLr rNrXrMrXrQr4rVr )r5r rTr rVr ) __future__rtypingrrrrpaddle.base.data_feederr paddle.devicerpaddle.device.cudar paddle.frameworkr r typing_extensionsr r paddle._typingrr__annotations__rr$r2rDrJrPrSr%r#rcs#) /5 +(:E9$K0J 0*% D D D D D  DT%$ 5 5 5 5 5  5  5v"&$e e e e e  e  ee eV"& ? ? ? ? ?  ?  ?D$r%