ϑi?SSKJr SSKrSSKJr SSKrSSKJrJr SSKJ r \(aSSK J r \RSSj5r SSSjjrSS jrSSS jjrSS jrSSS jjrSSS jjrSSSjjrSSSjjrg)) annotationsN) TYPE_CHECKING)Tensor_C_ops)in_dynamic_or_pir_mode)Sequencec[5$)z&Get tensor with no entries and no data)ra/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/incubate/nn/functional/fp8.py _empty_tensorr s  8Or c[5(a3U(a[R"U5$[R"U5$g)a Fused operation that performs stacking, optional transposition, and quantization on a list of bfloat16 tensors. Args: x (list[Tensor] or tuple[Tensor]): A list or tuple of bfloat16 tensors, where each tensor has shape `[M, K]`. All tensors should have the same shape and dtype. transpose (bool, optional): If True, applies a transpose before quantization. Default is True. Returns: tuple: - out (Tensor): The quantized output tensor with dtype `float8_e4m3fn`. - scale (Tensor): A float32 tensor representing the quantization scale. Examples: .. code-block:: pycon >>> # doctest: +SKIP('BF16 requires SM80 or higher env') >>> import paddle >>> import paddle.incubate.nn.functional as F >>> paddle.set_device('gpu') >>> x_vec = [] >>> num_experts = 1 >>> seq_len = 2048 >>> hidden_size = 128 >>> for _ in range(num_experts): ... x = paddle.randn([seq_len, hidden_size], dtype='bfloat16') ... x = paddle.clip(x, min=-50, max=50) ... x_vec.append(x) >>> out, scale = F.fused_stack_transpose_quant(x_vec, transpose=True) >>> print(out.shape) paddle.Size([128, 2048]) >>> print(scale.shape) paddle.Size([1, 16]) N)rrfused_stack_transpose_quantfused_stack_quant)x transposes r rr#s8R 55a8 8++A. .  r cN[5(a[R"X5$g)av Applies fused activation and dequantization operation to convert float8 quantized data back to bfloat16. Args: x (Tensor): Input quantized tensor with dtype float8_e4m3fn and shape [M, N]. This tensor contains the quantized activations from previous layers. x_scale (Tensor): Dequantization scale tensor with dtype float32 and shape [M, (N + 127) // 128]. Each scale value corresponds to a 128-column block in the input tensor. Returns: Tensor. Dequantized output tensor with dtype bfloat16 and shape [M, N]. The values are computed as input * scale for each corresponding 128-column block. N)rrfused_act_dequant)rx_scales r rrSs#"''33 r cP[5(a[R"XU5$g)u^ Computes gradients for fused weighted SwiGLU activation function in backward pass. Note: This function performs the backward propagation for the SwiGLU (Swish-Gated Linear Unit) activation with probability weighting. It computes gradients with respect to both the input activations and the probability weights, while also recomputing forward pass values for memory efficiency. The kernel automatically selects between vectorized and standard implementations based on input dimensions. Args: o1 (Tensor): Forward pass input tensor with dtype bfloat16 and shape [..., intermediate_size * 2]. The tensor is split into two halves: - Left half [0:intermediate_size]: x1 values (gate inputs) - Right half [intermediate_size:]: x2 values (activation inputs) This is the same input used in the forward SwiGLU computation. do2_s (Tensor): Upstream gradient tensor with dtype bfloat16 and shape [..., intermediate_size]. Contains gradients flowing back from the next layer, representing ∂L/∂output before probability weighting. Each element corresponds to the gradient of one output element. unzipped_probs (Tensor): Probability weighting tensor with dtype float32 and shape matching the batch dimensions of o1 and do2_s [...]. Each probability value was used to weight the corresponding row's output in the forward pass. Returns: tuple: - do1 (Tensor). Input gradients with dtype bfloat16 and shape [..., intermediate_size * 2]. Layout matches o1: - [0:intermediate_size]: ∂L/∂x1 (gradients w.r.t. gate inputs) - [intermediate_size:]: ∂L/∂x2 (gradients w.r.t. activation inputs) - probs_grad (Tensor). Probability gradients with dtype float32 and shape [...]. Each element is ∂L/∂prob for the corresponding batch item, computed as the sum of (∂L/∂output_i * SwiGLU_output_i) across the intermediate dimension. - o2_s (Tensor). Recomputed forward output with dtype bfloat16 and shape [..., intermediate_size]. Contains SwiGLU(x1, x2) * prob values. This avoids storing forward activations, trading computation for memory. Examples: .. code-block:: pycon >>> # doctest: +SKIP('BF16 requires SM80 or higher env') >>> import paddle >>> import paddle.incubate.nn.functional as F >>> paddle.set_device('gpu') >>> batch_size, seq_len = 32, 128 >>> intermediate_size = 2048 >>> o1 = paddle.randn( ... [batch_size, seq_len, intermediate_size * 2], ... dtype='bfloat16', ... ) >>> do2_s = paddle.randn([batch_size, seq_len, intermediate_size], dtype='bfloat16') >>> expert_probs = paddle.rand([batch_size, seq_len, 1], dtype='float32') >>> do1, probs_grad, o2_s = F.fused_swiglu_weighted_bwd(o1, do2_s, expert_probs) >>> print(do1.shape) paddle.Size([32, 128, 4096]) >>> print(probs_grad.shape) paddle.Size([32, 128, 1]) >>> print(o2_s.shape) paddle.Size([32, 128, 2048]) N)rrfused_swiglu_weighted_bwd)o1do2_sunzipped_probsnames r rrhs&N//>JJ r cUVs/sHn[U5PM nnURSS:XdURSS:Xa//4$[5(a[R"XX#5$gs snf)a Applies fused transpose, split, and quantization operation for Mixture of Experts (MoE) models. Note: This function performs three operations in a single optimized CUDA kernel: 1. Quantizes input from bfloat16 to float8_e4m3fn format using column-wise scaling 2. Transposes the matrix from [M, K] to [K, M] layout 3. Splits the transposed data across multiple experts based on token distribution Args: x (Tensor): Input tensor of shape [M, K] with dtype bfloat16, where M is the total number of tokens and K is the feature dimension. M must be divisible by 128 for optimal performance. tokens_per_expert (List[int]): List containing the number of tokens assigned to each expert. Each value should be a multiple of 128 for optimal performance. The sum should equal M (total tokens). Values can be 0 for unused experts. pow_2_scales (bool, optional): Whether to constrain quantization scales to powers of 2 for better hardware efficiency. If True, scales will be rounded to the nearest power of 2. Default: False. Returns: tuple: - outs (List[Tensor]). List of quantized and transposed output tensors, one per expert. Each tensor has shape [K, tokens_per_expert[i]] and dtype float8_e4m3fn. Empty tensors are included for experts with 0 tokens. - scales (List[Tensor]). List of dequantization scale tensors, one per expert. Each tensor has shape [K // 128, tokens_per_expert[i] // 128] and dtype float32. These are the reciprocal of quantization scales. Examples: .. code-block:: pycon >>> # doctest: +SKIP('BF16 requires SM80 or higher env') >>> import paddle >>> import paddle.incubate.nn.functional as F >>> paddle.set_device('gpu') >>> x = paddle.randn([384, 512], dtype='bfloat16') >>> x = paddle.clip(x, min=-50, max=50) >>> tokens_per_expert = [128, 128, 128] >>> outs, scales = F.fused_transpose_split_quant(x, None, tokens_per_expert, pow_2_scales=True) >>> print(outs[0].shape) paddle.Size([512, 128]) >>> print(scales[0].shape) paddle.Size([1, 512]) rN)intshaperrfused_transpose_split_quant)r input_scalestokens_per_expert pow_2_scalests r r r srd*;;):AQ):;wwqzQ!''!*/2v 11 .   >> # doctest: +SKIP('BF16 requires SM80 or higher env') >>> import paddle >>> import paddle.incubate.nn.functional as F >>> paddle.set_device('gpu') >>> batch_size, seq_len, expert_dim = 32, 128, 2048 >>> x = paddle.randn([batch_size, seq_len, expert_dim], dtype='bfloat16') >>> quantized_out, scales = F.fused_weighted_swiglu_act_quant(x) >>> print(x.shape) paddle.Size([32, 128, 2048]) >>> print(quantized_out.shape) paddle.Size([4096, 1024]) >>> print(scales.shape) paddle.Size([4096, 8]) N)rrfused_weighted_swiglu_act_quant)rprobusing_pow2_scalingrs r r)r)s,t55 '   r c UbS5eUc [5nO5UR[R[R4;dS5eUR upUR upUc[R "X4US9nOFUR U U /:XdSX4SUR 35eUR5(dS5e[5(a[RRR5RS:aSOS n[R "U/[RS9nS unnS nS n[R"UUUUUU[5UUUUUUUU U 5un nU$g) NzBias is not supportedz*Only fp16 and bfloat16 bias are supported.)dtypezExpected shape z, got z Output tensor is not contiguous. ii@)TFFp)r r-paddlefloat16bfloat16rempty is_contiguousrdevicecudaget_device_propertiesmajoruint8rfp8_gemm_blockwise_)aa_decode_scalebb_decode_scale out_dtypeoutbias accumulateuse_split_accumulatoris_a_1d_scaledis_b_1d_scaledMKNK_bworkspace_size workspacetransatransbgrad math_sm_countoutput_s r fp8_gemm_blockwiserR;s <000< |zz NN OO   8 8 8 77DA WWFA {llA63yy    7aVHF399+ 6 7   ""F$FF"}}!!779??1D   LL.!1F $ 11     O      !   ! 1$ ? r c US:XaSn OUS:XaSn O [S5eUS:XaSn O [S5e[5(a8[R"UUU UUUU U5uppU(dX4$U(aX4$XX4$g)N1x128T128x128FzUnsupported quantization methode4m3zUnsupported output type) ValueErrorrrfp8_quant_blockwise)repsiloninput_transposeoutput_scale_transposereturn_transpose_onlyusing_pow2_scale quant_method output_typer using_1x128 using_e5m2x_fp8scalex_fp8_tscale_ts r rXrXsw  " :;;f 233)/)C)C    " !   * &g<  "# #1 1# r )returnr)T)rzSequence[Tensor]rboolrftuple[Tensor, Tensor])rrrrrfrr&) rrrrrrr str | Nonerfztuple[Tensor, Tensor, Tensor])F)rrr"z Sequence[int]r#rgrfz!tuple[list[Tensor], list[Tensor]])NFN) rrr* Tensor | Noner+rgrrirfrh)NNFTTT) r@rjrArjrBrgrCrgrDrgrErg)gFTFTrTrVN)rrrYfloatrZrgr[rgr\rgr]rgr^strr_rlrri) __future__r functoolstypingrr0rrpaddle.frameworkrcollections.abcrcacher rrrr r'r)rRrXr r r rss# !3(  ,0-/-/$(-/-/`4 4 4 42 HKHK HKHK  HK # HKX6;: |GL   "/ ?C & $ =  = = =   =  = L"&B  B BB BBBN!#'"'!(2 (2 (2(2! (2  (2  (2(2(2 (2r