{-jDddlmZddlZddlmZmZddlZddlmZddlm Z ddl m Z m Z ddl mZerddlmZdd lmZmZdd lmZGd d ejjZGd dejjZdS)) annotationsN) TYPE_CHECKINGLiteral)_C_ops) LayerHelper)in_dynamic_or_pir_modeno_grad)_BatchNormBase)Tensor) DataLayoutND ParamAttrLike)LayercDeZdZdZ ddfd ZddZddZxZS) BatchNormaa Applies Batch Normalization over a SparseCooTensor as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift . When use_global_stats = False, the :math:`\mu_{\beta}` and :math:`\sigma_{\beta}^{2}` are the statistics of one mini-batch. Calculated as follows: .. math:: \mu_{\beta} &\gets \frac{1}{m} \sum_{i=1}^{m} x_i \qquad &//\ \ mini-batch\ mean \\ \sigma_{\beta}^{2} &\gets \frac{1}{m} \sum_{i=1}^{m}(x_i - \ \mu_{\beta})^2 \qquad &//\ mini-batch\ variance \\ When use_global_stats = True, the :math:`\mu_{\beta}` and :math:`\sigma_{\beta}^{2}` are not the statistics of one mini-batch. They are global or running statistics (moving_mean and moving_variance). It usually got from the pre-trained model. Calculated as follows: .. math:: moving\_mean = moving\_mean * momentum + \mu_{\beta} * (1. - momentum) \quad &// global \ mean \\ moving\_variance = moving\_variance * momentum + \sigma_{\beta}^{2} * (1. - momentum) \quad &// global \ variance \\ The normalization function formula is as follows: .. math:: \hat{x_i} &\gets \frac{x_i - \mu_\beta} {\sqrt{\sigma_{\beta}^{2} + \epsilon}} \qquad &//\ normalize \\ y_i &\gets \gamma \hat{x_i} + \beta \qquad &//\ scale\ and\ shift - :math:`\epsilon` : add a smaller value to the variance to prevent division by zero - :math:`\gamma` : trainable proportional parameter - :math:`\beta` : trainable deviation parameter Parameters: num_features(int): Indicate the number of channels of the input ``Tensor``. momentum(float, optional): The value used for the moving_mean and moving_var computation. Default: 0.9. epsilon(float, optional): The small value added to the variance to prevent division by zero. Default: 1e-5. weight_attr(ParamAttr|bool, optional): The parameter attribute for Parameter `scale` of batch_norm. If it is set to None or one attribute of ParamAttr, batch_norm will create ParamAttr as weight_attr. If it is set to False, the weight is not learnable. If the Initializer of the weight_attr is not set, the parameter is initialized with Xavier. Default: None. bias_attr(ParamAttr|bool, optional): The parameter attribute for the bias of batch_norm. If it is set to None or one attribute of ParamAttr, batch_norm will create ParamAttr as bias_attr. If it is set to False, the weight is not learnable. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None. data_format(str, optional): Specify the input data format, may be "NDHWC" or "NHWC". Default "NDHWC". use_global_stats(bool|None, optional): Whether to use global mean and variance. If set to False, use the statistics of one mini-batch, if set to True, use the global statistics, if set to None, use global statistics in the test phase and use the statistics of one mini-batch in the training phase. Default: None. name(str, optional): Name for the BatchNorm, default is None. For more information, please refer to :ref:`api_guide_Name`.. Shape: - x: A SparseCooTensor with layout = 'NDHWC' or 'NHWC'. - output: SparseCooTensor with same shape as input x. Returns: None. Examples: .. code-block:: pycon >>> import paddle >>> paddle.seed(123) >>> channels = 3 >>> x_data = paddle.randn((1, 6, 6, 6, channels)).astype('float32') >>> dense_x = paddle.to_tensor(x_data) >>> sparse_x = dense_x.to_sparse_coo(4) >>> batch_norm = paddle.sparse.nn.BatchNorm(channels) >>> batch_norm_out = batch_norm(sparse_x) >>> print(batch_norm_out.shape) paddle.Size([1, 6, 6, 6, 3]) ?h㈵>NNDHWC num_featuresintmomentumfloatepsilon weight_attrParamAttrLike | None bias_attr data_formatLiteral['NDHWC', 'NHWC']use_global_stats bool | Nonename str | NonereturnNonec Zt||||||||dS)N)rrrrrrr super__init__) selfrrrrrrrr __class__s [/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/paddle/sparse/nn/layer/norm.pyr'zBatchNorm.__init__msH  ##-  inputc,|dvrtddS)N)rNHWCz:sparse BatchNorm only support layout of "NDHWC" and "NHWC") ValueError)r(r,s r*_check_data_formatzBatchNorm._check_data_formats+ ) ) )L  * )r+r c ||j|jrtjd|j|j |_d}n|j }|jddkrdnd}t rPtj||j |j |j |j |j |j |j||j| \}}}}}}|S||j |j |j |j d}|j |j||j |j|dd}d }t|} |j} | | d } | | d } | | d } | | d }| | d }| | }|| | | ||d }| |||| |S)Nz ,2,E   M! &$ - - )NAq!Q" !   N F!N=*#},$($:(<"'E*G ))FKE@@4AH"DD4EL BB4CJ$FF4GN#EE4FMBB5IIC$ ,("0!. G   VWE    Jr+)rrNNrNN)rrrrrrrrrrrrrrr r!r"r#)r,rr"r#)r,r r"r )__name__ __module__ __qualname____doc__r'r0r] __classcell__r)s@r*rr#sGGX,0*.07(,       , NNNNNNNNr+rcReZdZdZ ddfd ZddZeddZxZS) SyncBatchNorma This interface is used to construct a callable object of the ``SyncBatchNorm`` class. It implements the function of the Cross-GPU Synchronized Batch Normalization Layer, and can be used as a normalizer function for other operations, such as conv2d and fully connected operations. The data is normalized by the mean and variance of the channel based on whole mini-batch , which including data in all gpus. Refer to `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `_ for more details. When model in training mode, the :math:`\\mu_{\\beta}` and :math:`\\sigma_{\\beta}^{2}` are the statistics of whole mini-batch data in all gpus. Calculated as follows: .. math:: \mu_{\beta} &\gets \frac{1}{m} \sum_{i=1}^{m} x_i \qquad &//\ \ mini-batch\ mean \\ \sigma_{\beta}^{2} &\gets \frac{1}{m} \sum_{i=1}^{m}(x_i - \ \mu_{\beta})^2 \qquad &//\ mini-batch\ variance \\ - :math:`x` : whole mini-batch data in all gpus - :math:`m` : the size of the whole mini-batch data When model in evaluation mode, the :math:`\\mu_{\\beta}` and :math:`\sigma_{\beta}^{2}` are global statistics (moving_mean and moving_variance, which usually got from the pre-trained model). Global statistics calculated as follows: .. math:: moving\_mean = moving\_mean * momentum + \mu_{\beta} * (1. - momentum) \quad &// global \ mean \\ moving\_variance = moving\_variance * momentum + \sigma_{\beta}^{2} * (1. - momentum) \quad &// global \ variance \\ The formula of normalization is as follows: .. math:: \hat{x_i} &\gets \frac{x_i - \mu_\beta} {\sqrt{\ \sigma_{\beta}^{2} + \epsilon}} \qquad &//\ normalize \\ y_i &\gets \gamma \hat{x_i} + \beta \qquad &//\ scale\ and\ shift - :math:`\epsilon` : add a smaller value to the variance to prevent division by zero - :math:`\gamma` : trainable scale parameter vector - :math:`\beta` : trainable shift parameter vector Note: If you want to use container to pack your model and has ``SyncBatchNorm`` in the evaluation phase, please use ``nn.LayerList`` or ``nn.Sequential`` instead of ``list`` to pack the model. Parameters: num_features(int): Indicate the number of channels of the input ``Tensor``. epsilon(float, optional): The small value added to the variance to prevent division by zero. Default: 1e-5. momentum(float, optional): The value used for the moving_mean and moving_var computation. Default: 0.9. weight_attr(ParamAttr|bool, optional): The parameter attribute for Parameter `scale` of this layer. If it is set to None or one attribute of ParamAttr, this layer will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. If it is set to False, this layer will not have trainable scale parameter. Default: None. bias_attr(ParamAttr|bool, optional): The parameter attribute for the bias of this layer. If it is set to None or one attribute of ParamAttr, this layer will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. If it is set to False, this layer will not have trainable bias parameter. Default: None. data_format(str, optional): Specify the input data format, may be "NCHW". Default "NCHW". name(str, optional): Name for the BatchNorm, default is None. For more information, please refer to :ref:`api_guide_Name`.. Shapes: input: Tensor that the dimension from 2 to 5. output: Tensor with the same shape as input. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> import paddle.sparse.nn as nn >>> paddle.device.set_device('gpu') >>> x = paddle.to_tensor([[[[0.3, 0.4], [0.3, 0.07]], [[0.83, 0.37], [0.18, 0.93]]]], dtype='float32') >>> x = x.to_sparse_coo(len(x.shape)-1) >>> if paddle.is_compiled_with_cuda(): ... sync_batch_norm = nn.SyncBatchNorm(2) ... hidden1 = sync_batch_norm(x) ... print(hidden1) Tensor(shape=[1, 2, 2, 2], dtype=paddle.float32, place=Place(gpu:0), stop_gradient=False, indices=[[0, 0, 0, 0], [0, 0, 1, 1], [0, 1, 0, 1]], values=[[-0.40730840, -0.13725480], [-0.40730840, -1.20299828], [ 1.69877410, -0.23414057], [-0.88415730, 1.57439375]]) rrNr4rrrrrrrrrr r r!r"r#c Vt|||||||dS)Nr%) r(rrrrrrr r)s r*r'zSyncBatchNorm.__init__<s@             r+r5r c |tj||j|j|j|j|j |j|j |j dd \}}}}}}|S)NF) r0rsparse_sync_batch_norm_rQrRrSr7rLrTrUrK)r(r5sync_batch_norm_outrZs r*r]zSyncBatchNorm.forwardPsq !!!-3-K J N K I  N M     . . *Q1a#"r+layerrc |}t|tr|j?t|jts%|jj|jjdz|j_|j?t|jts%|jj|jjdz|j_t|t r9t|j|j |j |j|j|j |j }nHtj|j|j |j |j|j|j |j }|jdurG|jdur>t5|j|_|j|_dddn #1swxYwY|j|_|j|_|D].\}}||||/~|S)a Helper function to convert :class: `paddle.sparse.nn.BatchNorm` layers in the model to :class: `paddle.sparse.nn.SyncBatchNorm` layers. Parameters: layer(paddle.nn.Layer): model containing one or more `BatchNorm` layers. Returns: The original model with converted SyncBatchNorm layers. If BatchNorm layer in the model, use SyncBatchNorm layer instead. Examples: .. code-block:: python >>> import paddle >>> import paddle.sparse.nn as nn >>> model = paddle.nn.Sequential(nn.Conv3D(3, 5, 3), nn.BatchNorm(5)) >>> sync_model = nn.SyncBatchNorm.convert_sync_batchnorm(model) N_syncF) isinstancer _weight_attrboolr _bias_attrrre _num_featuresrTrUrK_namepaddlennr rSr7rQrRnamed_children add_sublayerconvert_sync_batchnorm)clsrj layer_outputr sublayers r*rwz$SyncBatchNorm.convert_sync_batchnormas(, e^ , ,- 5"."5#5t<</&+7*/*<*AG*K"' ,"5#3T::-$)5(-(8(=(G %%++ ,'ON&$&K    &y66'ON&$&K   "%//$E11YY33*/,L'(- L%333333333333333"'L %*_L "#2244  ND(  % %c00::     sFFF)rrNNr4N)rrrrrrrrrrrr r r!r"r#)r5r r"r )rjrr"r) r^r_r`rar'r] classmethodrwrbrcs@r*reres__H,0*.$*       (####"JJJ[JJJJJr+re) __future__rrMtypingrrrsrpaddle.base.layer_helperrpaddle.frameworkrr paddle.nn.layer.normr r paddle._typingr r paddle.nnrrt BatchNorm1Drrer+r*rsb#""""")))))))) 000000<<<<<<<<//////  ttttt %tttnRRRRRFI+RRRRRr+