ёiDSSKJr SSKrSSKJrJr SSKrSSKJr SSKJ r SSK J r J r SSK Jr \(aSSKJr SS KJrJr SS KJr "S S \R*R,5r"S S\R*R05rg)) annotationsN) TYPE_CHECKINGLiteral)_C_ops) LayerHelper)in_dynamic_or_pir_modeno_grad)_BatchNormBase)Tensor) DataLayoutND ParamAttrLike)Layercx^\rSrSrSrSSU4SjjjrS SjrS SjrSrU=r $) BatchNorm#a Applies Batch Normalization over a SparseCooTensor as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift . When use_global_stats = False, the :math:`\mu_{\beta}` and :math:`\sigma_{\beta}^{2}` are the statistics of one mini-batch. Calculated as follows: .. math:: \mu_{\beta} &\gets \frac{1}{m} \sum_{i=1}^{m} x_i \qquad &//\ \ mini-batch\ mean \\ \sigma_{\beta}^{2} &\gets \frac{1}{m} \sum_{i=1}^{m}(x_i - \ \mu_{\beta})^2 \qquad &//\ mini-batch\ variance \\ When use_global_stats = True, the :math:`\mu_{\beta}` and :math:`\sigma_{\beta}^{2}` are not the statistics of one mini-batch. They are global or running statistics (moving_mean and moving_variance). It usually got from the pre-trained model. Calculated as follows: .. math:: moving\_mean = moving\_mean * momentum + \mu_{\beta} * (1. - momentum) \quad &// global \ mean \\ moving\_variance = moving\_variance * momentum + \sigma_{\beta}^{2} * (1. - momentum) \quad &// global \ variance \\ The normalization function formula is as follows: .. math:: \hat{x_i} &\gets \frac{x_i - \mu_\beta} {\sqrt{\sigma_{\beta}^{2} + \epsilon}} \qquad &//\ normalize \\ y_i &\gets \gamma \hat{x_i} + \beta \qquad &//\ scale\ and\ shift - :math:`\epsilon` : add a smaller value to the variance to prevent division by zero - :math:`\gamma` : trainable proportional parameter - :math:`\beta` : trainable deviation parameter Parameters: num_features(int): Indicate the number of channels of the input ``Tensor``. momentum(float, optional): The value used for the moving_mean and moving_var computation. Default: 0.9. epsilon(float, optional): The small value added to the variance to prevent division by zero. Default: 1e-5. weight_attr(ParamAttr|bool, optional): The parameter attribute for Parameter `scale` of batch_norm. If it is set to None or one attribute of ParamAttr, batch_norm will create ParamAttr as weight_attr. If it is set to False, the weight is not learnable. If the Initializer of the weight_attr is not set, the parameter is initialized with Xavier. Default: None. bias_attr(ParamAttr|bool, optional): The parameter attribute for the bias of batch_norm. If it is set to None or one attribute of ParamAttr, batch_norm will create ParamAttr as bias_attr. If it is set to False, the weight is not learnable. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None. data_format(str, optional): Specify the input data format, may be "NDHWC" or "NHWC". Default "NDHWC". use_global_stats(bool|None, optional): Whether to use global mean and variance. If set to False, use the statistics of one mini-batch, if set to True, use the global statistics, if set to None, use global statistics in the test phase and use the statistics of one mini-batch in the training phase. Default: None. name(str, optional): Name for the BatchNorm, default is None. For more information, please refer to :ref:`api_guide_Name`.. Shape: - x: A SparseCooTensor with layout = 'NDHWC' or 'NHWC'. - output: SparseCooTensor with same shape as input x. Returns: None. Examples: .. code-block:: pycon >>> import paddle >>> paddle.seed(123) >>> channels = 3 >>> x_data = paddle.randn((1, 6, 6, 6, channels)).astype('float32') >>> dense_x = paddle.to_tensor(x_data) >>> sparse_x = dense_x.to_sparse_coo(4) >>> batch_norm = paddle.sparse.nn.BatchNorm(channels) >>> batch_norm_out = batch_norm(sparse_x) >>> print(batch_norm_out.shape) paddle.Size([1, 6, 6, 6, 3]) c .>[T U]UUUUUUUUS9 g)N)momentumepsilon weight_attr bias_attr data_formatuse_global_statsnamesuper__init__) self num_featuresrrrrrrr __class__s [/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/sparse/nn/layer/norm.pyrBatchNorm.__init__ms/  ##-  c&US;a [S5eg)N)NDHWCNHWCz:sparse BatchNorm only support layout of "NDHWC" and "NHWC") ValueError)rinputs r _check_data_formatBatchNorm._check_data_formats ) )L  *r"c 6URUR5 UR(a[R"S5 UR cUR(+UlSnOUR (+nURSS:XaSOSn[ 5(a~[R"UURURURURUR(+URURUUR U5 un nU$UURURURURS.nURURUUR(+UR USS.nS n[U5n UR n U R#U S S 9n U R#U S S 9n U R#U S S 9n U R#U S S 9nU R#U S S 9nU R%U 5nUU U U UUS .nU R'XUUS 9 U$)Nzr?r:rBs r forwardBatchNorm.forwards=  1 12 == MMN   ! ! ))-%6D "#( '+'='=#= $ 1 1! 4 ;f ! # #,2,E,E   MM! &&$ - )NAq!Q" !   NN F!NN==*#}},$($:$:(<"'E*G )FKKE@@4AH"DD4EL BB4CJ$FF4GN#EE4FMBB5IC$ ,("0!. G   WE  Jr")rH)?h㈵>NNr$NN)rintrfloatrr[rParamAttrLike | Nonerr\rLiteral['NDHWC', 'NHWC']rz bool | Noner str | NonereturnNone)r'r]r_r`)r'r r_r ) __name__ __module__ __qualname____firstlineno____doc__rr(rV__static_attributes__ __classcell__rs@r rr#sGX,0*.07(,     *  (  . &     , NNr"rc|^\rSrSrSrSSU4SjjjrS Sjr\S Sj5rSr U=r $) SyncBatchNorma This interface is used to construct a callable object of the ``SyncBatchNorm`` class. It implements the function of the Cross-GPU Synchronized Batch Normalization Layer, and can be used as a normalizer function for other operations, such as conv2d and fully connected operations. The data is normalized by the mean and variance of the channel based on whole mini-batch , which including data in all gpus. Refer to `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `_ for more details. When model in training mode, the :math:`\\mu_{\\beta}` and :math:`\\sigma_{\\beta}^{2}` are the statistics of whole mini-batch data in all gpus. Calculated as follows: .. math:: \mu_{\beta} &\gets \frac{1}{m} \sum_{i=1}^{m} x_i \qquad &//\ \ mini-batch\ mean \\ \sigma_{\beta}^{2} &\gets \frac{1}{m} \sum_{i=1}^{m}(x_i - \ \mu_{\beta})^2 \qquad &//\ mini-batch\ variance \\ - :math:`x` : whole mini-batch data in all gpus - :math:`m` : the size of the whole mini-batch data When model in evaluation mode, the :math:`\\mu_{\\beta}` and :math:`\sigma_{\beta}^{2}` are global statistics (moving_mean and moving_variance, which usually got from the pre-trained model). Global statistics calculated as follows: .. math:: moving\_mean = moving\_mean * momentum + \mu_{\beta} * (1. - momentum) \quad &// global \ mean \\ moving\_variance = moving\_variance * momentum + \sigma_{\beta}^{2} * (1. - momentum) \quad &// global \ variance \\ The formula of normalization is as follows: .. math:: \hat{x_i} &\gets \frac{x_i - \mu_\beta} {\sqrt{\ \sigma_{\beta}^{2} + \epsilon}} \qquad &//\ normalize \\ y_i &\gets \gamma \hat{x_i} + \beta \qquad &//\ scale\ and\ shift - :math:`\epsilon` : add a smaller value to the variance to prevent division by zero - :math:`\gamma` : trainable scale parameter vector - :math:`\beta` : trainable shift parameter vector Note: If you want to use container to pack your model and has ``SyncBatchNorm`` in the evaluation phase, please use ``nn.LayerList`` or ``nn.Sequential`` instead of ``list`` to pack the model. Parameters: num_features(int): Indicate the number of channels of the input ``Tensor``. epsilon(float, optional): The small value added to the variance to prevent division by zero. Default: 1e-5. momentum(float, optional): The value used for the moving_mean and moving_var computation. Default: 0.9. weight_attr(ParamAttr|bool, optional): The parameter attribute for Parameter `scale` of this layer. If it is set to None or one attribute of ParamAttr, this layer will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. If it is set to False, this layer will not have trainable scale parameter. Default: None. bias_attr(ParamAttr|bool, optional): The parameter attribute for the bias of this layer. If it is set to None or one attribute of ParamAttr, this layer will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. If it is set to False, this layer will not have trainable bias parameter. Default: None. data_format(str, optional): Specify the input data format, may be "NCHW". Default "NCHW". name(str, optional): Name for the BatchNorm, default is None. For more information, please refer to :ref:`api_guide_Name`.. Shapes: input: Tensor that the dimension from 2 to 5. output: Tensor with the same shape as input. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> import paddle.sparse.nn as nn >>> paddle.device.set_device('gpu') >>> x = paddle.to_tensor([[[[0.3, 0.4], [0.3, 0.07]], [[0.83, 0.37], [0.18, 0.93]]]], dtype='float32') >>> x = x.to_sparse_coo(len(x.shape)-1) >>> if paddle.is_compiled_with_cuda(): ... sync_batch_norm = nn.SyncBatchNorm(2) ... hidden1 = sync_batch_norm(x) ... print(hidden1) Tensor(shape=[1, 2, 2, 2], dtype=paddle.float32, place=Place(gpu:0), stop_gradient=False, indices=[[0, 0, 0, 0], [0, 0, 1, 1], [0, 1, 0, 1]], values=[[-0.40730840, -0.13725480], [-0.40730840, -1.20299828], [ 1.69877410, -0.23414057], [-0.88415730, 1.57439375]]) c 0>[TU]UUUUUUU5 g)Nr) rrrrrrrrrs r rSyncBatchNorm.__init__<s'         r"c UR5 [R"UURURUR UR UR(+URURURSS5 un nU$)NF) r(rsparse_sync_batch_norm_rJrKrLr0rErMrNrD)rr.sync_batch_norm_outrSs r rVSyncBatchNorm.forwardPsz !-3-K-K JJ NN KK II  NN MM      . *Q1a#"r"c Un[U[5(Ga#URb^[UR[5(d?URRb(URRS-URlUR b^[UR [5(d?UR Rb(UR RS-UR l[U[ 5(aX[URURURURUR URUR5nOk[RRURURURURUR URUR5nURSLaDUR SLa5[5 UR UlUR"UlSSS5 UR$UlUR&UlUR)5H%up4UR+X0R-U55 M' AU$!,(df  Nl=f)an Helper function to convert :class: `paddle.sparse.nn.BatchNorm` layers in the model to :class: `paddle.sparse.nn.SyncBatchNorm` layers. Parameters: layer(paddle.nn.Layer): model containing one or more `BatchNorm` layers. Returns: The original model with converted SyncBatchNorm layers. If BatchNorm layer in the model, use SyncBatchNorm layer instead. Examples: .. code-block:: python >>> import paddle >>> import paddle.sparse.nn as nn >>> model = paddle.nn.Sequential(nn.Conv3D(3, 5, 3), nn.BatchNorm(5)) >>> sync_model = nn.SyncBatchNorm.convert_sync_batchnorm(model) N_syncF) isinstancer _weight_attrboolr _bias_attrrrj _num_featuresrMrNrD_namepaddlennr rLr0rJrKnamed_children add_sublayerconvert_sync_batchnorm)clslayer layer_outputrsublayers r r~$SyncBatchNorm.convert_sync_batchnormas, e^ , ,""."5#5#5t<<&&++7*/*<*<*A*AG*K""'  ,"5#3#3T::$$))5(-(8(8(=(=(G  %%++,''OONN&&$$&&KK   &yy66''OONN&&$$&&KK  ""%/$$E1Y*/,,L'(- L%"'L %*__L "#224ND  % %00: 5 Ys /#I88 J)rXrYNNr-N)rrZrr[rr[rr\rr\rr rr^r_r`)r.r r_r )rrr_r) rarbrcrdrerrV classmethodr~rfrgrhs@r rjrjs_H,0*.$*     *  (  "     (#"JJr"rj) __future__rrFtypingrrrzrpaddle.base.layer_helperrpaddle.frameworkrr paddle.nn.layer.normr r paddle._typingr r paddle.nnrr{ BatchNorm1Drrjrr"r rs`#) 0</ t %%tnRFII++Rr"