x-j$ddlZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z m Z gZdZGdd ZeZd Zdd Z ddZdS)N) _legacy_C_ops)core)check_variable_and_dtype)Variable) LayerHelperin_dynamic_modemodel_parallel_rngcTeZdZdZdZdZdZdZdZe j e fdZ dS) RNGStatesTrackerz! Tracker the RNG states. c:i|_t|_dSNstates_setseeds_selfs j/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/paddle/distributed/fleet/layers/mpu/random.py__init__zRNGStatesTracker.__init__'s ee c:i|_t|_dSr rrs rresetzRNGStatesTracker.reset,s ee rc||jvrtd|d|j|||jvrtd|dtjd}tj|j|<t j|tj |ddS)Nzseed z already existsstate T use_index) r ValueErroraddrpaddleincubate get_rng_stateregister_rng_state_as_indexseed set_rng_state)rnamer#orig_rng_state_indexs rrzRNGStatesTracker.add0s 4;  :T:::;; ;  4<  ;d;;;<< <%<!>!>!N!NDL  O ) )*>$ ) O O O O O"(!>!>!>!N!NDL  O ) )*>$ ) O O O Os ,B;;A DN) __name__ __module__ __qualname____doc__rrrr*r, contextlibcontextmanagerMODEL_PARALLEL_RNGr.rrr r "s  L L L    L L L/ P P P P P Prr ctSr )RNG_STATE_TRACKERr6rrget_rng_state_trackerr9cs rcddlm}|}|}|}|}|}|r|}|dz||zz|z}n.tj dd}|dz||zz|z}t t t|tj|dS)Nr)fleeti')paddle.distributedr;get_hybrid_communicate_groupget_model_parallel_rankget_model_parallel_world_size get_stage_idget_pipe_parallel_world_sizenprandomrandintr8rrr5rr#) r#r;hcgmp_rankmp_sizepp_rankpp_size global_seed local_seeds rmodel_parallel_random_seedrMgs((((((  , , . .C))++G//11G  G..00G C AX' 11G; i''511  1_w'887B ,j999 K r?Tupscale_in_trainc F|)tjj||||||St |t t tfstdt |t t fr|dkr|Sd|cxkrdksnJtd|dvsJtd|Jtd|d krd n|}tr$tj|d |d | d dddd| \}}|St |tr#|j dgkstd|j tdit} t|dgdd| tj} | dd| i| |j}| t(jjjd}| d|g| d|g|gd|| |d|S)a Dropout is a regularization technique for reducing overfitting by preventing neuron co-adaption during training. The dropout operator randomly sets the outputs of some units to zero, while upscale others according to the given dropout probability. Args: x (Tensor): The input tensor. The data type is float32 or float64. p (float|int): Probability of setting units to zero. Default 0.5. axis (int|list|tuple): The axis along which the dropout is performed. Default None. rng_name (str): The random seed generator name, which used to obtain deterministic results. training (bool): A flag indicating whether it is in train phrase or not. Default True. mode(str): ['upscale_in_train'(default) | 'downscale_in_infer']. 1. upscale_in_train(default), upscale the output at training time - train: out = input * mask / ( 1.0 - dropout_prob ) - inference: out = input 2. downscale_in_infer, downscale the output at inference - train: out = input * mask - inference: out = input * (1.0 - dropout_prob) name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Returns: A Tensor representing the dropout, has same shape and data type as `x` . Examples: We use ``p=0.5`` in the following description for simplicity. 1. When ``axis=None`` , this is commonly used dropout, which dropout each element of x randomly. .. code-block:: text Let's see a simple case when x is a 2d tensor with shape 2*3: [[1 2 3] [4 5 6]] we generate mask with the same shape as x, which is 2*3. The value of mask is sampled from a Bernoulli distribution randomly. For example, we may get such mask: [[0 1 0] [1 0 1]] So the output is obtained from elementwise multiply of x and mask: [[0 2 0] [4 0 6]] Using default setting, i.e. ``mode='upscale_in_train'`` , if in training phase, the final upscale output is: [[0 4 0 ] [8 0 12]] if in test phase, the output is the same as input: [[1 2 3] [4 5 6]] we can also set ``mode='downscale_in_infer'`` , then if in training phase, the final output is: [[0 2 0] [4 0 6]] if in test phase, the scale output is: [[0.5 1. 1.5] [2. 2.5 3. ]] Nz4p argument should be a number(int|float) or Variablerr<z!p argument should between 0 and 1)downscale_in_inferrOzBmode argument should be 'downscale_in_infer' or 'upscale_in_train'z1unsupported axis when using random seed generatorrQdowngrade_in_infer dropout_probis_testfix_seedFr#dropout_implementationzGRequired p.shape == [1] if type(p) is Variable, but received p.shape = dropoutx)float16float32float64)dtypeOut)typeoutputsT)r\ stop_gradient)XSeed)r]Mask)rSrTrV)r^inputsr_attrs)rW)rnn functionalrW isinstancefloatintr TypeErrorrrrshaperlocalsr"create_variable_for_type_inferenceint32 append_opr\rVarDescVarTypeUINT8) rXpaxisrng_nametrainingmoder%outmaskhelperr#s rrWrWsNy#++Aq$$MMM a%h/ 0 0PNOOO!c5\""qAvv ;;;;Q;;;;; #FGG;;; = = = =zL@@ = = = <<;<<< !%(< < <$ ,!)  L    $    T a " " 17qc>>cZ[Zacc 33&((33 s555y   88v|8LLfudm<<<77ag7FF88,&,D9   d++ ED622 !'<*.   rr )rNNNTrON)r3numpyrCrr paddle.baserpaddle.base.data_feederrpaddle.common_ops_importrpaddle.frameworkrr__all__r5r r8r9rMrWr6rrrs0 <<<<<<------99999999 ) ;P;P;P;P;P;P;P;P|%$&&4    KKKKKKr