ΑiY`SSKJr SSKrSSKrSSKrSSKrSSKrSSKJrJ r J r J r SSK J r SSKJr SSKJrJr SSKJr SSKJr SS KJrJrJr SS KJrJrJr SS KJ r \(aSS K!J"r"J#r# SS K$J%r%J&r& "SS\ 5r'/r("SS5r)Sr*Sr+Sr,Sr-Sr.Sr/Sr0Sr1"SS5r2SSSjjr3g)) annotationsN) TYPE_CHECKINGAnyLiteral TypedDict)core) get_device)_get_trainers_numget_cluster_and_pod)use_paddlecloud)get_cluster_from_args) DeviceModeblock_windows_and_macos check_backend)_prepare_trainer_env_print_argumentsget_host_name_ip) set_flags)CallableIterable) NotRequiredUnpackc>\rSrSr%S\S'S\S'S\S'S\S'S rg ) _SpawnOptions3z3NotRequired[Literal['spawn', 'fork', 'forkserver']] start_methodzNotRequired[str | None]gpusxpuszNotRequired[str]ipsN)__name__ __module__ __qualname____firstlineno____annotations____static_attributes__r X/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/distributed/spawn.pyrr3sII%%%% r'rc\rSrSrSrSrg)ParallelEnvArgs=cXSUlSUlSUlSUlSUlSUlg)NT)cluster_node_ipsnode_ipr started_port print_configselected_devices)selfs r(__init__ParallelEnvArgs.__init__>s; $  $!! !%r')r-r.r0r1r/r N)r!r"r#r$r3r&r r'r(r*r*=s%r'r*c/SQn/SQnUH=nX1;dM X2;a![R"SUS3[5 M0[SUS35e g)N)rrrrr0backend)r1r/r-r.r zThe config option (z|) of `paddle.distributed.spawn` is deprecated. Please use the latest config options stated in the `spawn` API documentation.z1) of `paddle.distributed.spawn` is not supported.)warningswarnDeprecationWarning ValueError)optionssupported_optionsdeprecated_optionskeys r(_options_valid_checkr?Vsh  '( )#/dd& !)#._`r'cX[5nU[R"5;a([R"UR S5S5$SU;a[R "5$SU;a[R "5$SU;a[R"5$[SUS35e)N:rgpuxpucpuI`paddle.distributed.spawn` does not support parallel training on device `` now.) r rget_available_custom_deviceget_custom_device_countsplitget_cuda_device_countget_xpu_device_countmultiprocessing cpu_count RuntimeErrordevices r(_get_default_nprocsrQus \F 1133++FLL,=a,@AA &))++ &((** &((**WX^W__e f  r'c[5nU[R"5;agSU;agSU;agSU;ag[SUS 35e) NxcclrBncclrCbkclrDgloorErF)r rrGrNrOs r(_get_default_backendrWsW \F 1133 & & &WX^W__e f  r'cSnURS5Vs/sHo"R5PM nn[U5S:XaUSnU$[5upAU$s snf)N,r)rIstriplenr)rr.xnode_ips_s r( _get_node_ipr`sYG#&99S>2>a >H2 8}1+ N&'  N 3sAc  SU;d USS:Xa [5US'[US5 [US5 /n[5nUR SS5UlUR c+UR SS5UlUR cSUlUSS:XGaUR SS5UlUR cUR SS5Ul[R"S S5nUbUS :Xa8[[R"55Vs/sHn[U5PM nnOURS 5nUR ch[U5U:a[S [U5S US35eS R![SU5Vs/sHn[Xe5PM sn5UlGOUR RS 5n[U5U:wa[#S[U5SUS35eUH2nX;dM [#SR%US R!U555e GO>USS:XGaUR SS5UlUR cUR SS5Ul[R"SS5nUbUS :Xa8[[R&"55Vs/sHn[U5PM nnOURS 5nUR ch[U5U:a[S [U5S US35eS R![SU5Vs/sHn[Xe5PM sn5UlGOUR RS 5n[U5U:wa[#S[U5SUS35eUH2nX;dM [#SR%US R!U555e GOUSS:Xa[(R*"S5 SUlSUlUR UlUR SS5bS5e[UR RS 55S::dS 5e[15S:XdS!5eOUSS":XaSUl[R2"5Sn [R"S#U S$3S5nUbUS :Xa9[[R4"U 55Vs/sHn[U5PM nnOURS 5n[U5U:a[S [U5S US%U S&35eS R![SU5Vs/sHn[Xe5PM sn5UlUR S'S5UlUR6c[9UR 5UlUR S(S5UlUR SS5UlUR<c[=5UlUSS:Xa3[?[SU55n [AU[BRDU 5upO [GU5upU RHH!n URK[MXUS55 M# UR S)S*5Ul'URN(a [QU5 U$s snfs snfs snfs snfs snfs snf)+Nr6autorr-z 127.0.0.1rTrr1CUDA_VISIBLE_DEVICESrYzthe number of visible devices(z-) is less than the number of spawn processes(z), please ensure that the correct `nprocs` argument is passed or the environment variable `CUDA_VISIBLE_DEVICES` is correctly configured.rzThe number of selected devices(z0) is not equal to the number of spawn processes(zK), please ensure that the correct `nprocs` and `gpus` arguments are passed.zCThe selected gpu card {} cannot found in CUDA_VISIBLE_DEVICES ({}).rUrXPU_VISIBLE_DEVICESz), please ensure that the correct `nprocs` argument is passed or the environment variable `XPU_VISIBLE_DEVICES` is correctly configured.zK), please ensure that the correct `nprocs` and `xpus` arguments are passed.zBThe selected xpu card {} cannot found in XPU_VISIBLE_DEVICES ({}).rVzYour model will be trained under CPUONLY mode by using GLOO,because CPUPlace is specified manually or your installed PaddlePaddle only support CPU Device.Tr z.CPUONLY spawn doesn't support use paddle cloudrZzJCPUONLY spawn only support single trainer, that is len(ips)=1, but got %s.z+CPUONLY spawn doesn't support multi-trainerrSFLAGS_selected_szj), please ensure that the correct `nprocs` argument is passed or the environment variable `FLAGS_selected_zs` is correctly configured.r.r/r0F))rWrrr*getr-r1osgetenvrangerrJstrrIr\rNjoinr:formatrKr7r8paddle_cpuonlyrr get_all_custom_device_typerHr.r`r/r listr rCPUr trainersappendrr0r)nprocsr;processes_env_listargs env_devicesr]env_devices_listselected_device_listcard_idcustom_device_namedevices_per_procclusterpodtrainers r(_get_subprocess_env_listrs 79#5#?13 ')$%GI./  D$KKt4D $ ' ,> E  ($/D !yV# ' FD 9  ($+KK0BD$ID !ii 6=  +"3 %d&@&@&B C  C1A C    +005   (#$v-"4S9I5J4KL**02FF %(HH38F3CD3Ca%()3CD%D !$(#8#8#>#>s#C '(F2 5c:N6O5PQ55;H=HH 02$55;V#SXX.>%?60  v % ' FD 9  ($+KK0BD$ID !ii 5t<  +"3 %d&?&?&A B  B1A B    +005   (#$v-"4S9I5J4KL**02EE %(HH38F3CD3Ca%()3CD%D !$(#8#8#>#>s#C '(F2 5c:N6O5PQ55;H=HH 02$44:F#SXX.>%?50  v %  m # $(({{,d3; < ;4((..s349 X 9!"a' 9 '  v % $!<<>qAii/2D1EQ GN  +"3t;;A?c*\rSrSrSrSSjrSrSrg)MultiprocessContexticX lX0lXl[U5VVs0sHupEURU_M snnUlgs snnfN) error_queues return_queues processes enumeratesentinel sentinels)r2rrrindexprocesss r(r3MultiprocessContext.__init__sH( +":CI:N :NG  e #:N  sANc>[UR5S:Xag[RR URR 5US9nSnUHQnURR U5nURUnUR5 URS:wdMOUn O Uc[UR5S:H$URH8nUR5(aUR5 UR5 M: URU5 g)NrT)timeout) r\rrL connectionwaitkeysrrrmexitcodeis_alive terminate_throw_exception)r2rready error_indexrrrs r(rmMultiprocessContext.joins t~~ ! #**// NN   !70  HNN&&x0EnnU+G LLN1$#    t~~&!+ +~~G!!!!# LLN& k*r'crURUR5(adURURnUS:a3[R "U*5R n[SUSUS35e[SUSUS35eURUR5nSUS3nXT- n[U5e)NrzProcess z terminated with signal .z terminated with exit code z9 ---------------------------------------------- Process zV terminated with the following error: ---------------------------------------------- ) remptyrrsignalSignalsnamerrh)r2rrroriginal_tracemsgs r(r$MultiprocessContext._throw_exceptions   [ ) / / 1 1~~k2;;H!|~~xi055{m+CD6K {m+FxjPQR**;7;;= "m$A A nr')rrrrr)r!r"r#r$r3rmrr&r r'r(rrs  +6r'rc p[U5 US:Xa [5n[X%5nURSS5nUcSn[R "U5n/n /n /n [ U5Hn UR5n UR5nUR[UUU UXlUS4S9nXOl UR5 U RU 5 U RU5 U RU5 M [XU 5nU(dU$UR5(dUR5(dMU$)a Start multiple processes with ``spawn`` method for parallel training. .. note:: ``spawn`` now only supports GPU or XPU collective mode. The collective mode of GPU and XPU cannot be started at the same time, so the option `gpus` and `xpus` cannot be configured at the same time. Args: func (function): The target function is called by spawned process. This function need to be able to pickled, so it must be defined at the top level of a module. args (list|tuple, optional): Arguments passed to ``func``. nprocs (int, optional): Number of processed to start. Default: -1. when nprocs is -1, the available device will be obtained from the environment variable when the model is executed: If use GPU, the currently available device ID is obtained from the environment variable CUDA_VISIBLE_DEVICES; If use XPU, the currently available device ID is obtained from the environment variable XPU_VISIBLE_DEVICES. join (bool, optional): Perform a blocking join on all spawned processes. Default: True. daemon (bool, optional): The spawned processes' daemon flag. Default: False. **options(dict, optional): Other initial parallel execution environment configuration options. The following options are currently supported: (1) start_method (string): the way to start a process. The start method can be ``spawn`` , ``fork`` , ``forkserver`` . Because the CUDA runtime does not support the ``fork`` start method, when use CUDA in subprocesses, we should start process by ``spawn`` or ``forkserver`` method. Default: "spawn" ; (2) gpus (string): The training process will run on the selected gpus, such as "0,1,2,3". Default: None; (3) xpus (string): The training process will run on the selected xpus, such as "0,1,2,3". Default: None; (5) ips (string): Paddle cluster nodes ips, such as "192.168.0.16,192.168.0.17". Default: "127.0.0.1" . Returns: ``MultiprocessContext`` object, it hold the spawned processes. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:DISTRIBUTED) >>> import paddle >>> import paddle.nn as nn >>> import paddle.optimizer as opt >>> import paddle.distributed as dist >>> class LinearNet(nn.Layer): ... def __init__(self): ... super().__init__() ... self._linear1 = nn.Linear(10, 10) ... self._linear2 = nn.Linear(10, 1) ... def forward(self, x): ... return self._linear2(self._linear1(x)) >>> def train(print_result=False): ... # 1. initialize parallel environment ... group = dist.init_parallel_env() ... process_group = group.process_group if group else None ... # 2. create data parallel layer & optimizer ... layer = LinearNet() ... dp_layer = paddle.DataParallel(layer, group = process_group) # type: ignore[arg-type] ... loss_fn = nn.MSELoss() ... adam = opt.Adam( ... learning_rate=0.001, parameters=dp_layer.parameters()) ... # 3. run layer ... inputs = paddle.randn([10, 10], 'float32') ... outputs = dp_layer(inputs) ... labels = paddle.randn([10, 1], 'float32') ... loss = loss_fn(outputs, labels) ... if print_result is True: ... print("loss:", loss.numpy()) ... loss.backward() ... adam.step() ... adam.clear_grad() >>> # Usage 1: only pass function. >>> # If your training method no need any argument, and >>> # use all visible devices for parallel training. >>> if __name__ == '__main__': ... dist.spawn(train) >>> # Usage 2: pass function and arguments. >>> # If your training method need some arguments, and >>> # use all visible devices for parallel training. >>> if __name__ == '__main__': ... dist.spawn(train, args=(True,)) >>> # Usage 3: pass function, arguments and nprocs. >>> # If your training method need some arguments, and >>> # only use part of visible devices for parallel training. >>> # If your machine hold 8 cards {0,1,2,3,4,5,6,7}, >>> # this case will use cards {0,1}; If you set >>> # CUDA_VISIBLE_DEVICES=4,5,6,7, this case will use >>> # cards {4,5} >>> if __name__ == '__main__': ... dist.spawn(train, args=(True,), nprocs=2) >>> # Usage 4: pass function, arguments, nprocs and gpus. >>> # If your training method need some arguments, and >>> # only use part of visible devices for parallel training, >>> # but you can't set your machine's environment variable >>> # CUDA_VISIBLE_DEVICES, such as it is None or all cards >>> # {0,1,2,3,4,5,6,7}, you can pass `gpus` to >>> # select the GPU cards you want to use. For example, >>> # this case will use cards {4,5} if your machine hold 8 cards. >>> if __name__ == '__main__': ... dist.spawn(train, args=(True,), nprocs=2, gpus='4,5') rNspawnr6)targetrw)r?rQrrhrL get_contextrk SimpleQueueProcessrdaemonstartrtrrm)rrwrurmrr;procs_env_listrmprrrirrrcontexts r(rrs5r!|$& .f>N;;~t4L  $ $\ 2BLMI 6]nn& ~~' ** ! "     K(\*!%(")=IG llnn llnn Nr')r rTF)rzCallable[..., None]rwz Iterable[Any]ruintrmboolrrr;zUnpack[_SpawnOptions]returnr)4 __future__rrLrirrr7typingrrrr paddle.baser paddle.devicer paddle.distributed.cloud_utilsr r $paddle.distributed.fleet.cloud_utilsr paddle.distributed.fleet.launchr %paddle.distributed.fleet.launch_utilsrrr%paddle.distributed.utils.launch_utilsrrrpaddle.frameworkrcollections.abcrrtyping_extensionsrrr__all__r*r?rQrWr`rrrrrrr r'r(rs# 99$AA  '25  %%2>   up(20$==D o o o o  o  o % oor'