Αi-pSrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKJ r J r SSK J r SSK JrJr SSKJrJr SSKJrJrJrJrJrJrJrJrJrJrJrJrJ r J!r! /r"Sr#S r$S r%S r&S r'S r(Sr)Sr*Sr+Sr,Sr-\.S:Xa\-"5 gg)a fleetrun is a module that spawns multiple distributed process on each training node for gpu training and cpu training. Usage: In both of single node training or multiple node training, this module launch a process on each of the given gpu card or cpu machine. GPU training: 1. for single node training with all visible gpu cards: fleetrun your_training_py (arg1 arg2 and all others) 2. for single node training with [0,4) cards fleetrun --gpus="0,1,2,3" your_training_py (arg1 arg2 and all others) 3. for multiple node training such as two node:192.168.0.16, 192.168.0.17 on 192.168.0.16: fleetrun --ips="192.168.0.16,192.168.0.17" \ your_training_py (arg1 arg2 and all others) on 192.168.0.17: fleetrun --ips="192.168.0.16,192.168.0.17" \ your_training_py (arg1 arg2 and all others) CPU training: 1. for single node training with multi servers and workers: fleetrun --server_num=2 --worker_num=2 your_training_py (arg1 arg2 and all others) 2. for multiple node training such as two node:192.168.0.16, 192.168.0.17 \ with 2 servers and 4 workers. on 192.168.0.16: fleetrun --servers="192.168.0.16:6170,192.168.0.17:6170" \ --workers="192.168.0.16,192.168.0.17,192.168.0.16,192.168.0.17" \ your_training_py (arg1 arg2 and all others) on 192.168.0.17: fleetrun --servers="192.168.0.16:6170,192.168.0.17:6171" \ --workers="192.168.0.16,192.168.0.17,192.168.0.16,192.168.0.17" \ your_training_py (arg1 arg2 and all others) 3. use gloo backend for multiple node training such as two node:192.168.0.16, 192.168.0.17 \ with 2 servers and 4 workers. (workers should set port) on 192.168.0.16: fleetrun --servers="192.168.0.16:6170,192.168.0.17:6170" \ --workers="192.168.0.16:6171,192.168.0.17:6171,192.168.0.16:6172,192.168.0.17:6172" \ your_training_py (arg1 arg2 and all others) on 192.168.0.17: fleetrun --servers="192.168.0.16:6170,192.168.0.17:6170" \ --workers="192.168.0.16:6171,192.168.0.17:6171,192.168.0.16:6172,192.168.0.17:6172" \ your_training_py (arg1 arg2 and all others) N) REMAINDERArgumentParser) framework) cloud_utils launch_utils)enable_elasticlaunch_elastic) DeviceModeDistributeModeParameterServerLauncherblock_windows_and_macos check_backend direct_startfind_free_ports get_clusterget_host_name_ip get_loggerloggerstart_local_trainersterminate_local_procswatch_local_trainersc[S5 [[U5R55Hup[USU35 M [S5 g)Nz0----------- Configuration Arguments -----------z: z0------------------------------------------------)printsortedvarsitems)argsargvalues _/var/www/html/banglarbhumi/venv/lib/python3.13/site-packages/paddle/distributed/fleet/launch.py_print_argumentsr!YsE <=T$Z--/0  Rw 1 <=c[SS9nURS5nURS[SSS9 URS[[R R S S 5S S9 URS [S SS9 URS[S SS9 [RR5(a&URS[S SS9 URSSS9 [RR5(a&URS[S SS9 URSSS9 URS[SS9 URS[S9 URS5nURS [S!S"S9 URS#[S S$S9 URS%[S S&S9 URS'[S(S)S9 URS*5nURS+[S,S-S9 URS.[S,S/S9 URS0[S,S1S9 URS2[S,S3S9 URS4[S,S5S9 URS6[S7S9 URS8[S9S9 URS:[S;S9 URS<[S=S9 URS>[S?S9 URS@5nURSA[SBS9 URSC[SDS9 URSE[SFS9 URSG[SHS9 URSI[SJSKS9 URSL[SMS9 URSN[S(SOS9 UR5upVU$)PzI Helper function parsing the command line options @retval ArgumentParser zstart paddle training using multi-process mode. see: http://www.paddlepaddle.org/documentation/docs/zh/1.6/user_guides/howto/training/cluster_howto.html#permalink-8--nccl2- ) descriptionzBase Parametersz --log_dirlogz7The path for each process's log. Default --log_dir=log/)typedefaulthelpz --backendPADDLE_DISTRI_BACKENDautozhSpecify the backend, can be gloo|nccl|bkcl|auto|heter. Default value is auto which prefers nccl or bkcl.z--nproc_per_nodeNzThe number of processes to launch on a node.In gpu training, it should be less or equal to the gpus number of you system(or you set by --gpus). And so each process can bound to one or average number of gpus.z --run_modez.run mode of job, can be:collective/ps/ps-heterz--gpuszmIt's for gpu training.For example:--gpus="0,1,2,3" will launch four training processes each bound to one gpu.z--selected_gpusgpus)destz--xpuszoIt's for xpu training. For example: --xpus="0,1,2,3" will launch four training processes each bound to one xpu.z--selected_xpusxpustraining_scriptzThe full path to the single GPU training program/script to be launched in parallel, followed by all the arguments for the training script)r&r(training_script_args)nargszCollective Parameters--ipsz 127.0.0.1z=Paddle cluster nodes ips, such as 192.168.0.16,192.168.0.17..z--cluster_topo_pathz|A json format file will be stored in this path which is usedto represent the cluster topology information for auto parallel.z--rank_mapping_pathzkA json format file will be stored in this path which is usedto map processes to machines for auto parallel.z--enable_auto_mappingFz>Set true to enable the lazy launch for auto-parallel scenario.zParameter-Server Parameters --serverszUser defined servers ip:port --workerszUser defined workers ip:port--coordinatorsz!User defined coordinators ip:port--heter_workersz>##67#6a #6H7 8}1+ 99iiG)+JA   %gY.@ "M w'I LL %hZy YKXJ  ' ' ) ) MQ  JJNN- . 6$S)9%:;  !j)J KK*:,7 8 ::>>, - 9RZZ^^,>?@J% =M9N,NOP   Z!HZTD$.Z!HI ,;K M8J"Is G*G/ c0UR(aA[URRS55S:a[SURS35eUR(aURS:XdS5eUR (a [S5eg) NrRrSzHCPUONLY launch only support single trainer, that is len(ips)=1, but got .cpuonlyz/CPUONLY launch only support run mode is CPUONLYz1CPUONLY launch can't have --servers as arguments.T)rWrZrX RuntimeErrorrun_modeserversrs r cpuonly_checkrv1s xxCs+,q0VW[W_W_V``a b   }}}} ) = ) ||NOO r"cURS:Xa [U5 UR(a[R/p!O[ R "U5up[R"5n[R"SUSUSU35 [R"S5nSnSnSn[RRS5b[RRS5nUR(GaHURcS5eUR =(d [R"S 5nU(Gd=[#S 5[RS '[#[%X55[RS '[&R("5R+5n [R,R/U S 5n[#U5[RS '[0R2SSn SR/U 5[RS'[#UR5[RS'[#UR5[RS'[ R4"X5unnXV4$[#S5[RS '[#[%X55[RS '[#UR5[RS'[#U5[RS '[#UR5[RS'[ R6"X5unnXV4$[R8"5(aFUS:wa@[R:"UR<XU5upV[R"SU35 XV4$[?XU5upV[R"SU35 XV4$)Ngloozparsed from args trainers_num:z mode:z devices:CUDA_VISIBLE_DEVICESirUzAThe cluster topology must be provided when enabling auto mapping.PADDLE_RANK_MAPPING_PATHTPADDLE_NEED_RANK_MAPPINGPADDLE_ENABLE_ELASTICzauto_parallel_rank_mapping.jsonrS PADDLE_ORIGINAL_CMD_ARGSPADDLE_CLUSTER_TOPO_PATHPADDLE_ENABLE_AUTO_MAPPINGFzget cluster from cloud:zget cluster from args:) backendrvenable_auto_mappingr GPUrget_device_proc_inforget_trainers_numrr]r@getenvrArBcluster_topo_pathrank_mapping_pathr?rpathlibPathcwdpathjoinsysargv1get_mapped_cluster_from_args_without_rank_mapping.get_mapped_cluster_from_args_with_rank_mappingr^get_cloud_clusterrWrn) rrcrd trainers_numcuda_visible_devicesclusterpodrjrr original_argss r get_cluster_infor?s5 ||vd +5>>2&*6*K*K + '//1L LL (f[MScRde99%;<G CJ zz~~()5ZZ^^$67  %%1 O 1!22 bii &7 !58YBJJ1 225t13BJJ. /,,.$$&C " 6! 699J5KBJJ1 2HHQRLM58XXm5LBJJ1 2589O9O5PBJJ1 27:((8BJJ3 4 NN @ <769ZBJJ1 225t13BJJ. /699O9O5PBJJ1 2589J5KBJJ1 27:((8BJJ3 4 KK  <  $ $ & &<1+<"44 HHkZ   .wi89 < - /   -gY78 <r"c[R"[RR55n[[R"SS55US'SUS'XS'UR US'U$)NPADDLE_WITH_GLOO03PADDLE_GLOO_RENDEZVOUSPADDLE_GLOO_FS_PATHr))copyr@rAr?rr)rtmp_dir global_envss r get_global_envsrs`))BJJOO-.K&)"))4F*L&MK"#,/K())0%&+/<%$6#((388Ab>2J(JW &%F SXXchhqn- - %  ;!$7 81 < L  ~~++-- ~~;;=  , , . . ~~::<   ;! :;-G\]i\j k ![!1C 4F!FG#C $4s;K7L$LM  !A %!** *!$$ $ !A % 12E1FmT`Sa b (((4466NN7799||L&(((%000 NN. ",, ,ms)M M*M M c[5n[5n[U5 URS:Xa [ U5nOIUR S:XdUR bS5e[ UR5 [RnURS:XaUR"S5 [UR5 [X5(a [X5 gU[R:Xa [U5 g[X5 g)aM Paddle distribution training entry ``python -m paddle.distributed.launch``. Usage: .. code-block:: bash :name: code-block-bash1 python -m paddle.distributed.launch [-h] [--log_dir LOG_DIR] [--nproc_per_node NPROC_PER_NODE] [--run_mode RUN_MODE] [--gpus GPUS] [--selected_gpus GPUS] [--ips IPS] [--servers SERVERS] [--workers WORKERS] [--heter_workers HETER_WORKERS] [--worker_num WORKER_NUM] [--server_num SERVER_NUM] [--heter_worker_num HETER_WORKER_NUM] [--http_port HTTP_PORT] [--elastic_server ELASTIC_SERVER] [--job_id JOB_ID] [--np NP] [--scale SCALE] [--host HOST] [--force FORCE] training_script ... Base Parameters: - ``--log_dir``: The path for each process's log. e.g., ``--log_dir=output_dir``. Default ``--log_dir=log``. - ``--nproc_per_node``: The number of processes to launch on a node. In gpu training, it should be less or equal to the gpus number of you system(or you set by --gpus). e.g., ``--nproc_per_node=8`` - ``--run_mode``: run mode of job, can be:collective/ps/ps-heter. e.g., ``--run_mode=ps``. Default ``--run_mode=collective``. - ``--gpus``: It's for gpu training. e.g., ``--gpus=0,1,2,3`` will launch four training processes each bound to one gpu. - ``--selected_gpus``: gpus aliases, recommend to use ``--gpus``. - ``--xpus``: It's for xpu training if xpu is available. e.g., ``--xpus=0,1,2,3``. - ``--selected_xpus``: xpus aliases, recommend to use ``--xpus``. - ``training_script``: The full path to the single GPU training program/script to be launched in parallel, followed by all the arguments for the training script. e.g., ``training.py`` - ``training_script_args``: The args of training_script. e.g., ``--lr=0.1`` Collective Parameters: - ``--ips``: Paddle cluster nodes ips, e.g., ``--ips=192.168.0.16,192.168.0.17``. Default ``--ips=127.0.0.1``. Parameter-Server Parameters: - ``--servers``: User defined servers ip:port, e.g., ``--servers="192.168.0.16:6170,192.168.0.17:6170"`` - ``--workers``: User defined workers ip:port, e.g., ``--workers="192.168.0.16:6171,192.168.0.16:6172,192.168.0.17:6171,192.168.0.17:6172"`` - ``--heter_workers``: User defined heter workers ip1:port1;ip2:port2, e.g., ``--heter_workers="192.168.0.16:6172;192.168.0.17:6172"`` - ``--worker_num``: Number of workers (It recommend to set when in the emulated distributed environment using single node) - ``--server_num``: Number of servers (It recommend to set when in the emulated distributed environment using single node) - ``--heter_worker_num``: Number of heter_workers in each stage (It recommend to set when in the emulated distributed environment using single node) - ``--heter_devices``: Type of heter_device in each stage - ``--http_port``: Gloo http Port Elastic Parameters: - ``--elastic_server``: etcd server host:port, e.g., ``--elastic_server=127.0.0.1:2379`` - ``--job_id``: job unique id, e.g., ``--job_id=job1`` - ``--np``: job pod/node number, e.g., ``--np=2`` - ``--host``: bind host, default to POD_IP env. Returns: ``None`` Examples 1 (collective, single node): .. code-block:: bash :name: code-block-example-bash1 # For training on single node using 4 gpus. python -m paddle.distributed.launch --gpus=0,1,2,3 train.py --lr=0.01 Examples 2 (collective, multi node): .. code-block:: bash :name: code-block-example-bash2 # The parameters of --gpus and --ips must be consistent in each node. # For training on multiple nodes, e.g., 192.168.0.16, 192.168.0.17 # On 192.168.0.16: python -m paddle.distributed.launch --gpus=0,1,2,3 --ips=192.168.0.16,192.168.0.17 train.py --lr=0.01 # On 192.168.0.17: python -m paddle.distributed.launch --gpus=0,1,2,3 --ips=192.168.0.16,192.168.0.17 train.py --lr=0.01 Examples 3 (ps, cpu, single node): .. code-block:: bash :name: code-block-example-bash3 # To simulate distributed environment using single node, e.g., 2 servers and 4 workers. python -m paddle.distributed.launch --server_num=2 --worker_num=4 train.py --lr=0.01 Examples 4 (ps, cpu, multi node): .. code-block:: bash :name: code-block-example-bash4 # For training on multiple nodes, e.g., 192.168.0.16, 192.168.0.17 where each node with 1 server and 2 workers. # On 192.168.0.16: python -m paddle.distributed.launch --servers="192.168.0.16:6170,192.168.0.17:6170" --workers="192.168.0.16:6171,192.168.0.16:6172,192.168.0.17:6171,192.168.0.17:6172" train.py --lr=0.01 # On 192.168.0.17: python -m paddle.distributed.launch --servers="192.168.0.16:6170,192.168.0.17:6170" --workers="192.168.0.16:6171,192.168.0.16:6172,192.168.0.17:6171,192.168.0.17:6172" train.py --lr=0.01 Examples 5 (ps, gpu, single node): .. code-block:: bash :name: code-block-example-bash5 # To simulate distributed environment using single node, e.g., 2 servers and 4 workers, each worker use single gpu. export CUDA_VISIBLE_DEVICES=0,1,2,3 python -m paddle.distributed.launch --server_num=2 --worker_num=4 train.py --lr=0.01 Examples 6 (ps, gpu, multi node): .. code-block:: bash :name: code-block-example-bash6 # For training on multiple nodes, e.g., 192.168.0.16, 192.168.0.17 where each node with 1 server and 2 workers. # On 192.168.0.16: export CUDA_VISIBLE_DEVICES=0,1 python -m paddle.distributed.launch --servers="192.168.0.16:6170,192.168.0.17:6170" --workers="192.168.0.16:6171,192.168.0.16:6172,192.168.0.17:6171,192.168.0.17:6172" train.py --lr=0.01 # On 192.168.0.17: export CUDA_VISIBLE_DEVICES=0,1 python -m paddle.distributed.launch --servers="192.168.0.16:6170,192.168.0.17:6170" --workers="192.168.0.16:6171,192.168.0.16:6172,192.168.0.17:6171,192.168.0.17:6172" train.py --lr=0.01 Examples 7 (ps-heter, cpu + gpu, single node): .. code-block:: bash :name: code-block-example-bash7 # To simulate distributed environment using single node, e.g., 2 servers and 4 workers, two workers use gpu, two workers use cpu. export CUDA_VISIBLE_DEVICES=0,1 python -m paddle.distributed.launch --server_num=2 --worker_num=2 --heter_worker_num=2 train.py --lr=0.01 Examples 8 (ps-heter, cpu + gpu, multi node): .. code-block:: bash :name: code-block-example-bash8 # For training on multiple nodes, e.g., 192.168.0.16, 192.168.0.17 where each node with 1 server, 1 gpu worker, 1 cpu worker. # On 192.168.0.16: export CUDA_VISIBLE_DEVICES=0 python -m paddle.distributed.launch --servers="192.168.0.16:6170,192.168.0.17:6170" --workers="192.168.0.16:6171,192.168.0.17:6171" --heter_workers="192.168.0.16:6172,192.168.0.17:6172" train.py --lr=0.01 # On 192.168.0.17: export CUDA_VISIBLE_DEVICES=0 python -m paddle.distributed.launch --servers="192.168.0.16:6170,192.168.0.17:6170" --workers="192.168.0.16:6171,192.168.0.17:6171" --heter_workers="192.168.0.16:6172,192.168.0.17:6172" train.py --lr=0.01 Examples 9 (elastic): .. code-block:: bash :name: code-block-example-bash9 python -m paddle.distributed.launch --elastic_server=127.0.0.1:2379 --np=2 --job_id=job1 --gpus=0,1,2,3 train.py r*rNz7When backend is not 'auto', run mode must be collectiverxzlaunch start with CPUONLY mode)rPrr!rrrsrr rrr rr rr)rrrs r launchr5sV =D \FT ||v0  }} , 0E E E dll#(33 ||v78 d,,t-.333$$(r"__main__)/__doc__rr@rrrrrargparserrpaddlerpaddle.distributed.fleetrr paddle.distributed.fleet.elasticrr %paddle.distributed.fleet.launch_utilsr r r r rrrrrrrrrr__all__r!rPrnrvrrrrrrr__name__r"r rs)V   .>K" >bJ)X Rj"J $R-jJ)Z z Hr"