Bj"? ddlZddlZddlZddlmZddlmZmZddlm Z ddl Z ddl m cm cmcmZddlmZmZddlmZmZddlmZddlmZdd lmZmZmZdd lm Z m!Z!m"Z"dd l#m$Z$dd l%m&Z&dd l'm(Z(ddl)m*Z*ddl+m,Z,gdZ-e*e.Z/eGddZ0GddZ1dee2zdzde3e de2fdZ4de&de5e2dze6dzffdZ7de0dee2zdzde3e de8e6e ffdZ9dS)N)Callable) dataclassfield)Any)get_default_numa_optionsjustknobs_check)eventsmetrics) WorkerSpec)create_healthcheck_server)_AliveCallbackProxyLocalElasticAgentTORCHELASTIC_HEALTH_CHECK_PORT)DefaultLogsSpecs LogsSpecsSignalException)ChildFailedError)RendezvousParameters)parse_rendezvous_endpoint) get_logger) NumaOptions) LaunchConfigelastic_launch launch_agentceZdZUdZeed<eed<eed<dZedzed<dZe ed<d Z e ed <dZ e ed <d Z e ed <e eZee efed<dZeed<dZeed<dZeed<dZe ed<dZe dzed<e eZee e fed<dZe dzed<dZe ed<dZedzed<dZe ed<dZee dzed <dZee dzed!<d"Z e!ed#<dZ"edzed$<d%Z#dS)&ra Creates a rendezvous config. Args: min_nodes: Minimum amount of nodes that the user function will be launched on. Elastic agent ensures that the user function start only when the min_nodes amount enters the rendezvous. max_nodes: Maximum amount of nodes that the user function will be launched on. nproc_per_node: On each node the elastic agent will launch this amount of workers that will execute user defined function. rdzv_backend: rdzv_backend to use in the rendezvous (zeus-adapter, etcd). rdzv_endpoint: The endpoint of the rdzv sync. storage. rdzv_configs: Key, value pair that specifies rendezvous specific configuration. rdzv_timeout: Legacy argument that specifies timeout for the rendezvous. It is going to be removed in future versions, see the note below. The default timeout is 900 seconds. run_id: The unique run id of the job (if not passed a unique one will be deduced from run environment - flow workflow id in flow - or auto generated). role: User defined role of the worker (defaults to "trainer"). max_restarts: The maximum amount of restarts that elastic agent will conduct on workers before failure. monitor_interval: The interval in seconds that is used by the elastic_agent as a period of monitoring workers. start_method: The method is used by the elastic agent to start the workers (spawn, fork, forkserver). metrics_cfg: configuration to initialize metrics. local_addr: address of the local node if any. If not set, a lookup on the local machine's FQDN will be performed. local_ranks_filter: ranks for which to show logs in console. If not set, show from all. event_log_handler: name of the event logging handler as registered in `elastic/events/handlers.py `_. duplicate_stdout_filters: If non-empty, duplicates stdout to a file containing only lines that match _any_ of the filter strings. duplicate_stderr_filters: If non-empty, duplicates stderr to a file containing only lines that match _any_ of the filter strings. virtual_local_rank: Enable virtual local rank mode for workers (defaults to False). When enabled, LOCAL_RANK is set to 0 for all workers and CUDA_VISIBLE_DEVICES is adjusted so each worker accesses its assigned GPU at device index 0. shutdown_timeout: Time in seconds to wait for graceful shutdown of workers before sending SIGKILL. Can also be set via TORCH_ELASTIC_SHUTDOWN_TIMEOUT environment variable. Defaults to 30 seconds. .. note:: `rdzv_timeout` is a legacy argument that will be removed in future. Set the timeout via `rdzv_configs['timeout']` min_nodes max_nodesnproc_per_nodeN logs_specsrun_id default_rolerole rdzv_endpointetcd rdzv_backend)default_factory rdzv_configs rdzv_timeout max_restartsg?monitor_intervalspawn start_methodlog_line_prefix_template metrics_cfg local_addrnullevent_log_handler numa_optionszSIGTERM,SIGINT,SIGHUP,SIGQUITsignals_to_handleduplicate_stdout_filtersduplicate_stderr_filtersFvirtual_local_rankshutdown_timeoutcRd}|jdkr|j|jd<nd|jvr ||jd<|jt|_|jxt jrZt j|j kr3t|_t d|j|j 4ttjdd|_ dS|j dkrt%d|j dS) Nir)timeoutzUsing default numa options = %rTORCH_ELASTIC_SHUTDOWN_TIMEOUT30rz+shutdown_timeout must be non-negative, got )r*r(rrr5torchcuda is_available device_countrrloggerinfor:intosenvironget ValueError)selfdefault_timeouts a/var/www/html/Carbon-Document/venv/lib/python3.11/site-packages/torch/distributed/launcher/api.py __post_init__zLaunchConfig.__post_init__|s.   " "+/+SUU ' &)$__name__ __module__ __qualname____doc__rE__annotations__rrr!strr#r$r&rdictr(rr*r,r-floatr/r0r1r2r4r5rr6r7listr8r9boolr:rMrNrLrr.s22hNNNNNN#'J D '''FCD#M3L##(5#>#>#>L$sCx.>>>L#L#!e!!!L#+/cDj///"'%"="="=Kc3h===!Jd !!!#s###'+L+$+++<s<<<15d3i$.55515d3i$.555$$$$#'cDj'''rNrc4eZdZdZdedeezdzfdZdZdS)ra Launches an torchelastic agent on the container that invoked the entrypoint. 1. Pass the ``entrypoint`` arguments as non ``kwargs`` (e.g. no named parameters)/ ``entrypoint`` can be a function or a command. 2. The return value is a map of each worker's output mapped by their respective global rank. Usage :: def worker_fn(foo): # ... def main(): # entrypoint is a function. outputs = elastic_launch(LaunchConfig, worker_fn)(foo) # return rank 0's output return outputs[0] # entrypoint is a command and ``script.py`` is the python module. outputs = elastic_launch(LaunchConfig, "script.py")(args) outputs = elastic_launch(LaunchConfig, "python")("script.py") config entrypointNc"||_||_dSN)_config _entrypoint)rJr[r\s rL__init__zelastic_launch.__init__s  %rNcRt|j|jt|Sr^)rr_r`rW)rJargss rL__call__zelastic_launch.__call__sDL$*:DJJGGGrN) rOrPrQrRrrrTrardrYrNrLrrsa4&&sNT)&&&&HHHHHrNrr\rcreturnct|tr|jSt|tr,|tjkrt d|DdS|SdS)aRetrieve entrypoint name with the rule: 1. If entrypoint is a function, use ``entrypoint.__qualname__``. 2. If entrypoint is a string, check its value: 2.1 if entrypoint equals to ``sys.executable`` (like "python"), use the first element from ``args`` which does not start with hifen letter (for example, "-u" will be skipped). 2.2 otherwise, use ``entrypoint`` value. 3. Otherwise, return empty string. c32K|]}|ddk|VdS)r-NrY).0args rL z'_get_entrypoint_name..s*>>A#  >>rNr ) isinstancerrOrTsys executablenext)r\rcs rL_get_entrypoint_namerpsh*h''"" J $ $  ' '>>>>>CC C rrNrdzv_parametersc|jdkrdS|j}|}|stdt |d\}}|dkrtd|d||fS)Nstatic)NNzKEndpoint is missing in endpoint. Try to add --master-addr and --master-portr)) default_portzport is missing in endpoint: z. Try to specify --master-port)backendendpointstriprIr)rqrv master_addr master_ports rL_get_addr_and_portrzs(**|'H~~H   Y    9PRSSSKb TH T T T     %%rNr[cR|jsGttjj}t d|||_t||}t did|d|j d|j d|j d|jd|j d |j d |jd |jd |jd |jjd|jd|jd|jd|jd|jd|jt3d"|j |j |j|j |j |jd|j}t7|\}}|jt8jd<d}d} t9jt>} | tAddr tC} tE| t | d}|#t d| n0#tH$r#t ddd}d} YnwxYwtK|j&|j |tO|tQj)||j|j|||j|j|j|j|j|j*} tW| |j|j,|j-|j.| } | | /| j0d} tcj2tcj3|j| 4}tkj6| 7|j|8rts||j:!|j;| r| j<=SS#tr$rt|$r0d} tkj6| ?|jtH$r.tkj6| ?|jwxYw#| r| j<=wwxYw)#Nz3config has no run_id, generated a random run_id: %saRStarting elastic_operator with launch configs: entrypoint : %(entrypoint)s min_nodes : %(min_nodes)s max_nodes : %(max_nodes)s nproc_per_node : %(nproc_per_node)s run_id : %(run_id)s rdzv_backend : %(rdzv_backend)s rdzv_endpoint : %(rdzv_endpoint)s rdzv_configs : %(rdzv_configs)s max_restarts : %(max_restarts)s monitor_interval : %(monitor_interval)s log_dir : %(log_dir)s metrics_cfg : %(metrics_cfg)s event_log_handler : %(event_log_handler)s numa_options : %(numa_options)s signals_to_handle : %(signals_to_handle)s duplicate_stdout_filters : %(duplicate_stdout_filters)s duplicate_stderr_filters : %(duplicate_stderr_filters)s r\rrrr!r&r$r(r,r-log_dirr1r4r5r6r7r8)rurvr!rrr2TORCHELASTIC_SIGNALS_TO_HANDLEzNai_infra/pytorch_distributed:torchelastic_enable_healthcheck_before_rendezvousF)default<)alive_callbackportr<z>Started early health check server on port %s before rendezvousz)Failed to start early health check serverT)exc_info)r#local_world_sizer\rc rdzv_handlerr,r-rxryr2r4r5r7r8r9)specrr/r0r:health_check_server)namefailuresrY)@r!rTuuiduuid4rErCwarningrprDrrrr&r$r(r,r-r root_log_dirr1r4r5r6r7r8rr2rzrFrGgetenvrrr r start Exceptionr r#tuple rdzv_registryget_rendezvous_handlerr9rr/r0r: set_delegate_get_alive_timer initialize_metrics MetricsConfigrunr recordget_event_succeeded is_failedrr return_valuesrshutdownrget_event_failed)r[r\rcr!entrypoint_namerqrxryralive_callback_proxyhealthcheck_portragent shutdown_rdzvresults rLrrs  =TZ\\%&&LfUUU *:t<rs  $$$$$$(((((((( EEEEEEEEEEEEEEEKKKKKKKK55555555AAAAAA  NMMMMMEEEEEEPPPPPP>>>>>>****** = < < H   iiiiiii iX$H$H$H$H$H$H$H$HNX^d%:$s)PS(&)& 3:sTz !"&&&&&^) ^)3%^) s)^) #s(^ ^)^)^)^)^)^)rN