j, ddlZddlZddlmZmZddlmZmZmZm Z m Z m Z m Z ddl mcmcmcmZddlmZmZddlmZddlmZddlmZmZmZddlmZdd l m!Z!dd l"m#Z#dd l$m%Z%gd Z&e%e'Z(eGd dZ)GddZ*de ee+dfde ede+fdZ,de!de e e+e e-ffdZ.de)de ee+dfde edee-effdZ/dS)N) dataclassfield)AnyCallableDictListOptionalTupleUnion)eventsmetrics) WorkerSpec)LocalElasticAgent)DefaultLogsSpecs LogsSpecsSignalException)ChildFailedError)RendezvousParameters)parse_rendezvous_endpoint) get_logger) LaunchConfigelastic_launch launch_agentcZeZdZUdZeed<eed<eed<dZeeed<dZ e ed<d Z e ed <dZ e ed <d Z e ed <eeZee efed<dZeed<dZeed<dZeed<dZe ed<dZee ed<eeZee e fed<dZee ed<dZdS)raM Creates a rendezvous config. Args: min_nodes: Minimum amount of nodes that the user function will be launched on. Elastic agent ensures that the user function start only when the min_nodes amount enters the rendezvous. max_nodes: Maximum amount of nodes that the user function will be launched on. nproc_per_node: On each node the elastic agent will launch this amount of workers that will execute user defined function. rdzv_backend: rdzv_backend to use in the rendezvous (zeus-adapter, etcd). rdzv_endpoint: The endpoint of the rdzv sync. storage. rdzv_configs: Key, value pair that specifies rendezvous specific configuration. rdzv_timeout: Legacy argument that specifies timeout for the rendezvous. It is going to be removed in future versions, see the note below. The default timeout is 900 seconds. run_id: The unique run id of the job (if not passed a unique one will be deduced from run environment - flow workflow id in flow - or auto generated). role: User defined role of the worker (defaults to "trainer"). max_restarts: The maximum amount of restarts that elastic agent will conduct on workers before failure. monitor_interval: The interval in seconds that is used by the elastic_agent as a period of monitoring workers. start_method: The method is used by the elastic agent to start the workers (spawn, fork, forkserver). metrics_cfg: configuration to initialize metrics. local_addr: address of the local node if any. If not set, a lookup on the local machine's FQDN will be performed. local_ranks_filter: ranks for which to show logs in console. If not set, show from all. ..note: `rdzv_timeout` is a legacy argument that will be removed in future. Set the timeout via `rdzv_configs['timeout']` min_nodes max_nodesnproc_per_nodeN logs_specsrun_id default_rolerole rdzv_endpointetcd rdzv_backend)default_factory rdzv_configs rdzv_timeout max_restartsg?monitor_intervalspawn start_methodlog_line_prefix_template metrics_cfg local_addrcd}|jdkr|j|jd<nd|jvr ||jd<|jt|_dSdS)Nir(timeout)r)r'rr)selfdefault_timeouts OC:\Projects\Historical\venv\Lib\site-packages\torch/distributed/launcher/api.py __post_init__zLaunchConfig.__post_init__Zsf   " "+/+#>#>L$sCx.>>>L#L#!e!!!L#.2hsm222"'%"="="=Kc3h=== $J $$$ 1 1 1 1 1r8rc:eZdZdZdedeeedffdZdZ dS)ra Launches an torchelastic agent on the container that invoked the entrypoint. 1. Pass the ``entrypoint`` arguments as non ``kwargs`` (e.g. no named parameters)/ ``entrypoint`` can be a function or a command. 2. The return value is a map of each worker's output mapped by their respective global rank. Usage :: def worker_fn(foo): # ... def main(): # entrypoint is a function. outputs = elastic_launch(LaunchConfig, worker_fn)(foo) # return rank 0's output return outputs[0] # entrypoint is a command and ``script.py`` is the python module. outputs = elastic_launch(LaunchConfig, "script.py")(args) outputs = elastic_launch(LaunchConfig, "python")("script.py") config entrypointNc"||_||_dSN)_config _entrypoint)r4rDrEs r6__init__zelastic_launch.__init__s  %r8cRt|j|jt|SrG)rrHrIlist)r4argss r6__call__zelastic_launch.__call__sDL$*:DJJGGGr8) r9r:r;r<rr rr?rJrNrBr8r6rrfsd4&&(C-.&&&&HHHHHr8rrErMreturnct|tr|jSt|tr,|tjkrt d|DdS|SdS)aRetrieve entrypoint name with the rule: 1. If entrypoint is a function, use ``entrypoint.__qualname__``. 2. If entrypoint is a string, check its value: 2.1 if entrypoint equals to ``sys.executable`` (like "python"), use the first element from ``args`` which does not start with hifen letter (for example, "-u" will be skipped). 2.2 otherwise, use ``entrypoint`` value. 3. Otherwise, return empty string. c32K|]}|ddk|VdS)r-NrB).0args r6 z'_get_entrypoint_name..s*>>A#  >>r8r) isinstancerr9r?sys executablenext)rErMs r6_get_entrypoint_namerZsh*h''"" J $ $  ' '>>>>>CC C rr8rdzv_parametersc|jdkrdS|j}|}|stdt |d\}}|dkrtd|d||fS)Nstatic)NNzKEndpoint is missing in endpoint. Try to add --master-addr and --master-portr() default_portzport is missing in endpoint: z. Try to specify --master-port)backendendpointstrip ValueErrorr)r[r` master_addr master_ports r6_get_addr_and_portres(**|'H~~H   Y    9PRSSSKb TH T T T     %%r8rDc|jsGttjj}t d|||_t||}t d||j |j |j |j|j |j |j|j|j|jj|jd t)d |j |j |j|j |j |jd|j}t-|\}}t/|j|j |t3|t5j||j|j|||j }t9||j|j|j} d} t?j t?j!|j| "} tGj$| %| &rtO|| j(| j)| r|j*+SS#tN$rtX$r*d } tGj$| -t\$r(tGj$| -wxYw#| r|j*+wwxYw) Nz3config has no run_id, generated a random run_id: %saStarting elastic_operator with launch configs: entrypoint : %(entrypoint)s min_nodes : %(min_nodes)s max_nodes : %(max_nodes)s nproc_per_node : %(nproc_per_node)s run_id : %(run_id)s rdzv_backend : %(rdzv_backend)s rdzv_endpoint : %(rdzv_endpoint)s rdzv_configs : %(rdzv_configs)s max_restarts : %(max_restarts)s monitor_interval : %(monitor_interval)s log_dir : %(log_dir)s metrics_cfg : %(metrics_cfg)s ) rErrrr r%r#r'r+r,log_dirr0)r_r`r rrr1) r"local_world_sizerErM rdzv_handlerr+r,rcrdr1)specrr.r/T)namefailuresFrB)/r r?uuiduuid4r=loggerwarningrZinforrrr%r#r'r+r,r root_log_dirr0rr1rerr"tuple rdzv_registryget_rendezvous_handlerrr.r/r initialize_metrics MetricsConfigrunr recordget_event_succeeded is_failedrrl return_valuesrishutdownrget_event_failed Exception) rDrErMr entrypoint_namer[rcrdrjagent shutdown_rdzvresults r6rrs =TZ\\%&&LfUUU *:t<rs ((((((((DDDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEE55555555AAAAAAXXXXXX NMMMMMEEEEEEPPPPPP>>>>>> = < < H   @1@1@1@1@1@1@1 @1F$H$H$H$H$H$H$H$HNhT)*26s),&)& 8C=(3- '(&&&&&k) k)hT)*k) s)k) #s(^ k)k)k)k)k)k)r8