j<%ZddlZddlmZmZmZmZmZmZmZm Z m Z ddl Z ddl m Z ddlmZddlmZddlmZmZdZGdd eZd ed ed efd ZGddZee dde jdeee j jd e jfdZde jd efdZdS)N) AnycastDictIterableListNoReturnOptionalSetTuple)_State)DistributedDataParallel) _get_registrycontractc PeZdZdfd Zefdejdeejdeejde ddf dZ dd Z dejdeejddfd Z dd Z dd Zdejd eedfdee efdefdZdejdeejdejdejfdZxZS)_ReplicateStatereturnNcttj|_d|_tj|_|j|_g|_d|_ d|_ i|_ g|_ dS)NF) super__init__nn ParameterListmodulehas_initialized _param_list _orig_module _param_names_no_sync _init_args _init_kwargs_comm_hook_args)self __class__s XC:\Projects\Historical\venv\Lib\site-packages\torch/distributed/_composable/replicate.pyrz_ReplicateState.__init__sz !#!1!3!3 %*-/-=-?-?!K')# 59,.*,rignored_modulesignored_paramsprefixct|rdS||vrdS|tkr|dnt}|dD]A\}}||vr8|j||j||B|D]"\}} || ||||#dS)N.F)recurse)r))_is_fully_sharded_ROOT_MODULE_PREFIXnamed_parametersrappendrnamed_children_collect_params) r#rr'r(r)recurse_prefixnpname child_modules r%r2z_ReplicateState._collect_params s V $ $  F _ $ $ F#&999vLLLL?R ++E+:: A ADAq&& ''***!((N)?A)?)?@@@"("7"7"9"9   D,  (0$00 !      r&c\tjdfd}|dS)NT) recursivecjJjjijd_i_dS)N)r initr!register_comm_hook)r#sr% _lazy_initz-_ReplicateState.lazy_init.._lazy_initAsT?... DIt <$*; < < <  # # % % % DO "D   r&)torch_disable_dynamo)r#r>s` r% lazy_initz_ReplicateState.lazy_init@sF   . . . # # # # / . #  r&c |jrdSd|_||_d|D}|D]8}t|r'||9ddlm}|||||||d|vr\|d:|d}t|tj r|j dkrd|d<n |g|d<nd|d<| dt|jfi||_t#j|jt&|j_dS) NTc@h|]}|D]}|Sr;) parameters).0mr5s r% z'_ReplicateState.init..Vs-MMMallnnMM!MMMMr&r)_localize_dtensor)r( device_idcpu device_ids)rrmodulesr-updaterD%torch.distributed.tensor.parallel.ddprHr2 isinstancer?devicetypepopr r_ddpweakrefref replicatestate _ddp_weakref)r#rr'kwargsr( submodulerHrIs r%r<z_ReplicateState.initKs~    F# MM_MMM)) > >I ++ >%%i&:&:&<&<===KKKKKK&@@@@ V_nEEE & k".";/ i6679>U;R;R+/F<((-6;F<(('+|$ JJ{ # # #+D,<GGGG 4;K 4J4J  $$111r&cv|jD]\}}|jj|i||jdSN)r"rSr=clear)r# comm_args comm_kwargss r%r=z"_ReplicateState.register_comm_hookusS&*&: D D "I{ (DI () C{ C C C C ""$$$$$r&c"||_||_dSr\)r r!r#argsrYs r%record_init_argsz _ReplicateState.record_init_argszs"r&rb.rYc|js|jr||j |j_|jj|i|Sr\)r r!rArrSrequire_backward_grad_sync _pre_forward)r#rrbrYs r%forward_pre_hookz _ReplicateState.forward_pre_hook~sQ ? d/  NN   37=/@ ,%ty%t6v666r&inputoutputc6|j|Sr\)rS _post_forward)r#rrhris r%forward_post_hookz!_ReplicateState.forward_post_hooks y&&v...r&rN)__name__ __module__ __qualname__rr.rModuler Parameterstrr2rAr<r=rcr rrrgr?Tensorrl __classcell__)r$s@r%rrs - - - - - -&*  RYBL)     @    (K (KRY(K  (K(K(K(KT%%%% ####7i7',S#X7@DS#X7 7777/ /U\"/ /  ////////r&rrbrYrc td)NzGDDP does not support deepcopy. Please use state dict for serialization.)AssertionError)rbrYs r%unimplemented_deepcopyrxs Q  r&c,eZdZdZdeddfdZddZdS)DDPc>|jd}|j|g|Ri|S)z Override ``__new__`` to remove the DDP class and directly construct the original class for cases like indexing into a container module. )__mro____new__)clsrbrYorig_clss r%r~z DDP.__new__s2;q>x:4:::6:::r&requires_gradient_syncrNcF| t|_dS)a Sets if the module should sync gradients. This can be used to implement gradient accumulation without communication. Args: requires_gradient_sync (bool): Whether to reduce gradients for the module's parameters. N)rVrWr)r#rs r%set_requires_gradient_synczDDP.set_requires_gradient_syncs".D)C &&&r&cnt|j||fdSr\)rVrWr"r0ras r%r=zDDP.register_comm_hooks/-44dF^DDDDDr&rm)rnrorpr~boolrr=r;r&r%rzrzse;;; D D$ D D D DEEEEEEr&rz) state_clsrr'c ptjdd|vrLt|dttjfs%t dt|dt|rt d|i}nt|}ttt |}||jd|dd}|Sd d lm}||}||kr2d d lm}m}||||||j|j||fi||j} d t6i} td | jt:| f| } | |_|S)zReplicates a module Args: module (torch.nn.Module): module to replicate Example:: >>> # xdoctest: +REQUIRES(module:torch._C._distributed_c10d) >>> module = nn.Linear(3, 3) >>> replicate(module) ztorch.distributed.replicaterIz6Expected device_id to be int or torch.device, but got zGCannot apply `replicate()` on a Module already managed by `fully_shard`NT) with_kwargs device_meshr)_mesh_resources)rH_reconstruct_dtensor __deepcopy__rz)r?_C_log_api_usage_oncerOintrP RuntimeErrorrQr-setrrrVrWregister_forward_pre_hookrggettorch.distributed.device_meshr get_root_meshrNrHrregister_forward_hookrlrcr$rxrnrz) rr'rYrWrr root_meshrHrrdctnew_clss r%rVrVs  H  !>???f&-U\/BCC 7{ 34477      U   o.. )//&"9"9 : :E $$U%;$NNN**]D11KAAAAAA#11+>>   # #          , ,-A B B B  ( (): ; ; ;   !8999E6?==f===  C 1 2C'''#sS99GF Mr&c0t|}|dSd|vS)z+Check if module is marked with fully_shard.NF fully_shard)r)rregistrys r%r-r-s%V$$Hu H $$r&r\)rTtypingrrrrrrr r r r?torch.nnr#torch.distributed._composable_stater torch.nn.parallelr rrr.rrxrzrqrVrr-r;r&r%rsRRRRRRRRRRRRRRRRRRRRRR 666666555555--------{/{/{/{/{/f{/{/{/|#EEEEEEEE4 O$$$<@EE IEhux78EY EEE%$EP%bi%D%%%%%%r&