j7UddlZddlZddlmZddlmZmZmZmZm Z m Z m Z m Z m Z ddlZddlmZddlmZddlmZddlmZgZe eed<ejeZGdd ejZd e ed efd ZdS) N)deepcopy) AnyCallable CollectionDictListMappingOptionaloverloadUnion)optim) ShardedTensor)FullyShardedDataParallel__all__c eZdZdZ ddeeeeje ffde j de e eeefde ejddf dZd Zdeeeffd Zeddd Zed egefdefdZdd e egefde efdZedeejeffdZdeeefddfdZdeeefddfdZddZdeeeffdZdeeeffdZdS)_NamedOptimizera ``_NamedOptimizer`` takes a dict of parameters and exposes ``state_dict`` by parameter key. We replace the original key (number) in an optim to the fully qualified name (FQN) string. User can initialize the optim as they initialize a PyTorch optim, the only difference is that they also need to pass in the FQN of each parameters. Args: named_parameters (Mapping[str, Union[torch.Tensor, ShardedTensor]]): Mapping from FQN to parameter. optimizer_class (optim.Optimizer): The class of optimizer to instantiate. param_groups (Collection[Mapping[str, Any]]): `param_groups` to pass to optimizer if specified. The key of the inner map needs to be FQNs. Default: None module (nn.Module): the module whose parameters to updated by the optimizer. args: arguments to pass to the optimizer constructor. kwargs: arguments to pass to the optimizer constructor. Example:: >>> # xdoctest: +SKIP("distributed") >>> from torch import optim >>> from torch.distributed.optim import _NamedOptimizer >>> >>> # Define the named optimizer. >>> m = Model(...) >>> named_optim = _NamedOptimizer(m.named_parameters(), optim.SGD) >>> # Forward pass + backward pass. >>> named_optim.step() >>> ... >>> # Call state_dict for the named optimizer returns a FQN state_dict. >>> named_optim.state_dict() Warning: This API is still in development and subject to change. TODO: Add tutorial for _NamedOptimizer. TODO: Add documentation in the docstring for the public attributes like self.param_groups and self.named_parameters. Nnamed_parametersoptimizer_class param_groupsmodulereturnctjd||_|t ||_||jn|}||g|Ri||_||_ |,t|j |_ ntjdd|jD}g} |D]?} | dD]4} | |vrt!d| d| || 5@| |_ |jj|_dS)Nz'torch.distributed.optim._NamedOptimizerzvSince we pass in param_groups, we will use param_groups to initialize the optimizer, not all parameters of the module.ci|]\}}|| Sr.0keyparams XC:\Projects\Historical\venv\Lib\site-packages\torch/distributed/optim/named_optimizer.py z,_NamedOptimizer.__init__..fsWWW:3E3WWWparamszExpect param name z% found in param group but is missing.)torch_C_log_api_usage_oncer_param_groups_checkdictrvalues _optimizerrlistkeysordered_param_keyswarningswarnitems ValueErrorappend) selfrrrrargskwargsparams_for_optimizer param_to_keyr,grouprs r__init__z_NamedOptimizer.__init__Is $$%NOOO;G   """ $%5 6 6.:.BD ! ( ( * * *  */          &*4+@+E+E+G+G&H&HD # # MN   XW9N9T9T9V9VWWWL!# % C C"8_CCEL00(]]]]'--l5.ABBBB C '9D # O8r!c~|j|jD]}t|ts Jdd|vs Jd|d}t|tjr|g}t |}|D]@}t|tjs$t dtj|zA||d<dSdS)Nparam group must be a dictr"z#param group must contain key paramsz>optimizer can only optimize Tensors, but one of the params is )r isinstancer'r#Tensorr* TypeErrortypename)r2 param_groupr"rs rr&z#_NamedOptimizer._param_groups_checkss   (#0 / / !+t44RR6RRR4;...0U...$X.fel33&$XFf#E%eU\::'8:?.:O:OP )/ H%% ) ( / /r!cj}|d}fd|dD}g}|D]n}fd|dD}dt|i}|D]\}} |dkrt | ||<||o||dS)z Return the ``state_dict`` of the optimizer. Instead of using number to index parameters, we will use module fully qualified name (FQN) as the key. rc2i|]\}}j||Srr,)rst_key state_valr2s rr z._NamedOptimizer.state_dict..s7   !   #F +Y   r!statec*g|]}j|SrrB)rrr2s r z._NamedOptimizer.state_dict..s!VVVU$1%8VVVr!r")rEr)r) state_dictr/sortedrr1_post_state_dict) r2rHr ret_state ret_groupsr7 param_keys ret_groupkvs ` rrHz_NamedOptimizer.state_dicts_//11 !.1     %/%8%>%>%@%@    ! ) )EVVVVeHoVVVJ!6*#5#56I  / /1==#+A;;IaL   i ( ( ( ($$y*%U%UVVVr!.closurecdSNrr2rQs rstepz_NamedOptimizer.step r!cdSrSrrTs rrUz_NamedOptimizer.steprVr!c8|j|S)z Perform a single optimization step. This will call :meth:`torch.optim.Optimizer.step` on the wrapped optimizer. rQ)r)rUrTs rrUz_NamedOptimizer.steps##G#444r!c|jjSrS)r)rE)r2s rrEz_NamedOptimizer.states $$r!rHc 0|j}||}|d}|d}t|dkrt dt |jD]J\}}||vrt||t||kr>t dt||d|dt||||D]\}}|||vrt d|d|d|||} t|trt| tsJt| } t| } | | krt d | d | d |d |t| | D]6\} } | j | j 70t|t jrEt| t jsJ|| t%| |||<L|d }|d }i}|D])}t'|d}||t)|<*i}|D]A}g}|dD]"}||j|#||t)|<B|D]\}}||vr ||}t|t|kr3t dt|d|d t|d|D]:}||vrt d|d|d|dkrt%||||<;|j|dS)a Define the default behavior to load a state_dict for ``_NamedOptimizer``. Sample Code ``` my_model = MyModule() optimizer = _NamedOptimizer(my_model.named_parameters(), Adagrad) ... optim_state_dict = optimizer.state_dict() ... ... optimizer.load_state_dict(optim_state_dict) ... ``` Args: state_dict (Dict[str, Any]) : A ``state_dict`` to load into the optimizer. Note that this state dict update is performed in place. .. note:: PyTorch is using lazy init to initialize the optim states. So it is possible that there is no optim state when user call ``load_state_dict`` and for ``_NamedOptimizer`` we make it stricter that users can only call ``load_state_dict`` after the state is initialized. By doing this, we can validate the optim ``state_dict`` to be loaded. rErzJExpects the optim to be initialized before load but found not initialized.zExpects equal length as z for parameter z but found: zExpects state z but not found.z"Expects equal number of shards as z but found z for /rr"z"Expects equal param_group size as z for group .zExpects group key z to be in group z in `state_dict` but is missing.N)r)rH_pre_load_state_dictlenr0 enumerater,r+r/r;r local_shardsziptensordetachcopy_r#r<rr*_gen_param_group_keyr1load_state_dict)r2rHnew_state_dictrE new_stateidx param_key state_keyrD src_state_val num_shardsnum_new_shardsshard src_shardsrc_param_groupsnew_param_groups src_group_mapr7rM new_group_map new_group group_key src_grouprOs rrgz_NamedOptimizer.load_state_dicts63355..z:: 7#"7+ y>>Q  \ ((?@@ H HNC ,,5#$$IcN(;(;;; Bs9S>/B/BBBS\BBjmnst}n~jjBB)2#(<(<(>(> H H$ 9E)$444$]]]9]]]!&i 0 ; i77H%m]CCCCC!$Y%;%;%=%=!>!>J%()C)C)E)E%F%FN!^33(EEE\fEEmvEEzCEE-0!..00-2L2L2N2N--FF(y ++--33I4DEEEEF 5<88H%mU\BBBBB$$&&,,];;;;080G0GIcN9--/ H4&n5).9 % D DEeHo..J>CM.z:: ; ; ) H HIJ&x0 F F !!$"9)"DEEEE>GM.z:: ; ;$1$7$7$9$9 : : Iy --%i0I9~~Y// {Y{{T]{{jmnwjxjx{{{ : :I%%$kQkk kkk==#+IaL#9#9IaL  : ''77777r!r?ct|ts Jd|d}t|tjr|g|d<nt ||d<d|jD}|dD]5}||vrtd|j ||6|j ||j j |_ dS)z Add a param group to the :class:`_NamedOptimizer` s `param_groups`. Warning: This API is still in development and subject to change. r:r"ci|]\}}|| Srrrs rr z3_NamedOptimizer.add_param_group..)sSSSzsEsSSSr!z%some parameters are not in the moduleN) r;r'r#r<r*rr/r0r,r1r)add_param_groupr)r2r?r"r6rs rr{z_NamedOptimizer.add_param_groups +t,,JJ.JJJ,X& fel + + 1%+HK ! !$(LLK !SST5J5P5P5R5RSSS  * @ @EL(( !HIII  # * *<+> ? ? ? ? '' 444 O8r!c|jD]A}|jr8tj|}tj||_B|ddS)z Run a dummy optimizer step, which allows to initialize optimizer state because we do lazy init for most optimizers. This allows doing in-place loading of optimizer state from a checkpoint. NrY) rr( requires_gradr# zeros_likeautogradVariablegradrU)r2rts r init_statez_NamedOptimizer.init_state3so *1133 8 8E" 8$U++"^44Q77  $ r!c~t|jtr"tj|j|j|dS|S)NT)is_named_optimizer)r;rFSDPoptim_state_dict_to_loadr)r2rHs rr^z$_NamedOptimizer._pre_load_state_dict@sG dk4 ( ( 0 T_jT r!czt|jtr tj|j|j||SrS)r;rroptim_state_dictr)rs rrJz _NamedOptimizer._post_state_dictIs8 dk4 ( ( L  !$+t K K Kr!)NN).)rQNrNrS)rN) __name__ __module__ __qualname____doc__r strr r#r<rr Optimizerr rrnnModuler8r&rrHr rUrfloatpropertyrErgr{rr^rJrr!rrrs[))^AE&* (9(9!#uU\=-H'I"IJ(9(9z'#s(*;<= (9 # (9 (9(9(9(9T///"WDcNWWWW4    X  HRY/ E   X 55HXb%i%895Xe_5555%wu|S01%%%X%f8'#s(*;f8f8f8f8f8P9738+<999990     $sCx.d38nr!rrMrcFdt|S)zGConcatenate all param keys as a unique indentifier for one param group.r\)joinrI)rMs rrfrfQs 88F:&& ' ''r!) loggingr-copyrtypingrrrrrr r r r r#torch.nnrr 'torch.distributed._shard.sharded_tensorrtorch.distributed.fsdprrrr__annotations__ getLoggerrloggerrrrfrr!rrsy                       AAAAAACCCCCCc  8 $ $qqqqqeoqqqh (T#Y(3((((((r!