jOUddlmZmZmZddlZddlmcmZddlm Z ddl m Z gZ ee ed<ejjGddZdS))DictListOptionalN)Tensor)2_scripted_functional_optimizer_deprecation_warning__all__ceZdZ ddeeded ed ed ed ed ededededededefdZdeeefdZ dS)_FunctionalAdagrad{Gz??绽|=TFparamslrlr_decay weight_decayinitial_accumulator_valuewarmup_lr_multiplierwarmup_num_iterseps coalesce_gradforeachfusedmaximize_allow_empty_param_listctd|||||||d|_| |_| |_| |_| |_t jtt j ttt j ffi|_ t|dkr| stdd|i|_|jdD]9}t j|j|t jdd|j |<:dS) N) stacklevel)rrrrrrrrz%optimizer got an empty parameter listrr )sumstep)rdefaultsrrrrtorchjitannotaterrstrstatelen ValueError param_group full_likedatatensor)selfrrrrrrrrrrrrrps [C:\Projects\Historical\venv\Lib\site-packages\torch/distributed/optim/functional_adagrad.py__init__z_FunctionalAdagrad.__init__s ;aHHHH ()B$8 0   +    Y''U\4U\@Q;R-R(SUWXX v;;!  $; DEE E%f-!(+  Aqv/HII S))DJqMM   gradientsc\|jd}g}g}g}g}t|t|kr6tddt|dzdt|zd\}}t|jd|D]\} } | || jz}|t j| z}|| || |j| } || d|| dt j 5tj |||||j d |j d |j d |j d ||j |j||jdd ddddS#1swxYwYdS)NrzEthe gradients passed in does not equal to the size of the parameters!zParams length: z. zGradients length: )FFrr rrrr) rrrrhas_sparse_gradrr has_complexr grad_scale found_inf)r)r'r(zip is_sparser" is_complexappendr&no_gradFadagradr!rrr) r-r2rparams_with_gradgrads state_sums state_stepsr4r5paramgradientr&s r/r z_FunctionalAdagrad.stepIs!(+ $& v;;#i.. ( (W3CKK33347s9~~778  (4$"4#3H#=yII 2 2OE8#8#55u/666  ''... X&&& 5)!!%,///""5=111 ]__   I=&!]>:z2M%( / 'j                      s5AF!!F%(F%N) r r r r r r rTFFFF) __name__ __module__ __qualname__rrfloatboolr0rr r1r/r r s !+.&)"%"(---V - - -  - $) -$- ------"&----^*d8F#34******r1r )typingrrrr"torch.optim._functionaloptim _functionalr=r,torch.distributed.optim._deprecation_warningrrr%__annotations__r#scriptr rJr1r/rRs''''''''''' ######### cZZZZZZZZZZr1