˲jhQ$dZddlmZmZmZmZddlZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZddgZGd deZd d ed e d e d ed e d ze_deedeedeedeedeedeededededededededededef dZdeedeedeedeedeedeededededededededededef d Ze e! d%deedeedeedeedeedeed#eededededededededededef"d$ZdS)&z)Implementation for the RMSprop algorithm.)castListOptionalUnionN)Tensor) _capturable_doc_default_to_fused_or_foreach_differentiable_doc_disable_dynamo_if_unsupported _foreach_doc!_get_capturable_supported_devices_get_scalar_dtype _maximize_doc _params_doc_use_grad_for_differentiable _view_as_real OptimizerParamsTRMSproprmspropceZdZ ddedeeefd ed ed ed ed ededeededeffd Z fdZ dZ e ddZ xZS)r{Gz?Gz?:0yE>rFNparamslralphaeps weight_decaymomentumcentered capturableforeachmaximizedifferentiablec t|tr'|dkrtdd|kstd|d|kstd|d|kstd|d|kstd|d|kstd|t |||||||| | | } t || dS) NrzTensor lr must be 1-elementgzInvalid learning rate: zInvalid epsilon value: zInvalid momentum value: zInvalid weight_decay value: zInvalid alpha value: ) rr!rrr"r r#r$r%r&) isinstancernumel ValueErrordictsuper__init__)selfrrrrr r!r"r#r$r%r&defaults __class__s DC:\Projects\Historical\venv\Lib\site-packages\torch/optim/rmsprop.pyr-zRMSprop.__init__s, b& ! ! zerosrr7 zeros_likepreserve_format) r.rCparams_with_gradgrads square_avgsmomentum_buffer_list grad_avgs state_steps has_complexrDr<s r1 _init_groupzRMSprop._init_group\s x$ 4$ 4Av~ 5+A.. .K  # #A & & &v P"#NOOO LL JqME5zzQ\*DEK*;*=*=ahOOOOR/@/B/BCCCf ',&6U%:'''l#$q((/4/?)>000E+,$(-(8)>)))E*%   u\2 3 3 3   uV} - - -Z 1$$$++E2C,DEEEZ  4  z!2333r2c|d}|5tj5|}dddn #1swxYwY|jD]}g}g}g}g}g}g} |||||||| } t |||||| |d|d|d|d|d|d|d|d |d |d | |S) zPerform a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrrrr r!r"r$r%r&r#) rrrr r!r"r$r%r&r#r[) _cuda_graph_capture_health_checkr? enable_gradr:r\r) r.closurelossrCrUrVrWrYrXrZr[s r1r4z RMSprop.stepsv --///  "$$ ! !wyy ! ! ! ! ! ! ! ! ! ! ! ! ! ! !&$ $ E-/ "$E(*K&(I13 (*K** $K  $;Gn%L">2z*z*i(z*$%56 .'#     ( s AA A) rrrrrFFNFFN)__name__ __module__ __qualname__rrrArboolrr-r9r\rr4 __classcell__)r0s@r1rrs&$( "&$'+'+'+ %- '+ '+  '+  '+'+'+'+$'+'+'+'+'+'+'+'+R*111f"444"!44444r2aj Implements RMSprop algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \alpha \text{ (alpha)}, \: \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, \: f(\theta) \text{ (objective)} \\ &\hspace{13mm} \lambda \text{ (weight decay)},\: \mu \text{ (momentum)}, \: centered, \: \epsilon \text{ (epsilon)} \\ &\textbf{initialize} : v_0 \leftarrow 0 \text{ (square average)}, \: \textbf{b}_0 \leftarrow 0 \text{ (buffer)}, \: g^{ave}_0 \leftarrow 0 \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}if \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}v_t \leftarrow \alpha v_{t-1} + (1 - \alpha) g^2_t \hspace{8mm} \\ &\hspace{5mm} \tilde{v_t} \leftarrow v_t \\ &\hspace{5mm}if \: centered \\ &\hspace{10mm} g^{ave}_t \leftarrow g^{ave}_{t-1} \alpha + (1-\alpha) g_t \\ &\hspace{10mm} \tilde{v_t} \leftarrow \tilde{v_t} - \big(g^{ave}_{t} \big)^2 \\ &\hspace{5mm}if \: \mu > 0 \\ &\hspace{10mm} \textbf{b}_t\leftarrow \mu \textbf{b}_{t-1} + g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \\ &\hspace{10mm} \theta_t \leftarrow \theta_{t-1} - \gamma \textbf{b}_t \\ &\hspace{5mm} else \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \hspace{3mm} \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `lecture notes `_ by G. Hinton. and centered version `Generating Sequences With Recurrent Neural Networks `_. The implementation here takes the square root of the gradient average before adding epsilon (note that TensorFlow interchanges these two operations). The effective learning rate is thus :math:`\gamma/(\sqrt{v} + \epsilon)` where :math:`\gamma` is the scheduled learning rate and :math:`v` is the weighted moving average of the squared gradient. z Args: a0 lr (float, Tensor, optional): learning rate (default: 1e-2) alpha (float, optional): smoothing constant (default: 0.99) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) momentum (float, optional): momentum factor (default: 0) centered (bool, optional) : if ``True``, compute the centered RMSProp, the gradient is normalized by an estimation of its variance z z rrVrWrYrXrZrrrr r!r"r%r&r#r[c t|D]P\}}||}tjsF|rDt }|jj|jjkr|jj|vsJd|d||}| s|n| }||}|dz }| dkr||| }tj|}|rz(_multi_tensor_rmsprop..js]  4 HMT[- - > !==      r2rirjg?cpu)r7rkrrlrn)r>r?rprqrallzipr"_group_tensors_by_device_and_dtypevaluesrrrrOr _foreach_negis_cpu _foreach_add_rB _foreach_add _foreach_mul__foreach_addcmul__foreach_lerp__foreach_addcmul_foreach_sqrt_ _foreach_sqrt_foreach_addcdiv_r( _foreach_mul _foreach_div_)"rrVrWrYrXrZrrrr r!r"r%r&r#r[grouped_tensorsgrouped_params_grouped_grads_grouped_square_avgs_grouped_grad_avgs_grouped_momentum_buffer_list_grouped_state_steps__grouped_params grouped_gradsgrouped_square_avgsgrouped_state_stepsstate_and_gradsgrouped_momentum_buffer_listgrouped_grad_avgsr momentum_lrrs" @r1_multi_tensor_rmsproprOs& 6{{aDDDDD  > & & ( (wZw'H'J'J$    v{33      w w wWs v v v  w w  B  Y0DkRO  " " $ $YWYW     ) d6lO<<T&\>:: "4<1EFF"4<1EFF  <,.ABO!||/3L"?00, &&'CDDD :$(f7I$J$J!&&'8999 . ;? ; ; ; ;  >!.}==M ~**,, 81DQ1G1N 8  #U\#e%D%D%DC        3Q 7 7 7 1   #M>VVVVV % 2!>!!!  /777   QY      * $T&\3E F F   !2M1u9 M M M(#%68IQSC   % % %  S ) ) ) )%&9::C  S ) ) ) a<<+/V ;,, (   .s?55() 1el##555555r2zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsNF) use_fusedz6torch.jit.script not supported with foreach optimizers) rrrr r!r"r%r#r&r[) r?rprqrrQr jit is_scriptingrr)rrVrWrYrXrZr$r%r&r#r[rrrr r!r"rfuncs r1rrs0: > & & ( ( 55-855522  ^   1 Ne   7U59))++USTTT&uy--//&$%D   !%!r2)NFFFF)__doc__typingrrrrr?r optimizerr r r r r rrrrrrrr__all__rrArfrrrrHr2r1rs10............ " i gggggigggV+X     Y<BD1 LD1 <D1fD1F| D1 v, D1 f D1 D1 D1 D1D1D1D1D1D1 !D1"#D1D1D1D1N}W L}W <}Wf}WF| }W v, }W f }W }W }W }W}W}W}W}W}W !}W"#}W}W}W}W@ 1GHHH# AA LA <AfAF| A v, A f Ad^AAAAA A !A" #A$%A&'A()AAAIHAAAr2