o ei@s^UddlZddlmmZddlmZddlmZgZe e e d<ej j GdddZdS)N)Tensor)2_scripted_functional_optimizer_deprecation_warning__all__c@seZdZ         ddeededeeefd ed ed ed ed ededefddZdededBfddZ deedBfddZ dS)_FunctionalAdamMbP?g?g+?:0yE>Fparamslrbetaseps weight_decayamsgradmaximizeforeachfused_allow_empty_param_listc Cs tddd|kstd|d|kstd|d|dkr'dks1ntd|dd|d kr=dksGntd |d d|ksRtd ||||d|d |d |_||_||_||_| |_tj t tj t t tj ffi|_ t|dkr| std d|i|_dS)N) stacklevelr zInvalid learning rate: zInvalid epsilon value: rg?z#Invalid beta parameter at index 0: z#Invalid beta parameter at index 1: zInvalid weight_decay value: )r r beta1beta2rz%optimizer got an empty parameter listr )r ValueErrordefaultsrrrrtorchjitannotatedictrstrstatelen param_group) selfr r r r rrrrrrr$q/var/www/addictedbytheproject.nl/epg/venv/lib/python3.10/site-packages/torch/distributed/optim/functional_adam.py__init__s2 $z_FunctionalAdam.__init__paramgradNc Csvg}g}g}g}g}g}t|} |dur||||||jvrVi|j|<|j|} td| d<tj|tjd| d<tj|tjd| d<|jrVtj|tjd| d<|j|} || d|| d|jrs|| d|| dt3t j |||||||j| |j |j d|j d |j d |j d |j d |j |jddd WddS1swYdS)zo Similar to step, but operates on a single parameter and optionally a gradient tensor. Nr step memory_formatexp_avg exp_avg_sqmax_exp_avg_sqrrr rr r has_complexrrrr rr rr grad_scale found_inf)r is_complexappendr tensor zeros_likepreserve_formatrno_gradFadamrrrr) r#r'r(params_with_gradgradsexp_avgs exp_avg_sqsmax_exp_avg_sqs state_stepsr0r r$r$r% step_paramFsh          "z_FunctionalAdam.step_param gradientsc Cs|jd}g}g}g}g}g}g}d} t|t|kr.tddt|ddt|t|jd|D]t\} } | dur| t| O} || || | |jvri|j| <|j| } td| d<tj | tj d | d <tj | tj d | d <|j rtj | tj d | d <|j| } || d || d |j r|| d || dq6t 3t j|||||||j | |j|jd |jd|jd|jd|jd|j|jdddWddS1swYdS)Nr FzEthe gradients passed in does not equal to the size of the parameters!zParams length: z. zGradients length: r r)r*r,r-r.rrr rr r/)r"r!rziprr3r4r r5r6r7rr8r9r:rrrr) r#rBr r;r<r=r>r?r@r0r'gradientr r$r$r%r)s           "z_FunctionalAdam.step) rrrr FFFFF) __name__ __module__ __qualname__listrfloattupleboolr&rAr)r$r$r$r%rsB    -s