RЦi<NSrSSKr"SS\RR5rg)zAdaHessian Optimizer Lifted from https://github.com/davda54/ada-hessian/blob/master/ada_hessian.py Originally licensed MIT, Copyright 2020, David Samuel Nc^\rSrSrSrS U4Sjjr\S5rSrSr \ R"5S5r \ R"5S Sj5r S rU=r$) Adahessian a Implements the AdaHessian algorithm from "ADAHESSIAN: An Adaptive Second OrderOptimizer for Machine Learning" Arguments: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 0.1) betas ((float, float), optional): coefficients used for computing running averages of gradient and the squared hessian trace (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0.0) hessian_power (float, optional): exponent of the hessian trace (default: 1.0) update_each (int, optional): compute the hessian trace approximation only after *this* number of steps (to save time) (default: 1) n_samples (int, optional): how many times to sample `z` for the approximation of the hessian trace (default: 1) c T>SU::d[SU35eSU::d[SU35eSUSs=::aS:dO [SUS35eSUSs=::aS:dO [SUS35eSUs=::aS::dO [S U35eXlXplXlS Ul[ R "5RUR5Ul[UUUUUS 9n [[U]3X5 UR5Hn SU lSURU S 'M g) NzInvalid learning rate: zInvalid epsilon value: r?z#Invalid beta parameter at index 0: z#Invalid beta parameter at index 1: zInvalid Hessian power value: i)lrbetaseps weight_decay hessian_power hessian step) ValueError n_samples update_eachavg_conv_kernelseedtorch Generator manual_seed generatordictsuperr__init__ get_paramshessstate) selfparamsr r r r rrrrdefaultsp __class__s T/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/optim/adahessian.pyrAdahessian.__init__s4by6rd;< <cz6se<= =eAh$$B58*MN NeAh$$B58*MN Nm*s*<]OLM M"&. *66tyyA%'   j$(:"AAF,-DJJqM. )#cg)NTrs r$is_second_orderAdahessian.is_second_orderFsr&c(SUR5$)z8 Gets all parameters in all param_groups with gradients c3b# UH%oSHo"R(dMUv M M' g7f)r N) requires_grad).0groupr"s r$ (Adahessian.get_params..Os"]0e/Q__/0s/ /) param_groupsr)s r$rAdahessian.get_paramsJs ^t00]]r&cUR5Hcn[UR[5(aM$URUSUR -S:XdMIURR 5 Me g)z+ Zeros out the accumulated hessian traces. rrN)r isinstancerfloatrrzero_)rr"s r$ zero_hessianAdahessian.zero_hessianQsT "Aaffe,,A~1NQUQaQa1aef1f #r&c /n[SUR55HQnURUSUR-S:XaUR U5 URUS==S- ss'MS [ U5S:XagUR RUSR:waA[R"USR5RUR5UlUVs/sHo"RPM nn[UR5HnUVs/sHCn[R"SSUR!5UR URS9S-S - PME nn[R"RX1US X@RS- :S 9n[%XeU5H*upxnU=R&Xx-UR- - slM, M gs snfs snf) zm Computes the Hutchinson approximation of the hessian trace and accumulates it for each trainable parameter. cURSL$N)grad)r"s r$(Adahessian.set_hessian..as !&&"4r&rrr N)rdeviceg@rT) grad_outputs only_inputs retain_graph)filterrrrappendlenrrBrrrrr>rangerrandintsizeautogradzipr) rr r"gradsizsh_zsh_zzs r$ set_hessianAdahessian.set_hessianZs 4doo6GHAzz!}^,t/?/??1D a JJqM. )Q . )I v;!   >> F1I$4$4 4"__VAY-=-=>JJ499UDN!'(A(t~~&Apvwpvkl%--1affh$..QRQYQYZ]``cffpvBw>>&&BDqSaSadeSeOe'gD 62 #'DNN223 ')xs /G"A G!cvSnUbU"5nUR5 UR5 URGHnUSGHnURb URcM UR (apUR 5S:Xa\[R"UR5RSS/SS9RUR5R5UlURSUS US -- 5 URUn[U5S:Xa7S US '[R"U5US '[R"U5US'US USpvUSupUS ==S- ss'URU5R!URSU- S9 URU 5R#URURSU - S9 SXS -- n SXS -- n USn X{- R%U S- 5R!US5n US U - nUR'XmU*S9 GM GM U$)z Performs a single optimization step. Arguments: closure (callable, optional) -- a closure that reevaluates the model and returns the loss (default: None) Nr rAT)dimkeepdimr r r rstepexp_avgexp_hessian_diag_sqr )alpha)valuerr )r9rTr3r>rrrYrabsmean expand_asclonemul_rrH zeros_likeadd_addcmul_pow_addcdiv_)rclosurelossr0r"rr\r]beta1beta2bias_correction1bias_correction2kdenom step_sizes r$r[Adahessian.stepvs)  9D  &&E8_66>QVV^''AEEGqL"YYqvv.33A3MWWXYX^X^_eegAFq5;~)>>>? 1 u:?$%E&M','7'7':E)$383C3CA3FE/0/4Y/?G\A],$W~ f "  U#((q5y(A#((/88qSXy8Y#$uf '=#= #$uf '=#= /*,?EEa!eLQQRWX]R^_"$K*::  7) rrr r Fr=)__name__ __module__ __qualname____firstlineno____doc__rpropertyr*rr9rno_gradrTr[__static_attributes__ __classcell__)r#s@r$rr s{&!*.X^ ]]_336 ]]_55r&r)rxroptim Optimizerrr(r&r$rs'  c&&cr&