RЦi$SrSSKrSSKJrJr SSKrSSKJr SSKJ r S\RS\RS \RS \RS \ S \ S \ S\ S\RS\ S\ S\ S\ S\S\S\S\\ \ 4S\4$Sjr"SS\5rg)aPyTorch MARS Optimizer Code simplified from https://github.com/AGI-Arena/MARS Paper: MARS: Unleashing the Power of Variance Reduction for Training Large Models - https://arxiv.org/abs/2411.10438 @article{yuan2024mars, title={MARS: Unleashing the Power of Variance Reduction for Training Large Models}, author={Yuan, Huizhuo and Liu, Yifeng and Wu, Shuang and Zhou, Xun and Gu, Quanquan}, journal={arXiv preprint arXiv:2411.10438}, year={2024} } N)OptionalTuple) Optimizer)ParamsTpgradexp_avg exp_avg_sqlr weight_decaybeta1beta2 last_gradepsstepgamma mars_type is_grad_2d optimize_1d lr_1d_factorbetas_1dcautioncXU(dU (GaSU- nU S:XaUnOHX- RXU- -5RU5n[R"U5nUS:aUU- nURU5RUUS9 U(aQX!-S:R UR 5nUR UR5RSS95 UU-nU S:XaURU5RUUSU- S9 SXj-- nSXz-- nUR5[R"U5- RU 5nX-UU- R U5-nOU S :XaX-UR5-nOeURUU*S9 X#4$UunnURU5RUSU- S9 URU5RXSU- S9 SUU -- nSUU -- nUR5[R"U5- RU 5nU(aQX!-S:R UR 5nUR UR5RSS95 UU-nX-UU- R U5-nURUXO-*S9 X#4$) N?r)alphargMbP?)minadamw)valuelion) mul_add_torchnormtodtypediv_meanclamp_addcmul_sqrtmathsign)rr r r r r rrrrrrrrrrrrone_minus_beta1c_tc_t_normmaskbias_correction1bias_correction2denomupdatebeta1_1dbeta2_1ds N/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/optim/mars.py_mars_single_tensor_stepr9s*ju* 19C#))%?3J*KLQQRVWCzz#H"}Hn U  O < NQ&**4::6D IIdiik((T(2 3nG   OOE " + +CBJ + G"U]2 "U]2 __&3C)DDJJ3OE%3C)C(I(I%(PPF & % 6F 5 vbS!  &( X##DX #>!**4R(]*KT!11T!11"TYY/?%@@FFsK NQ&**4::6D IIdiik((T(2 3nG!W/?%?$E$Ee$LL vr012  c^\rSrSrSrSS\S\S\\\4S\S\S\S \S \ S \S \ \\\4S \ 4U4Sjjjr U4Sjr \ R"5SSj5rSrU=r$)Mars[zMARS Optimizer Paper: MARS: Unleashing the Power of Variance Reduction for Training Large Models https://arxiv.org/abs/2411.10438 paramsr betasrr rrrrrrc >SU::d[SRU55eSU::d[SRU55eSUSs=::aS:dO [SRUS55eSUSs=::aS:dO [SRUS55eUS ;dS 5e[UUUUUUUU U =(d UU S 9 n [[U]X5 g) NzInvalid learning rate: {}zInvalid epsilon value: {}rrz%Invalid beta parameter at index 0: {}rz%Invalid beta parameter at index 1: {})rr zMARS type not supported) r r?rr rrrrrr) ValueErrorformatdictsuperr<__init__)selfr>r r?rr rrrrrrdefaults __class__s r8rF Mars.__init__bsby8??CD Dcz8??DE EeAh$$DKKERSHUV VeAh$$DKKERSHUV V--H/HH-%#%&   dD"64r:cv>[[U] U5 URHnUR SS5 M g)NrF)rEr< __setstate__ param_groups setdefault)rGstategrouprIs r8rLMars.__setstate__s2 dD&u-&&E   Y .'r:cSnUb%[R"5 U"5nSSS5 URGH+nUSGHnURcMURnUR(a [ S5eUR Un[U5S::aPSUS'[R"U5US'[R"U5US'[R"U5US 'US==S- ss'USnUSnUS n USn US n US n US upURS :n[UUUU U U U UU USUUSUSUUSUSUSUSS9 XVS'GM GM. U$!,(df  GNM=f)zPerforms a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nr>zJAdam does not support sparse gradients, please consider SparseAdam insteadrrrr rr r r r?rrrrrrr)rrrrrr) r# enable_gradrMr is_sparse RuntimeErrorrOlen zeros_likendimr9)rGclosurelossrPrr rOrr r rr wdrrrs r8r Mars.steps  ""$y%&&E8_66>vv>>&'stt 1 u:?$%E&M','7'7':E)$).)9)9!)*$W~ !YY!^ )%L'N#K0) %m 4!&~!6":.!),%*&*k"e%'j q%$s E)) E8) g~jth?)g?gGz?g:0yE>rAg?rFrNF)N)__name__ __module__ __qualname____firstlineno____doc__rfloatrstrboolrrFrLr#no_gradr__static_attributes__ __classcell__)rIs@r8r<r<[s)4"$ $ %"%6:!$5$5$5& $5  $5  $5$5$5$5 $5uUE\23$5$5$5L/  ]]_AAr:r<)rcr,typingrrr#torch.optim.optimizerr_typesrTensorrdintrerfr9r<r^r:r8ros  " +? <<?ll??LL ?  ?  ???<<??????? !?"u %#?$%?Dt9tr: