RЦi.2SrSSKrSSKJr "SS\5rg)amPyTorch LARS / LARC Optimizer An implementation of LARS (SGD) + LARC in PyTorch Based on: * PyTorch SGD: https://github.com/pytorch/pytorch/blob/1.7/torch/optim/sgd.py#L100 * NVIDIA APEX LARC: https://github.com/NVIDIA/apex/blob/master/apex/parallel/LARC.py Additional cleanup and modifications to properly support PyTorch XLA. Copyright 2021 Ross Wightman N) Optimizerc^\rSrSrSrSU4SjjrU4Sjr\R"5SSj5r Sr U=r $) LarsaLARS for PyTorch Paper: `Large batch training of Convolutional Networks` - https://arxiv.org/pdf/1708.03888.pdf Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups. lr (float, optional): learning rate (default: 1.0). momentum (float, optional): momentum factor (default: 0) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) dampening (float, optional): dampening for momentum (default: 0) nesterov (bool, optional): enables Nesterov momentum (default: False) trust_coeff (float): trust coefficient for computing adaptive lr / trust_ratio (default: 0.001) eps (float): eps for division denominator (default: 1e-8) trust_clip (bool): enable LARC trust ratio clipping (default: False) always_adapt (bool): always apply LARS LR adapt, otherwise only when group weight_decay != 0 (default: False) c >US:a[SU35eUS:a[SU35eUS:a[SU35eU(aUS::dUS:wa [S5e[UUUUUUUU U S9 n [T U] X5 g)NgzInvalid learning rate: zInvalid momentum value: zInvalid weight_decay value: rz8Nesterov momentum requires a momentum and zero dampening) lrmomentum dampening weight_decaynesterov trust_coeffeps trust_clip always_adapt) ValueErrordictsuper__init__) selfparamsrr r r r r rrrdefaults __class__s N/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/optim/lars.pyr Lars.__init__#s 86rd;< < c>7zBC C # ;L>JK K Q)q.WX X%#!%   *cn>[TU]U5 URHnURSS5 M g)Nr F)r __setstate__ param_groups setdefault)rstategrouprs rrLars.__setstate__Fs0 U#&&E   Z /'rc SnUb%[R"5 U"5nSSS5 URGHnUSnUSnUSnUSnUSnUSn USGHqn U RcMU Rn US :wd US (aU R S 5n U R S 5n X-XU--U -- n[R "U S :[R "U S :US 5S 5nUS (a[R "XS- S S9nU RXS9 U RU5 US :waURU nSU;a*[R"U 5R5=nUS'O'USnURU5RU S U- S9 U(aU RUUS9n OUn U RXS*S9 GMt GM U$!,(df  GN=f)zPerforms a single optimization step. Args: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nr r r r r rrrrg@?rr)max)alphamomentum_buffer) torch enable_gradrgradnormwhereclampadd_mul_r clonedetachadd)rclosurelossr!r r r r r rpr*w_normg_norm trust_ratio param_statebufs rstep Lars.stepKs  ""$y%&&E 0LZ(Hk*IZ(H .K,C8_66>vv 1$n(=VVC[F!YYs^F"-"6&LCX:X[^:^"_K#(++  FQJ SA#K \*&+kk+d 2KQT&U IIaI4IIk*q="&**Q-K( ;?D{{4?P?W?W?YYk*;<)*;<*//BN/K#xx8x<"t$K<0I%'\ c%$s G G*) r$rrrFgMbP?g:0yE>FF)N) __name__ __module__ __qualname____firstlineno____doc__rrr(no_gradr;__static_attributes__ __classcell__)rs@rrrsJ(  !+F0  ]]_99rr)rBr(torch.optim.optimizerrrr=rrrGs   +t9tr