RЦiؽSrSSKrSSKJr SSKJr SSKJrJrJ r J r J r J r J r JrJrJr SSKJr SSKrSSKrSSKJr SSKrSSKJrJr SS KJrJrJr SS KJr SS K J!r! SS K"J#r# SS K$J%r% SSK&J'r' SSK(J)r) SSK*J+r+ SSK,J-r- SSK.J/r/ SSK0J1r1 SSK2J3r3 SSK4J5r5 SSK6J7r7 SSK8J9r9 SSK:J;r; SSKJ?r? SSK@JArA SSKBJCrC SSKDJErE SSKFJGrG SSKHJIrI SS KJJKrK SS!KLJMrM \R"\O5rPS"\QS#\4S$jrR\"S%S&9"S'S(55rS"S)S*5rTS+\TS#S4S,jrUS+\TS#S4S-jrVS+\TS#S4S.jrWS+\TS#S4S/jrXS+\TS#S4S0jrYS+\TS#S4S1jrZS+\TS#S4S2jr[S+\TS#S4S3jr\\T"5r]SOS4jr^\^"5 SPS5\\Q\ \Q4S6\ \ \QS7\_S#\ \\Q\\Q\Q444S8jjr`S9\QS#\S4S:jraSQS9\QS;\_S#\\\44S<jjrbSRS=\\R\4S>\QS?\ \dS@\dSA\dSB\ \_SC\_SD\ \QSE\_SF\ \dSG\dSH\ \dSI\ \\R/\4SJ\S#\RR4SKjjrgSLrhSQSM\\R\4SC\_S#\RR4SNjjrig)SzrOptimizer Factory w/ custom Weight Decay & Layer Decay support Hacked together by / Copyright 2021 Ross Wightman N) dataclass)partial) AnyCallable CollectionDictListOptionalSetTupleTypeUnionfnmatch)param_groups_layer_decayparam_groups_weight_decay)ParamsT OptimTypeOptimizerCallable) AdaBelief) Adafactor)AdafactorBigVision) Adahessian)AdamP) AdamWLegacy)Adan)Adopt)Kron)Lamb)LaProp)Lars)Lion) Lookahead)MADGRAD)Mars)Muon) NAdamLegacy)NAdamW) NvNovoGrad) RAdamLegacy) RMSpropTF)SGDP)SGDW class_stringreturncURSS5up[R"U5n[X25$![[ 4an[ SUSU35eSnAff=f)z)Dynamically import a class from a string..rzCould not import z: N)rsplit importlib import_modulegetattr ImportErrorAttributeError)r/ module_name class_namemodulees X/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/optim/_optim_factory.py _import_classr>.shC"."5"5c1"= ((5v**  (C-l^2aSABBCs47AAAT)frozenc\rSrSr%Sr\\S'\\\4\S'Sr \\S'Sr \ \S'S r \ \S 'S r \ \S 'S r\\S 'S r\ \S'Sr\\\\4\S'Srg) OptimInfo9a$Immutable configuration for an optimizer. Attributes: name: Unique identifier for the optimizer opt_class: The optimizer class description: Brief description of the optimizer's characteristics and behavior has_eps: Whether the optimizer accepts epsilon parameter has_momentum: Whether the optimizer accepts momentum parameter has_betas: Whether the optimizer accepts a tuple of beta parameters num_betas: number of betas in tuple (valid IFF has_betas = True) defaults: Optional default parameters for the optimizer name opt_class descriptionThas_epsF has_momentum has_betas num_betas second_orderNdefaults)__name__ __module__ __qualname____firstlineno____doc__str__annotations__rrrFrGboolrHrIrKintrLrMr rr__static_attributes__rNr=rArA9sp  IS)^$$KGTL$ItIsL$)-HhtCH~&-rYrAc\rSrSrSrS%SjrS\SS4SjrS\S \SS4S jr S \SS4S jr S&S \ \\ \4S\ \ \S\S\ \ \\\\444SjjrS \S\4SjrS'S\ \\4S\S\ \\44SjjrS(S\ \R,\4S\S\ \S\S\S\ \S\S\\S\S\ \S\ \S \ \S!\ \\R,/\4S"\S\R:R<4S#jjrS$r g))OptimizerRegistryRzRegistry managing optimizer configurations and instantiation. This class provides a central registry for optimizer configurations and handles their instantiation with appropriate parameter groups and settings. r0Nc"0UlS1Ulg)Nlion) _optimizers_foreach_defaults)selfs r=__init__OptimizerRegistry.__init__Ys13,28rYinfocURR5nX R;a[R SUS35 XRU'g)zxRegister an optimizer configuration. Args: info: The OptimInfo configuration containing name, type and description Optimizer z already registered, overwritingN)rClowerr__loggerwarning)rardrCs r=registerOptimizerRegistry.register]sD yy  ## # OOj.NO P!%rYaliastargetcUR5nX R;a[SU35eURUURUR5'g)zRegister an alias for an existing optimizer. Args: alias: The alias name target: The target optimizer name Raises: KeyError: If target optimizer doesn't exist z/Cannot create alias for non-existent optimizer N)rgr_KeyError)rarlrms r=register_alias OptimizerRegistry.register_aliashsO )) )LVHUV V*.*:*:6*B'rYrCcVURRUR55 g)z4Register an optimizer as defaulting to foreach=True.N)r`addrgrarCs r=register_foreach_default*OptimizerRegistry.register_foreach_defaultws ""4::<0rYfilterexclude_filterswith_descriptionc^ [URR55nU(aT[U[5(aU/nOUn[ 5nUHm UR U 4SjU55 M [U5nU(a-UH'nUVs/sHn[X5(aMUPM nnM) U(a+UV s/sHoURU R4PM sn $U$s snfs sn f)aGList available optimizer names, optionally filtered. Args: filter: Wildcard style filter string (e.g., 'adam*') exclude_filters: Optional list of wildcard patterns to exclude with_description: If True, return tuples of (name, description) Returns: List of either optimizer names or (name, description) tuples c3N># UHn[UT5(dMUv M g7fNr).0nfs r= 4OptimizerRegistry.list_optimizers..s%HA'!Q-aas% %) sortedr_keys isinstancerTsetupdaterrF) rarwrxrynamesfiltersfiltered_namesexclude_filterr~rCrs @r=list_optimizers!OptimizerRegistry.list_optimizers{s t'',,./ &#&&!(  UN%%%H%HH>*E "1$)LEq1KEL#2 KPQ544++D1==>5Q Q MRsC((C(?$C-c|UR5nXR;a[SUS35eURU$)Get the OptimInfo for an optimizer. Args: name: Name of the optimizer Returns: OptimInfo configuration Raises: ValueError: If optimizer is not found rfz not found in registry)rgr_ ValueErrorrts r=get_optimizer_info$OptimizerRegistry.get_optimizer_infosAzz| '' 'z$/EFG G%%rY name_or_info bind_defaultsc`[U[5(aURU5nO[U[5(deUn[UR[5(aURR S5(aA[ RR5(dS5e[UR5nOURR S5(aA[ RR5(dS5e[UR5nO"[UR5nO URnU(a'UR(a[U40URD6nU$![an[S5UeSnAff=f![an[S5UeSnAff=f)aGet the optimizer class with any default arguments applied. This allows direct instantiation of optimizers with their default configs without going through the full factory. Args: name_or_info: Name of the optimizer bind_defaults: Bind default arguments to optimizer class via `partial` before returning Returns: Optimizer class or partial with defaults applied Raises: ValueError: If optimizer not found zapex.z!CUDA required for APEX optimizersz,APEX optimizers require apex to be installedNz bitsandbytes.z)CUDA required for bitsandbytes optimizerszr7rMr)rarropt_inforDr<s r=get_optimizer_class%OptimizerRegistry.get_optimizer_classsp( lC ( (..|->?I##]%&TU[\\] #m%&dekllms0+E4 F4 F> F  F F- F((F-model_or_paramsoptlr weight_decaymomentumforeachweight_decay_exclude_1d fallback_listfallback_no_weight_decay layer_decaylayer_decay_min_scalelayer_decay_no_opt_scaleparam_group_fnkwargsc [U[R5(ai[USS5"5nU (a U "U5nOHU b[ UUU UUU UU U S9 nSnO1U(aU(a[ UUUUU S9nSnOUR 5nOUnUR5RS5nUSn[U5S :aUS S :HOS nURU5nS U0UEnUbUUS'UR(a6URR5HunnURUU5 M UR(aURSU5 UR(dUR!SS5 UR"(dUR!SS5 UbURSU5 UR%US S9nU"U40UD6nU(a ['U5nU$)aCreate an optimizer instance. Args: model_or_params: Model or parameters to optimize opt: Name of optimizer to create lr: Learning rate weight_decay: Weight decay factor momentum: Momentum factor for applicable optimizers foreach: Enable/disable foreach operation weight_decay_exclude_1d: Whether to skip weight decay for 1d params (biases and norm affine) fallback_list: Collection of parameter name patterns to use fallback optimizer for hybrid optimizers fallback_no_weight_decay: If True, params in no_weight_decay list will use fallback optimizer (e.g., AdamW for Muon) layer_decay: Layer-wise learning rate decay layer_scale_min_scale: Minimum layer scale factor clamp value layer_scale_no_opt_scale: Layer scale below which optimization is disabled param_group_fn: Optional custom parameter grouping function **kwargs: Additional optimizer-specific arguments Returns: Configured optimizer instance Raises: ValueError: If optimizer not found or configuration invalid no_weight_decayc[5$r|)rrNrYr=4OptimizerRegistry.create_optimizer..sRURWrYN)rrno_weight_decay_listrrr min_scale no_opt_scale)rrrr_rr lookaheadFrrrepsbetasrr)rnnModuler6rr parametersrgsplitlenrrMitems setdefaultrHrGpoprIrr$)rarrrrrrrrrrrrrrrparams opt_splitopt_name use_lookaheadropt_argskvrD optimizers r=create_optimizer"OptimizerRegistry.create_optimizersV oryy 1 1%o7H-XZO'8(1#!- +)8"/-E,C3!9  " "92#!-)8"/-E  " (335%FIIK%%c* R=7:9~7I ! 3u **84%3L#KF#K >HTN    ))//11##Aq)2     H 5 LL %!! LL$ '      7 3,,XU,K f11  !),IrY)r`r_r0NrENFT) Nr?NTrNFNNNN)!rOrPrQrRrSrbrArjrTrprurr r rVr rrrrrrrrfloatrrrroptim OptimizerrrXrNrYr=r[r[Rs 4 &Y &4 & CC C C C1S1T1 -/37%* ##tCy.)#&d3i0## # eCsCx() * #J&s&y&(#'1Y/1 1 y++ , 1n#'"$!&*,0-/-2+/598Cautious SGD with decoupled weight decay and Nesterov momentum)rrrcadampzQAdd the spherical cautious optimizer and the standard cautious optimizer to AdamPr)rrr)rArrrrrr r!r#r&r)r,r.rrj)rcautious_optimizersrs r=_register_cautious_optimizersr7s,&  (7&  !(&   (&  G!%%8   Y!%$7   ;#'D9  '&   C!%$?   )&   '!4(   '&   )&   ;"t4   X"&48   k"&DTJ  Uqd###rYc[S[SSSS0S9[S[SSSS0S9[S [RRS S 9[S [RR S SS0S9[S[ SSS0SSS9[S[ SSS0SSS9[S[SSSS9[S[SSSS9[S[SSSS S0S!9[S"[S#SS$9[S%[S&SSS'9[S([S)SS*9[S+[S,SS S0S-9[S.[S/SS$9[S0[S1SSSS29[S3[S4SSSS5S0S69[S7[S8SSSS9S70S69[S:[S;SSSS7SS<.S69[S=[S>SS$9[S?[RRS@SSASB0S-9[SC[ SDSSASB0S-9/nUHnUR#U5 M UR%S%5 gE)Fz!Register miscellaneous optimizers adabeliefz7Adapts learning rate based on gradient prediction errorTrectifyFr radabeliefz,Rectified AdaBelief with variance adaptationadadeltazQtorch.optim.Adadelta, Adapts learning rates based on running windows of gradientsradagradzMtorch.optim.Adagrad, Adapts learning rates using cumulative squared gradientsrg:0yE>radanz$Adaptive Nesterov Momentum Algorithmr(r)r*adanwz6Adaptive Nesterov Momentum with decoupled weight decay adahessianz"An Adaptive Second Order Optimizer)rCrDrFrIrLkronz5PSGD optimizer with Kronecker-factored preconditioner)rCrDrFrGrHkronwzPPSGD optimizer with Kronecker-factored preconditioner and decoupled weight decayr)rCrDrFrHrGrMlapropz*Separating Momentum and Adaptivity in Adamrr^z8Evolved Sign Momentum optimizer for improved convergence)rCrDrFrGrImadgradz'Momentum-based Adaptive gradient methodrmadgradwz#MADGRAD with decoupled weight decayrmarszDUnleashing the Power of Variance Reduction for Training Large ModelsmuonzJMomentUm Orthogonalized by Newton-schulz with AdamW fallback for 1D params)rCrDrFrHrGrInmuonzXMomentUm Orthogonalized by Newton-schulz with Nesterov and NAdamW fallback for 1D paramsr)rCrDrFrHrGrIrMadamuonzQAdaMuon: Muon with adaptive second moment estimation on orthogonalized directionsalgonadamuonz@AdaMuon with Nesterov momentum and NAdamW fallback for 1D params)rJrnovogradz3Normalized Adam with L2 norm gradient normalizationrmspropz1torch.optim.RMSprop, Root Mean Square Propagationrr rmsproptfzETensorFlow-style RMSprop implementation, Root Mean Square PropagationN)rArrrAdadeltaAdagradrrrr!r#r%r&r'r*RMSpropr,rjru)rother_optimizersrs r=_register_other_optimizersrSs Q'   F&   kk**k kk))gT]  >'   P&    <   O   j'.   D  R   A  ='.   ^  d   r $'  ki(  Z'T:   M  kk))Ks^   _s^  aWp #  %%f-rYc [SSSSSSS0S9[SS S SS S0S 9[S S SSS S0S 9[SSSSS9[SSSSSS0S 9/nUHnURU5 M g)z&Register APEX optimizers (lazy import)fusedsgdzapex.optimizers.FusedSGDz8NVIDIA APEX fused SGD implementation for faster trainingFTrr fusedadamzapex.optimizers.FusedAdamz%NVIDIA APEX fused Adam implementation adam_w_moder fusedadamwz&NVIDIA APEX fused AdamW implementation fusedlambzapex.optimizers.FusedLAMBz%NVIDIA APEX fused LAMB implementationr fusednovogradzapex.optimizers.FusedNovoGradz)NVIDIA APEX fused NovoGrad implementationr)gffffff?g\(\?NrArj)rapex_optimizersrs r=_register_apex_optimizersr]s 0R $'   1?#U+   1@#T*   1?   5C|,  ;$OJ#rYc0[SSSSSSS0S9[SS S SSSS0S9[S S S SS9[SS SSS9[SSSSS9[SSSSS9[SSSSSS9[SSSSSS9[SSSSS S!9[S"S#S$SS S!9/ nUHnURU5 M g%)&z.Register bitsandbytes optimizers (lazy import)bnbsgdzbitsandbytes.optim.SGDzbitsandbytes SGDFTrr bnbsgd8bitzbitsandbytes.optim.SGD8bitz0bitsandbytes 8-bit SGD with dynamic quantizationbnbadamzbitsandbytes.optim.Adamzbitsandbytes Adamr bnbadam8bitz1bitsandbytes 8-bit Adam with dynamic quantizationbnbadamwzbitsandbytes.optim.AdamWzbitsandbytes AdamW bnbadamw8bitz2bitsandbytes 8-bit AdamW with dynamic quantizationbnblionzbitsandbytes.optim.Lionzbitsandbytes Lion)rFrGrI bnblion8bitzbitsandbytes.optim.Lion8bitz1bitsandbytes 8-bit Lion with dynamic quantization bnbademamixzbitsandbytes.optim.AdEMAMixzbitsandbytes AdEMAMixr))rFrIrKbnbademamix8bitzbitsandbytes.optim.AdEMAMix8bitz5bitsandbytes 8-bit AdEMAMix with dynamic quantizationNr[)rbnb_optimizersrs r=_register_bnb_optimizersrjs$ .* $'   2J $'   /+  /K  0,  0L    %+     )K     )/     -O  }ENL#rYcL[[5 [[5 [[5 [ [5 [ [5 [ [5 [[5 [[5 [RSS5 [RSS5 g)z7Register all default optimizers to the global registry.rr nesterovwrN) rdefault_registryrr rSr]rjr7r!rprNrYr=_register_default_optimizersrn2sn+,,-()/0./-.!"23()9:##J6##K8rYrwrxryc.[RXU5$)aList available optimizer names, optionally filtered. List all registered optimizers, with optional filtering using wildcard patterns. Optimizers can be filtered using include and exclude patterns, and can optionally return descriptions with each optimizer name. Args: filter: Wildcard style filter string or list of filter strings (e.g., 'adam*' for all Adam variants, or ['adam*', '*8bit'] for Adam variants and 8-bit optimizers). Empty string means no filtering. exclude_filters: Optional list of wildcard patterns to exclude. For example, ['*8bit', 'fused*'] would exclude 8-bit and fused implementations. with_description: If True, returns tuples of (name, description) instead of just names. Descriptions provide brief explanations of optimizer characteristics. Returns: If with_description is False: List of optimizer names as strings (e.g., ['adam', 'adamw', ...]) If with_description is True: List of tuples of (name, description) (e.g., [('adam', 'Adaptive Moment...'), ...]) Examples: >>> list_optimizers() ['adam', 'adamw', 'sgd', ...] >>> list_optimizers(['la*', 'nla*']) # List lamb & lars ['lamb', 'lambc', 'larc', 'lars', 'nlarc', 'nlars'] >>> list_optimizers('*adam*', exclude_filters=['bnb*', 'fused*']) # Exclude bnb & apex adam optimizers ['adam', 'adamax', 'adamp', 'adamw', 'nadam', 'nadamw', 'radam'] >>> list_optimizers(with_description=True) # Get descriptions [('adabelief', 'Adapts learning rate based on gradient prediction error'), ('adadelta', 'torch.optim Adadelta, Adapts learning rates based on running windows of gradients'), ('adafactor', 'Memory-efficient implementation of Adam with factored gradients'), ...] )rmr)rwrxrys r=rrHsT  + +FEU VVrYrCc,[RU5$)r)rmr)rCs r=rrus  . .t 44rYrc([RXS9$)a)Get optimizer class by name with option to bind default arguments. Retrieves the optimizer class or a partial function with default arguments bound. This allows direct instantiation of optimizers with their default configurations without going through the full factory. Args: name: Name of the optimizer to retrieve (e.g., 'adam', 'sgd') bind_defaults: If True, returns a partial function with default arguments from OptimInfo bound. If False, returns the raw optimizer class. Returns: If bind_defaults is False: The optimizer class (e.g., torch.optim.Adam) If bind_defaults is True: A partial function with default arguments bound Raises: ValueError: If optimizer name is not found in registry Examples: >>> # Get SGD with nesterov momentum default >>> SGD = get_optimizer_class('sgd') # nesterov=True bound >>> opt = SGD(model.parameters(), lr=0.1, momentum=0.9) >>> # Get raw optimizer class >>> SGD = get_optimizer_class('sgd') >>> opt = SGD(model.parameters(), lr=1e-3, momentum=0.9) r)rmr)rCrs r=rrsD  / / / RRrYrrrrrrfilter_bias_and_bnrrrrrrrc  J[R"U4UUUUUUUUU U U U S. U D6$)aH Create an optimizer instance via timm registry. Creates and configures an optimizer with appropriate parameter groups and settings. Supports automatic parameter group creation for weight decay and layer-wise learning rates, as well as custom parameter grouping. Args: model_or_params: A PyTorch model or an iterable of parameters/parameter groups. If a model is provided, parameters will be automatically extracted and grouped based on the other arguments. opt: Name of the optimizer to create (e.g., 'adam', 'adamw', 'sgd'). Use list_optimizers() to see available options. lr: Learning rate. If None, will use the optimizer's default. weight_decay: Weight decay factor. Will be used to create param groups if model_or_params is a model. momentum: Momentum factor for optimizers that support it. Only used if the chosen optimizer accepts a momentum parameter. foreach: Enable/disable foreach (multi-tensor) implementation if available. If None, will use optimizer-specific defaults. filter_bias_and_bn: If True, bias, norm layer parameters (all 1d params) will not have weight decay applied. Only used when model_or_params is a model and weight_decay > 0. fallback_list: Collection of parameter name patterns to use fallback optimizer for hybrid optimizers (e.g., AdamW for Muon). Supports wildcard matching. fallback_no_weight_decay: If True, params in model's no_weight_decay() list will use fallback optimizer for hybrid optimizers (e.g., AdamW for Muon). layer_decay: Optional layer-wise learning rate decay factor. If provided, learning rates will be scaled by layer_decay^(max_depth - layer_depth). Only used when model_or_params is a model. param_group_fn: Optional function to create custom parameter groups. If provided, other parameter grouping options will be ignored. **kwargs: Additional optimizer-specific arguments (e.g., betas for Adam). Returns: Configured optimizer instance. Examples: >>> # Basic usage with a model >>> optimizer = create_optimizer_v2(model, 'adamw', lr=1e-3) >>> # SGD with momentum and weight decay >>> optimizer = create_optimizer_v2( ... model, 'sgd', lr=0.1, momentum=0.9, weight_decay=1e-4 ... ) >>> # Adam with layer-wise learning rate decay >>> optimizer = create_optimizer_v2( ... model, 'adam', lr=1e-3, layer_decay=0.7 ... ) >>> # Custom parameter groups >>> def group_fn(model): ... return [ ... {'params': model.backbone.parameters(), 'lr': 1e-4}, ... {'params': model.head.parameters(), 'lr': 1e-3} ... ] >>> optimizer = create_optimizer_v2( ... model, 'sgd', param_group_fn=group_fn ... ) Note: Parameter group handling precedence: 1. If param_group_fn is provided, it will be used exclusively 2. If layer_decay is provided, layer-wise groups will be created 3. If weight_decay > 0 and filter_bias_and_bn is True, weight decay groups will be created 4. Otherwise, all parameters will be in a single group ) rrrrrrrrrrrr)rmr)rrrrrrrrrrrrrrrs r=create_optimizer_v2rtsNf  , ,   ! 2#!93!9%   rYcURURURURS.n[ USS5=nbX!S'[ USS5=nbX1S'[ USS5=nbXAS'[ USS5=nbXQS'[ US S5=nbXaS '[ US S5=nbUR U5 [ US S5=nbXS 'U$) zGConvert argparse-style `cfg` object to kwargs for an optimizer factory.)rrrropt_epsNr opt_betasrrrrr opt_foreachr)rrrrr6r) cfgrrrrld_min ld_no_optrrs r=optimizer_kwargsr|s)) F sIt,,9u k400=wsM488 E +}#6==J*0&'S"rrrrArrr rCr!rr"r^r#rr$rDr%rFr&rGr'rr(rr) nvnovogradr*rr+ rmsprop_tfr,rr-rr. getLoggerrOrhrTr>rAr[rrr r!r7rSr]rjrmrnrVrrrrrrrrtr|rrNrYr=rs![[[ N99 ," "!   H %CCC $...0KK\$%6$4$N^&7^D^B;"3;;|{3D{{|t,=t$tl\.):\.t\.~((9(d(VI'8ITIX%&9$ )+/3!&*Wc49n%*W!$s),*W*W %U38_$ %& *WZ 5S 5Y 5"#"S"S"S 9' '("SN" "&#')+).'+'*48CGbryy'12b b UOb b  b $ b!b"#b#'be_b %b#+5/b!299+w*>!?@bb [[bJ:$( RYY'( !  [[ rY