RЦi&pSrSSKrSSKrSSKJrJrJrJrJrJ r SSK r SSK J r Sr SSKJr SS KJr SS KJr \R&"\5rS rS rS //SQ/SQ/SQS.r\ \\\\\4\\\\\44rS\S\\S4S\4Sjr\S\ R<S4S\ R>S\S\\\\\4S\S\S\ R@S\!S\ R>4Sjjr"SBS\ RFS \S\4S!jjr$SBS\ RFS \S\\\!44S"jjr%SCS#\ R>S$\S%\S&\!S\ \!\\!\444 S'jjr&SDS(\ R>S)\S\\ R>\ RF44S*jjr'S+\\ R>S,\\ R>S-\\ R>S.\S/\S0\S1\!S2\S3\S\S\S \\S4\S5\!S\!SS4 S6jr(S+\\ R>S,\\ R>S-\\ R>S7\\ R>S8\\ R>S.\S/\S0\S1\!S9\S2\S3\S\S\S \\S4\S5\!S\!SS4&S:jr)S+\\ R>S,\\ R>S-\\ R>S.\S/\S0\S1\!S2\S3\S\S\S \\S4\S5\!S\!SS4 S;jr*S+\\ R>S,\\ R>S-\\ R>S7\\ R>S8\\ R>S.\S/\S0\S1\!S9\S2\S3\S\S\S \\S4\S5\!S\!SS4&S<jr+"S=S>\ RXRZ5r.S?\ \\\\\\4S@\\\\\4S\\\\\44SAjr/g!\a Sr GN f=f)EaMuon Optimizer Improved Muon optimizer implementation with flexible handling of high-dimensional tensors. Combines PyTorch-style structure with options for: - Batched spatial processing for convolutions in addition to flatten - Optional spatial normalization - Selectable coefficient presets - Automatic fallback to AdamW for 1D / scalar parameters (biases, norms, etc.) and optional fallback via param groups - AdaMuon (https://arxiv.org/abs/2507.11005) - mUP eps damping factor (https://arxiv.org/abs/2512.05620v1) TODO look into mUP LR scaling and independent weight-decay scale Based on implementation by Keller Jordan, see - https://github.com/KellerJordan/Muon/blob/master/muon.py - https://github.com/KellerJordan/modded-nanogpt/blob/master/train_gpt.py - https://github.com/KellerJordan/modded-nanogpt/blob/master/train_gpt_medium.py - https://github.com/NoahAmsel/PolarExpress/blob/main/polar_express.py Hacked together by Ross Wightman N)ListMappingOptionalSequenceTupleUnion)DTensorTF)ParamsT)adamw)nadamwgHz>)guV @ggn@@))gͪV@gg"~j@)gv@ggj+6g F%u@)ga4 @gH}]g\Cm@)g2%@g?$ g/L F?)g6>W[@gQkgH}8?))g8y @gWr"b(7g=90@)g3kT@g)6$g}U"?)g")i@gV}gT"?)g߸%~b @g_O"Dveg)E?)gT`@gU4N/g ?)gͶ?g!̲gG߆?)g_?g ]sig??N!?)g9+_?gWg-|?))gh('P @g־{6g ]!/@)g=.+@gixgےl%?)g""@gP+ .gPYJخ1?)g.69I @gG4gH5?)gSO|g@gVg@ę?)gΐX?gxDgPh?)gkei?gB?Ggd݈ؼ?)g1?gjgZ?)originalquintic polar_expresspolar_express_saferepsshape.returnc(USUSp2XU- S--$)uScale epsilon for Newton-Schulz based on matrix dimensions (μP-style). For μP compatibility, epsilon should scale as eps * sqrt(din/dout) to maintain consistent damping behavior across different model widths. Reference: https://arxiv.org/abs/2512.05620 Args: eps: Base epsilon value shape: Shape of the matrix (out, in) or (batch, out, in) Returns: Scaled epsilon value ?)rrdoutdins N/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/optim/muon.pyscale_eps_for_nsrXs%(rE"I# *$ $$?Gsteps coefficients safety_factordtype scale_epsc <URS;dSURS35e[U5nUS:a[US5S:XdeX::aUSUO X"S/X- --nU(a[X0R5nUR US S 9n U R S 5U R S5:n U (a U R n U(a?U RU RS S S S9RU5RU55 O True (or (True, "ok") if return_reason=True) (96, 3, 4, 4) -> True - will be flattened to (96, 48) (4, 2048) -> False - extreme aspect ratio (64,) -> False - insufficient dims (1, 196, 768) -> False - leading unit dims NOTE: these rules were created to balance complexity with covering common timm model cases Please let me know if there are non-optimal cases that you run into. r(c34# UHoS:dM Sv M g7f)r Nr).0dim_sizes r (_is_suitable_for_muon..sAq8qLQQqs  )Finsufficient_dimsFrr )Fleading_unit_dimsr)Nzmin_dim_too_small:zextreme_aspect_ratio:z.1f)TokT)rr3sumr-rZ) rcrdrerfsout_chin_ch_with_spatiald check_dimsmin_sizemax_size aspect_ratios r_is_suitable_for_muonrys :  A zzA~AqAAAE/<+G%G  tqyAaDAI/<+G%G zzQ112A  # 1  :H .xj99 9:H&L& 1,s1CDD D(<2d2rtensormodec^URnURS:XaX4$URS:a[SUR35eURSSup4US:XaURUS5U4$US:Xa(URX4S5nUR SSS5nXR4$[S U35e) aReshape high-dimensional tensor for Muon processing. Args: tensor: Input tensor of shape (out, in, *spatial) mode: How to handle spatial dimensions - "flatten": Flatten spatial into output dimension (out, in*H*W) - "batched": Batch over spatial positions (spatial_prod, out, in) for per-position orthogonalization Returns: Reshaped tensor and original shape for restoration r(z,Tensor must have at least 2 dimensions, got Nflattenrbatchedrr zUnknown mode: )rr3 ValueErrorreshapepermute)rzr{original_shaperrin_chreshapeds rreshape_for_muonrGs\\N {{a%% {{QG }UVVLL!$MF y~~fb)>99  >>&4##Aq!,''>$011rparamsgrads momentum_bufslr weight_decaymomentumnesterovns_stepsns_coefficients conv_modenormalize_spatialc 2[UUUUUUUUUU U U U U US9 g)z8Functional API that performs Muon algorithm computation. rrrrrrrr$rSrrr&N)_single_tensor_muon)rrrrrrrrrrr$rSrrr&s rmuonrjs9&  !' #!+r exp_avg_sqs state_stepsbeta2c <[UUUUU4UUUUU U U U U UUUUS. 6 g)a(Functional API that performs AdaMuon algorithm computation. AdaMuon extends Muon with element-wise second moment estimation applied to orthogonalized update directions, providing Adam-like adaptive scaling while preserving Muon's geometric benefits. Reference: https://arxiv.org/abs/2507.11005 rrrrrrrrr$rSrrr&N)_single_tensor_adamuon)rrrrrrrrrrrrrr$rSrrr&s radamuonrsI:   !' #!+%rc [U[5n[U5GH'unnXnX/nURSX4-- 5 UR USU- 5 U(aUR UU5OUR 5nUR S:a[UU S9unnOUnURn[UUUU U US9nU (a[URU 5nOSnU S:Xa?UR S:a/U (aUURSS--nURSS S5nURU5nURUU*U-S 9 GM* g ) zSingle tensor Muon update.r r r)r{rr$r&r~rrar(r1N)resolve_ns_coefficients _COEFFICIENTS enumeratemul_lerp_cloner3rrrQr^rrr;)rrrrrrrrrrr$rSrrr&ircgrad momentum_bufupdateupdate_reshapedr update_orthoscales rrrsU&.o}MOf%5x$'   1r(() 4h/7?L(3\EWEWEY ;;! .>vI.V +O^$O#\\N2   '    !3!3\BEE  !l&7&71&< ++A.$66'//1a8L$++N;   <sU{ 3[&rc ^[U [5n [U5GH unnUUnUUnUUnUUnUS- nUR5nUR SXV-- 5 UR USU- 5 U(aUR UU5OUR 5nURS:a[UUS9unnOUnURn[UU U U U US9nUS:Xa#URS:aURSSS5nURU5nUR U 5RUUSU - S 9 U(a[URU5unnOS unnU(a UR5R!U 5nO*SU U-- n UU - R5R!U 5nUU- n!U(a$U!R#5R!U 5n"U!U"- n!US:Xa1[%U5S:a"U(aSn#USS Hn$U#U$-n#M UU#S --nUR!U!U*U-S 9 GM g )uSingle tensor AdaMuon update. AdaMuon applies second-moment estimation to the orthogonalized directions, then rescales using RMS-alignment to maintain stable step sizes. Algorithm: 1. Update momentum buffer: M = β₁·M + (1-β₁)·G 2. Orthogonalize: O = Newton-Schulz(M) or Newton-Schulz(nesterov_update) 3. Update second moment: v = β₂·v + (1-β₂)·O² 4. Bias correct: v̂ = v/(1-β₂^t) 5. Adaptive scaling: Ô = O / (√v̂ + ε) 6. RMS-aligned rescaling and apply update r r r)rrr~r(r)value)r FNrar)rrritemrrrr3rrrQrraddcmul_rbsqrtr;r9r4)%rrrrrrrrrrrrrr$rSrrr&rrcrr exp_avg_sqstep_tsteprrrrr use_rms_normdenombias_correction2update_adaptive update_norm spatial_prodrts% rrrsXD.o}MOf%5Qx$Q'  ^ Q ! {{}  1r(() 4h/7?L(3\EWEWEY ;;! .>vI.V +O^$O#\\N2   '    !l&7&71&<'//1a8L#++N;  '' l#PU+'V "6|7I7I<"X E<", E< OO%**3/E #Ud]2 "2288:??DE'. )..055c:K- ;O  !c.&9Q&>  '+A A%L,--  ?2#+ 6[&rc#^\rSrSrSrSSSS\S\SS S S S S SSS4S\S\S\S\S\ S\ S\ S\S\S\ \ S\ S\ S\ \S\\\4S\ S\ S\ 4"U4S jjjrU4S!jr\R$"5S$S"j5rS#rU=r$)%Muoni|aMuon - MomentUm Orthogonalized by Newton-schulz Combines Muon for 2D+ parameters (weight matrices) with AdamW for 1D parameters (biases, norms) and parameter groups with 'use_fallback=True' set (or 'use_muon=False' for compatibility). Supports two algorithms: - "muon": Standard Muon algorithm with momentum + orthogonalization - "adamuon": AdaMuon algorithm that adds element-wise second moment estimation to orthogonalized directions for Adam-like adaptive scaling g{Gz?rffffff?Frr rVr}TN)g?rrrrrrrrrrr$rSrradamw_lrbetasalgor&verbosec>SU::d[SU35eSU::d[SU35eSUs=::aS:dO [SU35eSU::d[SU35eU S;a[SU 35eUS ;a[S US 35e[S0S U_S U_SU_SU_SU_SU_SU_SU _SU _SU _SU _SU bU OU_SU_SU_SU_SU_6n[TU] UU5 g)ufCreate Muon optimizer. Args: params: Iterable of parameters or dicts defining parameter groups lr: Learning rate (default: 0.02 for Muon parameters) weight_decay: Weight decay coefficient momentum: Momentum factor for Muon nesterov: Whether to use Nesterov momentum ns_steps: Number of Newton-Schulz iterations ns_coefficients: Coefficients for NS iteration eps: Numerical stability epsilon safety_factor: Multiplicative safety factor for NS norm adjust_lr_fn: LR adjustment function - "original", "match_rms_adamw", or "rms_to_rms". For adamuon mode, can set to None to disable (RMS rescaling handles scaling). conv_mode: How to handle convolutions - "flatten" or "batched" normalize_spatial: Whether to normalize by sqrt(spatial_size) in batched mode adamw_lr: Learning rate for AdamW (1D params), defaults to lr if not specified betas: Beta coefficients - (beta1, beta2) where beta1 is used for AdamW fallback and beta2 is used for both AdamW fallback and AdaMuon second moment algo: Algorithm - "muon" for standard Muon, "adamuon" for AdaMuon with adaptive second moment estimation (https://arxiv.org/abs/2507.11005) scale_eps: If True, scale epsilon by sqrt(din/dout) in Newton-Schulz for μP compatibility (https://arxiv.org/abs/2512.05620) verbose: Log parameter routing decisions (Muon vs AdamW) Example: ```python # Simple usage - automatically uses Muon for 2D+ params, AdamW for 1D optimizer = Muon(model.parameters(), lr=0.02) # Use AdaMuon algorithm for adaptive scaling optimizer = Muon(model.parameters(), lr=6e-4, algo="adamuon") # Manual control over parameter groups optimizer = Muon([ {'params': weight_matrices, 'lr': 0.02}, {'params': biases, 'use_fallback': True, 'lr': 3e-4}, # use AdamW if use_fallback=True ]) ``` r/zInvalid learning rate: zInvalid weight_decay value: r zInvalid momentum value: zInvalid epsilon value: )r}r~zInvalid conv_mode: )rrzInvalid algo: z. Must be 'muon' or 'adamuon'rrrrrrrr$rSrrrNrrr&rr)rdictsuper__init__)selfrrrrrrrrr$rSrrrrrr&rdefaults __class__s rr Muon.__init__snvby6rd;< <l";L>JK Kh$$7zBC Ccz6se<= = 2 229+>? ? * *~dV3PQR R  %      ,   ( &   0 "*!5X2     ! $ *rc>[TU]U5 URH'nURSS5 URSS5 M) g)Nrrr&F)r __setstate__ param_groups setdefault)rstategrouprs rrMuon.__setstate__s@ U#&&E   VV ,   [% 0'rc> SnUb%[R"5 U"5nSSS5 URRSS5nSnSnU(a0OSnURGHnURSS5n/n /n /n /n /n /n/n/n/n/nUSGHnUR cMUR R (a [S5eURUnS U;aSnURS S5(aSUS 'U(aS nO.+s,EWSVVWsr momentum_buffer) memory_formatrrr/rexp_avgrrrrrrrrr$rSrrr&rrr) foreachbeta1rrrrcautionmaximize capturablemax_lr) ramsgradrrrrrrrrrzMuon parameter routing: z Muon, z AdamW=z Breakdown: , rorAdamWz z -> : z z ... and z more)r? enable_gradrgetrr is_sparse RuntimeErrorrryjoinrappend zeros_likepreserve_formatrzrrr r r4_loggerinfosorteditems isEnabledForloggingINFO)"rclosurelossr muon_count adamw_countrouting_reasonsrr muon_params muon_gradsmuon_momentum_bufsmuon_exp_avg_sqsmuon_state_steps adamw_params adamw_gradsadamw_exp_avgsadamw_exp_avg_sqsadamw_state_stepsprreasonsuitable shape_strr_rr reason_groupsreasonsreason_summaryshapesoptimizer_namers" rr Muon.steps  ""$y%--##Iu5  '"T&&E99VV,DKJ!# ! ! LKN "  " 8_66>66##&'OPP 1 U*!Fyy77,1j)"%8F#u,,1*,=j)"%4F#/DQVZ/[,Hf' )!''*&&qvv.1$KU*(- R(8f +0+;+;AUMbMb+ci(.3.>.>qPUPePe.fl+"))% *:;%,,U<-@A%,,U6];Q%V9$$W~HAu#"*((  !;%*>%:!&z!2!&z!2#!&z!2(-.?(@!%L&+O&<%*>%:"' "4*/0C*D"' "4%*#"* ;%*>%:!&z!2!&z!2!&z!2(-.?(@!%L&+O&<%*>%:"' "4*/0C*D"' "4&|$W~ u$$#&)) $## ,%*>%:!%L %!&#(#$$#&)) $ %## ,%*>%:!%L %!&#(##{'d s?3a7 LL3J"?%%#f+&?@#@ LL=>)B(CD E##GLL11&,]-@-@-B&CNFF/5~V7NLL4xtN3C1!EF!' veW%56"-6{R' ~c&kB6F5Gu%MN 'D q%$s X  Xrr)__name__ __module__ __qualname____firstlineno____doc__DEFAULT_NS_STEPSMUON_EPSr floatboolintNSCoeffrrrrrr?no_gradr__static_attributes__ __classcell__)rs@rrr|s2 "#"",'0!#&*;&&*(,)4#!%Z+Z+Z+ Z+  Z+  Z+Z+%Z+Z+!Z+#3-Z+Z+ $Z+uoZ+&Z+ !Z+"#Z+$%Z+Z+x1  ]]_\\rrrpresetsc^^ Sm SmS[[S[[[[44UU 4Sjjn[U[5(aX;a9SR [ UR555n[SUSU35eXnT "U5(a[U5S :Xa[S US 35eUVs/sH oR"U5PM sn$T "U5(d [S 5e[U5S :Xa#[U4SjU55(a U"U5/$/n[U5H;upuT "U5(d[SUSU<35eURU"U55 M= U(d [S5eU$s snf)Ncf[U[5=(a [U[[45(+$r)r>rrbytesrs r)resolve_ns_coefficients..s!z!X.Rz!c5\7R3RRrcn[U[R5=(a [U[5(+$r)r>numbersRealrr%s rr&r's! 1gll3OJq$T"U5(a)[U5S:wd[U4SjU55(d[SU<35eUupn[U5[U5[U54$)Nr)c34># UH nT"U5v M g7frrrivis_reals rrk.as_coeff..s2Iq!71::qz3Coefficient must be length-3 of real numbers, got: )r4allrr)rrJrKrLr/is_seqs ras_coeff)resolve_ns_coefficients..as_coeffs]ayyCFaKs2Iq2I/I/IRSTRWXY YaQxq58++rrzUnknown coefficients preset 'z'. Valid options: rzPreset 'z' is empty or invalidz]Coefficients must be a preset name (str), a 3-sequence (a,b,c), or a sequence of 3-sequences.r)c34># UH nT"U5v M g7frrr-s rrk*resolve_ns_coefficients..s95awqzz5r1zItem z is not a sequence: z Coefficient list cannot be empty)rrrr>rrrkeysrr4 TypeErrorr2rr) rr!r4validseqrr2rr/r3s @@rrrst SFOG,HUO,eUE.A(B,, %  IIfW\\^45E"$ ;<< J)0s=E;)rV)g`@F)r})0rrr)typingrrrrrrr?torch.distributed.tensorr r= ImportError_typesr r r getLoggerrrrrrrrrrrbfloat16Tensorr%rrQSizer^rbryrrrrroptim OptimizerrrrrrrGs_,BB 0K   H %  "  5& R U5%./eE5%.OZZOO 5$;O:"&# E3||E3E3 E3 E3  4tSy! !" E3T 2  2 2 5<< #$ 2F#U\\"#ELL!#ELL)#  #  ####!###sm## # !#" ##L0U\\"0ELL!0ELL)0%,,' 0 %,,' 0 000000!000 sm!0"#0$ %0&'0( )0fB4U\\"B4ELL!B4ELL)B4  B4  B4B4B4B4!B4B4B4smB4B4 B4 !B4" #B4Jq7U\\"q7ELL!q7ELL)q7%,,' q7 %,,' q7 q7q7q7q7q7q7!q7q7q7 sm!q7"#q7$ %q7&'q7( )q7hK5;; K\ )S(5/8HUO+DDE)hx778) %ue# $%)WKsL))L54L5