RЦi7SrSSKrSSKJrJrJr SSKrSSKJr SSK Js J r SSK J r SSKJr SSKJr \R$"\5r"SS \R*5r"S S \5rg) zJToken-based distillation training task for models with distillation heads.N)DictOptionalUnion) create_model) unwrap_model) TrainingTaskcl^\rSrSrSrSS\\\R4S\ \ S\ S\ \S\ \ RS\ \ R4 U4S jjjrS \ RS \ R4S jrSS \ RS \ \ RS\ \ RS \ R4SjjrSrU=r$)TokenDistillationTeacheraWrapper for a teacher model used in token-based distillation. Creates and manages a pre-trained teacher model for token distillation, handling model creation and normalization differences between teacher and student. Can be created from: - A model name string (creates the model internally) - An existing nn.Module (wraps it with the necessary interface) Args: model_name_or_module: Either a model name string or an nn.Module num_classes: Number of output classes (required if model_name_or_module is a string) in_chans: Number of input channels (used if model_name_or_module is a string) pretrained_path: Optional path to pretrained weights (used if model_name_or_module is a string) device: Device to place the model on dtype: Model dtype (uses float32 if None) model_name_or_module num_classesin_chanspretrained_pathdevicedtypec >[TU]5 [U[5(aC[R SUS35 SS0nU(a [ UUS9US'[SUUUUUS.UD6nOC[U[R5(aUnO![S[U5R35eUR5 Xl[U5n [!U S 5(a9U R"R%S S 5n U R"R%S S 5n OS n S n [&R("XUS9R+SSSS5n [&R("XUS9R+SSSS5n UR-SU SS9 UR-SU SS9 g)Nz,Creating token distillation teacher model: '' pretrainedT)filerpretrained_cfg_overlay) model_namerrrrz8model_name_or_module must be a string or nn.Module, got pretrained_cfgmean)g ףp= ?gv/?gCl?std)gZd;O?gy&1?g?rrrmean_kdF persistentstd_kd)super__init__ isinstancestr_loggerinfodictrnnModule TypeErrortype__name__evalmodelrhasattrrgettorchtensorviewregister_buffer)selfr rrrrrpretrained_kwargsr0model_unwrappedrrrr! __class__s [/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/task/token_distillation.pyr$!TokenDistillationTeacher.__init__$s  *C 0 0 LLGH\G]]^_ `!-t 4 >B( +?!":; !/'!  $ E,bii 8 8(EJ4PdKeKnKnJop    'u- ?$4 5 5"1155f>STD!0044UCCAr1aP YEB Xv%@inputreturnc$URU5$)zForward pass through teacher model. Args: input: Input tensor (should already be normalized for teacher) Returns: Teacher logits r0)r7r>s r;forward TokenDistillationTeacher.forwardYszz%  r= student_mean student_stdcUbUcU$[R"X R5(a'[R"X0R5(aU$X-U-UR- UR- $)a7Normalize input to match teacher's expected normalization. Args: input: Input tensor (already normalized for student) student_mean: Student normalization mean buffer [1, 3, 1, 1] student_std: Student normalization std buffer [1, 3, 1, 1] Returns: Input tensor normalized for the teacher model )r3equalrr!)r7r>rDrEs r;normalize_input(TokenDistillationTeacher.normalize_inputds`  ;#6L ;;|\\ 2 2u{{;P[P[7\7\L#l2T\\AT[[PPr=rA)NNNN)NN)r. __module__ __qualname____firstlineno____doc__rr&r*r+rintr3rrr$TensorrBrH__static_attributes__ __classcell__r:s@r;r r s**.-1-1+/3A"'RYY"73A"#3A 3A &c] 3A U\\* 3AEKK(3A3Aj !U\\ !ell !4826 Q<<Q#5<<0Q"%,,/ Q  QQr=r c^\rSrSrSrSS\R S\\\R \ 4S\ \R S\ \S\S\ \ S \ \ S \ S \ \ RS \ \ RS \4U4SjjjrSS\ \SS4SjjrS\ R&S\ R&S\\\ R&44SjrSrU=r$)TokenDistillationTask{a,Token-based distillation task for models with distillation heads. For models like DeiT that have a dedicated distillation token/head that returns a tuple (main_logits, dist_logits) when distilled_training is enabled. The main head is trained against ground truth labels while the distillation head matches teacher outputs. Supports two distillation modes: - 'soft': KL divergence with temperature scaling (default) - 'hard': Cross-entropy with teacher's hard predictions (argmax) Loss weighting supports two modes: 1. Independent weights: loss = task_loss_weight * task_loss + distill_loss_weight * distill_loss 2. Complementary mode: loss = task_loss_weight * task_loss + (1 - task_loss_weight) * distill_loss (used when only task_loss_weight is specified) Args: student_model: Student model with set_distilled_training() method teacher_model: Teacher model - can be a model name string, nn.Module, or TokenDistillationTeacher criterion: Task loss function for main head (default: CrossEntropyLoss) teacher_pretrained_path: Path to teacher pretrained weights (used when teacher_model is a string) distill_type: 'soft' for KL-div or 'hard' for CE with teacher argmax distill_loss_weight: Weight for distillation loss task_loss_weight: Weight for task loss temperature: Softmax temperature for soft distillation (ignored for hard) device: Device for task tensors/buffers dtype: Dtype for task tensors/buffers verbose: Enable info logging Example: >>> # With model name string (num_classes/in_chans inferred from student) >>> task = TokenDistillationTask( ... student_model=model, teacher_model='deit_base_patch16_224', ... criterion=nn.CrossEntropyLoss(), ... distill_type='soft', temperature=3.0, task_loss_weight=0.5, ... device=torch.device('cuda'), ... ) >>> # With raw model >>> task = TokenDistillationTask( ... student_model=model, teacher_model=my_teacher_model, ... criterion=nn.CrossEntropyLoss(), ... distill_type='hard', task_loss_weight=0.5, ... ) student_model teacher_model criterionteacher_pretrained_path distill_typedistill_loss_weighttask_loss_weight temperaturerrverbosec 4>[TU]XU S9 [U5n [U S5(d#[ SU R R S35eU RS5 [U[5(aUn O[U[5(d[U[R5(a;U RnU Rn[UUUUURUR S9n O![#S[%U5R 35eXlXlUbUO[R*"5UlXPlXlUS;a[ S US 35e[2R4"U R6S URUR S 9R9S SS S 5n[2R4"U R6SURUR S 9R9S SS S 5nUR;SUSS9 UR;SUSS9 UbIX`lUbUOSUlUR@(a%[BRESUR>SU35 OUbGXplSU- UlUR@(a%[BRESUSUR<35 ONSUlSUlUR@(a/[BRESUR>SUR<35 UR@(a[BRESUSU35 gg)N)rrr_set_distilled_trainingzModel z does not have 'set_distilled_training' method. TokenDistillationTask requires a model with a distillation head (e.g., DeiT distilled variants).T)r rrrrrzWteacher_model must be a model name string, nn.Module, or TokenDistillationTeacher, got )softhardzUnsupported distill_type 'z'. Must be 'soft' or 'hard'.rrrrrrDFrrE?z9TokenDistillationTask: Independent weights - task_weight=z, distill_weight=z8TokenDistillationTask: Complementary mode - task_weight=z;TokenDistillationTask: Default equal weights - task_weight=z$TokenDistillationTask: distill_type=z, temperature=)#r#r$rr1 ValueErrorr:r.rar%r r&r*r+rrrrr,r-studentteacherCrossEntropyLossrYr[r^r3r4rr5r6r\r]r_r'r()r7rWrXrYrZr[r\r]r^rrr_student_unwrappedrgrrrDrEr:s r;r$TokenDistillationTask.__init__s WE)7(*BCC*44==>?ss  006 m%= > >#G  s + +z-/S/S+77K(11H.%2'! 7{{jj GM*3346  %  &/&;ATATAV(& / /9,Gcde e||  , ,V 4;;**  $q"a   ll  , ,U 3;;**  $q"a   ^\eL ]KEJ  *': $8H8T$4Z]D !|| ##'#8#8"99JK^J_a )$4 !'*-='=D $|| ##3"44EdF^F^E_a (+D $$'D !|| ##'#8#8"99J4KcKcJdf << LL6|nNS^R_`  r= device_idsr?c SSKJn URR5H nSUlM U"UR 4SU0UD6UlU$)aLPrepare task for distributed training. Wraps the student model in DistributedDataParallel (DDP) while leaving the frozen teacher model unwrapped. Args: device_ids: List of device IDs for DDP (e.g., [local_rank]) **ddp_kwargs: Additional arguments passed to DistributedDataParallel Returns: self (for method chaining) r)DistributedDataParallelFrk)torch.nn.parallelrmrg parameters requires_gradrf)r7rk ddp_kwargsDDPparams r;prepare_distributed)TokenDistillationTask.prepare_distributedsH" E\\,,.E"'E /4<<MJM*M  r=r>targetcURU5nUupEURXB5n[R"5 URR XR UR5nUR UR55nSSS5 URS:Xaj[R"XPR- SS9n [R"WUR- SS9n [R"XSSS9URS--n O%WRSS9n [R"X\5n UR U-UR"U --n U UUU S .$!,(df  N=f) arForward pass with token distillation. Args: input: Input tensor [B, C, H, W] target: Target labels [B] Returns: Dictionary containing: - 'loss': Combined training loss (task + distillation) - 'output': Main head logits (for metrics) - 'task_loss': Classification loss component - 'distill_loss': Distillation loss component Nrbr)dim batchmeanT) reduction log_target)lossoutput task_loss distill_loss)rfrYr3no_gradrgrHrDrEdetachr[F log_softmaxr^kl_divargmax cross_entropyr]r\)r7r>rvstudent_output main_logits dist_logitsrinput_kdteacher_logitsprob_sprob_tr teacher_hard total_losss r;rBTokenDistillationTask.forward+s<&e,#1 NN;7 ]]_||33E;L;LdN^N^_H!\\(//*;D4D4D#D"MF88FkVZ[_c_o_ost_tuL)00R08L??;EL**Y69Q9QT`9`` !"(   _s AE E')rYr\r[rfr]rgr^) NNrbNNrdNNT)N)r.rKrLrMrNr*r+rr&r rfloatr3rrboolr$listrtrPrrBrQrRrSs@r;rUrU{sB+b.259 &3704!$-1+/ g99g!bii1I!IJg  * g &.c] g  g"*%g'uoggU\\*gEKK(gggV*.  ! 2. <<. LL.  c5<< . . r=rU)rNloggingtypingrrrr3torch.nnr*torch.nn.functional functionalr timm.modelsr timm.utilsrtaskr getLoggerr.r'r+r rUr"r=r;rsYP(( $#   H %gQryygQT^ L^ r=