RЦi_ SrSSKrSSKJrJrJrJr SSKrSSKJ r SSK J s J r SSK Jr SSKJr SSKJr \R&"\5r"SS \ R,5rS \\\ R,\4S \ R,S \\S \\R2S\\R4S\4 Sjr"SS\5r"SS\ R,5r"SS\5rg)z5Knowledge distillation training tasks and components.N)DictOptionalTupleUnion) create_model) unwrap_model) TrainingTaskcv^\rSrSrSrSS\\\R4S\ \ S\ S\ \S\ \ RS\ \ R4 U4S jjjrSS \ RS \S \ R4S jjrSS \ RS\ \ RS\ \ RS \ R4SjjrSrU=r$)DistillationTeachera1Wrapper for a teacher model used in knowledge distillation. Creates and manages a pre-trained teacher model for knowledge distillation, handling model creation and normalization differences between teacher and student. Can be created from: - A model name string (creates the model internally with pretrained weights) - An existing nn.Module (wraps it with the necessary interface) Args: model_name_or_module: Either a model name string or an nn.Module num_classes: Number of output classes (required if model_name_or_module is a string) in_chans: Number of input channels (used if model_name_or_module is a string) pretrained_path: Optional path to pretrained weights (used if model_name_or_module is a string) device: Device to place the model on dtype: Model dtype (uses float32 if None) model_name_or_module num_classesin_chanspretrained_pathdevicedtypec >[TU]5 [U[5(aC[R SUS35 SS0nU(a [ UUS9US'[SUUUUUS.UD6nOC[U[R5(aUnO![S[U5R35eUR5 Xl[U5n [!U S 5(a9U R"R%S S 5n U R"R%S S 5n OS n S n [&R("XUS9R+SSSS5n [&R("XUS9R+SSSS5n UR-SU SS9 UR-SU SS9 g)NzCreating KD teacher model: '' pretrainedT)filerpretrained_cfg_overlay) model_namerrrrz8model_name_or_module must be a string or nn.Module, got pretrained_cfgmean)g ףp= ?gv/?gCl?std)gZd;O?gy&1?g?rrr mean_kdF persistentstd_kd)super__init__ isinstancestr_loggerinfodictrnnModule TypeErrortype__name__evalmodelrhasattrrgettorchtensorviewregister_buffer)selfrrrrrrpretrained_kwargsr1model_unwrappedrrrr" __class__s U/var/www/html/ai-image-ml/venv/lib/python3.13/site-packages/timm/task/distillation.pyr%DistillationTeacher.__init__%s  *C 0 0 LL78L7MQO P!-t 4 >B( +?!":; !/'!  $ E,bii 8 8(EJ4PdKeKnKnJop    'u- ?$4 5 5"1155f>STD!0044UCCAr1aP YEB Xv%@inputreturn_featuresreturncbU(a[URS5(a[URS5(d-[SURRRS35eURR U5nURR USS9$URU5$)aForward pass through teacher model. Args: input: Input tensor (should already be normalized for teacher) return_features: Whether to return pooled pre-logits features instead of logits Returns: Logits or pooled pre-logits features depending on return_features flag forward_features forward_headzModel zi does not support feature extraction. Ensure the model has 'forward_features' and 'forward_head' methods.T pre_logits)r2r1 ValueErrorr;r/rCrD)r8r?r@ feature_maps r<forwardDistillationTeacher.forwardZs 4::'9::'$**VdBeBe TZZ11::; student_mean student_stdcUbUcU$[R"X R5(a'[R"X0R5(aU$X-U-UR- UR- $)a7Normalize input to match teacher's expected normalization. Args: input: Input tensor (already normalized for student) student_mean: Student normalization mean buffer [1, 3, 1, 1] student_std: Student normalization std buffer [1, 3, 1, 1] Returns: Input tensor normalized for the teacher model )r4equalrr")r8r?rKrLs r<normalize_input#DistillationTeacher.normalize_inputss`  ;#6L ;;|\\ 2 2u{{;P[P[7\7\L#l2T\\AT[[PPr>r1)NNNN)F)NN)r/ __module__ __qualname____firstlineno____doc__rr'r+r,rintr4rrr%TensorboolrIrO__static_attributes__ __classcell__r;s@r<r r s**.-1-1+/3A"'RYY"73A"#3A 3A &c] 3A U\\* 3AEKK(3A3Ap%*%<<%"%  %84826 Q<<Q#5<<0Q"%,,/ Q  QQr>r teacher student_modelrrrrAc [U[5(aU$[U5nURnURn[UUUUUUS9$)anResolve teacher input to a DistillationTeacher instance. Args: teacher: Model name string, nn.Module, or DistillationTeacher student_model: Student model to infer num_classes/in_chans from pretrained_path: Optional path to teacher pretrained weights device: Device for teacher dtype: Dtype for teacher Returns: DistillationTeacher instance )rrrrrr)r&r rrr)r]r^rrrstudent_unwrappedrrs r<_resolve_teacherrasY&'.//%]3#//K ))H $'  r>c^\rSrSrSrSS\R S\\\R \ 4S\ \R S\ \S\S\ \ S \ \ S \ S \ \ RS \ \ RS \4U4SjjjrSS\ \SS4SjjrS\ R&S\ R&S\\\ R&44SjrSrU=r$)LogitDistillationTaskaLogit-based knowledge distillation task. Performs distillation by matching student and teacher output logits using KL divergence with temperature scaling. Loss weighting supports two modes: 1. Independent weights: loss = task_loss_weight * task_loss + distill_loss_weight * distill_loss 2. Complementary mode: loss = task_loss_weight * task_loss + (1 - task_loss_weight) * distill_loss (used when only task_loss_weight is specified) Args: student_model: Student model to train teacher_model: Teacher model - can be a model name string, nn.Module, or DistillationTeacher criterion: Task loss function (default: CrossEntropyLoss) teacher_pretrained_path: Path to teacher pretrained weights (used when teacher_model is a string) loss_type: Type of distillation loss (currently only 'kl' supported) distill_loss_weight: Weight for distillation loss task_loss_weight: Weight for task loss temperature: Softmax temperature for distillation (typical values: 1-4) device: Device for task tensors/buffers dtype: Dtype for task tensors/buffers verbose: Enable info logging Example: >>> # With model name string (num_classes/in_chans inferred from student) >>> task = LogitDistillationTask( ... student_model=model, teacher_model='resnet50', ... criterion=nn.CrossEntropyLoss(), ... task_loss_weight=0.3, temperature=4.0, ... device=torch.device('cuda'), ... ) >>> # With raw model >>> task = LogitDistillationTask( ... student_model=model, teacher_model=my_teacher_model, ... criterion=nn.CrossEntropyLoss(), ... task_loss_weight=0.3, temperature=4.0, ... ) r^ teacher_model criterionteacher_pretrained_path loss_typedistill_loss_weighttask_loss_weight temperaturerrverbosec >[TU]XU S9 [UUUURUR5n XlXlUbUO[R"5Ul XPl Xl US:wa[SUS35e[U5n [R"U R SURURS9R#SSSS5n[R"U R S URURS9R#SSSS5nUR%S US S 9 UR%S US S 9 UbIX`lUbUOSUlUR*(a%[,R/SUR(SU35 OUbGXplSU- UlUR*(a%[,R/SUSUR&35 ONSUlSUlUR*(a/[,R/SUR(SUR&35 UR*(a[,R/SUSU35 gg)NrrrlklzUnsupported loss_type 'z$'. Currently only 'kl' is supported.rrr rrrKFr rL?z9LogitDistillationTask: Independent weights - task_weight=, distill_weight=z8LogitDistillationTask: Complementary mode - task_weight=z;LogitDistillationTask: Default equal weights - task_weight=z!LogitDistillationTask: loss_type=z, temperature=)r$r%rarrstudentr]r+CrossEntropyLossrfrhrkrGrr4r5rr6r7rirjrlr(r))r8r^rerfrgrhrirjrkrrrlr]r`rKrLr;s r<r%LogitDistillationTask.__init__sU WE#   # KK JJ  %  &/&;ATATAV"&  6ykAefg g)7||  , ,V 4;;**  $q"a   ll  , ,U 3;;**  $q"a   ^\eL ]KEJ  *': $8H8T$4Z]D !|| ##'#8#8"99JK^J_a )$4 !'*-='=D $|| ##3"44EdF^F^E_a (+D $$'D !|| ##'#8#8"99J4KcKcJdf << LL3I;n[MZ  r> device_idsrAc SSKJn URR5H nSUlM U"UR 4SU0UD6UlU$)aLPrepare task for distributed training. Wraps the student model in DistributedDataParallel (DDP) while leaving the frozen teacher model unwrapped. Args: device_ids: List of device IDs for DDP (e.g., [local_rank]) **ddp_kwargs: Additional arguments passed to DistributedDataParallel Returns: self (for method chaining) rDistributedDataParallelFru)torch.nn.parallelrxr] parameters requires_gradrrr8ru ddp_kwargsDDPparams r<prepare_distributed)LogitDistillationTask.prepare_distributed*sH" E\\,,.E"'E /4<<MJM*M  r>r?targetc^URU5nURX25n[R"5 URR XR UR5nUR UR5SS9nSSS5 [R"X0R- SS9n[R"WUR- SS9n[R"XxSSS9URS --n URU-URU --n U UUU S .$!,(df  N=f) aqForward pass with logit distillation. Args: input: Input tensor [B, C, H, W] target: Target labels [B] Returns: Dictionary containing: - 'loss': Combined training loss (task + distillation) - 'output': Student logits (for metrics) - 'task_loss': Classification loss component - 'kd_loss': Logit distillation loss component Fr@Nr)dim batchmeanT) reduction log_targetlossoutput task_losskd_loss)rrrfr4no_gradr]rOrKrLdetachF log_softmaxrkkl_divrjri) r8r?rstudent_logitsrinput_kdteacher_logitsprob_sprob_tr total_losss r<rILogitDistillationTask.forwardCs$e,NN>: ]]_||33E;L;LdN^N^_H!\\(//*;U\SN~0@0@@bI~0@0@@bI((6[TRVZVfVfjkVkl**Y69Q9QT[9[[ $"   _s AD D,)rfrirhrrrjr]rk) NNroNNrpNNTN)r/rSrTrUrVr+r,rr'r rfloatr4rrrYr%listrrXrrIrZr[r\s@r<rcrcsB%V.259!3704!$-1+/ Q99Q!bii1D!DEQ  * Q &.c] Q  Q"*%Q'uoQQU\\*QEKK(QQQj*.  ! 2$ <<$ LL$  c5<< $ $ r>rcc^\rSrSrSrS S\R S\\R 4U4SjjjrS\ RS\ \ R\ R44Sjr S r U=r$) "FeatureDistillationTrainableModuleijaTrainable module for feature distillation. Wraps student model and projection layer into a single module where all trainable forward operations happen inside forward(). This ensures proper DDP wrapping when the module is used with DistributedDataParallel. r^ projectionc:>[TU]5 XlX lg)zCreate trainable module wrapper for feature distillation. Args: student_model: Student model to train projection: Optional projection layer (Linear layer or None) N)r$r%rrr)r8r^rr;s r<r%+FeatureDistillationTrainableModule.__init__rs $ $r>r?rAcURRU5nURRU5nURRUSS9nURbURU5nX44$)zForward pass through student and projection. Args: input: Input tensor [B, C, H, W] Returns: Tuple of (student_logits, student_features) where features are optionally projected to match teacher dimension. TrE)rrrCrDr)r8r?rHrstudent_featuress r<rI*FeatureDistillationTrainableModule.forwardsill33E: 22;?<<44[T4R ?? &#/?@ //r>)rrrr)r/rSrTrUrVr+r,rr%r4rXrrIrZr[r\s@r<rrjsd/3 %99 %!+ % %0U\\0eELL%,,4N.O00r>rc^\rSrSrSrSS\R S\\\R \ 4S\ \R S\ \S\ \ S\ \ S \ \ S \ \ S \ \ RS \ \ RS \4U4Sjjjr\S\R S\ 4Sj5rSS\ \SS4SjjrS\ R,S\ R,S\\\ R,44SjrSrU=r$)FeatureDistillationTaskiaFeature-based knowledge distillation task. Performs distillation by matching student and teacher intermediate features (pooled pre-logits) using MSE loss. Automatically creates a projection layer if student and teacher feature dimensions differ. Loss weighting supports two modes: 1. Independent weights: loss = task_loss_weight * task_loss + distill_loss_weight * distill_loss 2. Complementary mode: loss = task_loss_weight * task_loss + (1 - task_loss_weight) * distill_loss (used when only task_loss_weight is specified) Args: student_model: Student model to train teacher_model: Teacher model - can be a model name string, nn.Module, or DistillationTeacher criterion: Task loss function (default: CrossEntropyLoss) teacher_pretrained_path: Path to teacher pretrained weights (used when teacher_model is a string) distill_loss_weight: Weight for distillation loss task_loss_weight: Weight for task loss student_feature_dim: Student pre-logits dimension (auto-detected if None) teacher_feature_dim: Teacher pre-logits dimension (auto-detected if None) device: Device for task tensors/buffers dtype: Dtype for task tensors/buffers verbose: Enable info logging Example: >>> # With model name string (num_classes/in_chans inferred from student) >>> task = FeatureDistillationTask( ... student_model=model, teacher_model='resnet50', ... criterion=nn.CrossEntropyLoss(), ... distill_loss_weight=5.0, task_loss_weight=1.0, ... device=torch.device('cuda'), ... ) r^rerfrgrirjstudent_feature_dimteacher_feature_dimrrrlc >[TU]XU S9 [UUUURUR5n XlUbUO[ R"5UlUbIXPl UbUOSUl UR(a%[RSURSU35 OUbGX`l SU- Ul UR(a%[RSUSUR35 ONSUl SUl UR(a/[RSURSUR35 UcURU5nUcURU R5nSn Xx:waWUR(a[RSUSU35 [ R "XxURURS 9n O&UR(a[RS 5 [#X5Ul['U5n[(R*"UR,S URURS 9R/S S S S 5n[(R*"UR,SURURS 9R/S S S S 5nUR1SUSS9 UR1SUSS9 UR(a[RSUSU35 gg)Nrnrpz;FeatureDistillationTask: Independent weights - task_weight=rqz:FeatureDistillationTask: Complementary mode - task_weight=z=FeatureDistillationTask: Default equal weights - task_weight=zCreating projection layer: z -> rz.Feature dimensions match, no projection neededrr rrrKFr rLz%FeatureDistillationTask: student_dim=z, teacher_dim=)r$r%rarrr]r+rsrfrirjrlr(r)_detect_feature_dimr1Linearrtrainable_modulerr4r5rr6r7)r8r^rerfrgrirjrrrrrlr]rr`rKrLr;s r<r% FeatureDistillationTask.__init__s WE#   # KK JJ   &/&;ATATAV  *': $8H8T$4Z]D !|| ##'#8#8"99JK^J_a )$4 !'*-='=D $|| ##3"44EdF^F^E_a (+D $$'D !|| ##'#8#8"99J4KcKcJdf  &"&":":="I   &"&":":7=="I   5|| 12E1FdK^J_`#6TXT_T_gkgqgqrJ|| MN B= ])7||  , ,V 4;;**  $q"a   ll  , ,U 3;;**  $q"a   ^\eL ]KEJ << LL23>BUAVX  r>r1rAc[U5n[US5(a UR$[US5(a UR$[ S5e)z)Auto-detect feature dimension from model.head_hidden_size num_featureszCannot auto-detect feature dimension. Model must have 'head_hidden_size' or 'num_features' attribute, or you must specify student_feature_dim and teacher_feature_dim explicitly.)rr2rrrGrQs r<r+FeatureDistillationTask._detect_feature_dimsTU# 5, - -)) ) UN + +%% %R r>ruc SSKJn URR5H nSUlM U"UR 4SU0UD6UlU$)afPrepare task for distributed training. Wraps the trainable module (student + projection) in DistributedDataParallel (DDP) while leaving the frozen teacher model unwrapped. Args: device_ids: List of device IDs for DDP (e.g., [local_rank]) **ddp_kwargs: Additional arguments passed to DistributedDataParallel Returns: self (for method chaining) rrwFru)ryrxr]rzr{rr|s r<r+FeatureDistillationTask.prepare_distributed*sL" E\\,,.E"'E /!$D$9$9 _j _T^ _ r>r?rcURU5up4URX25n[R"5 URR XR UR5nUR UR5SS9nSSS5 [R"UW5nURU-URU--n U UUUS.$!,(df  NK=f)auForward pass with feature distillation. Args: input: Input tensor [B, C, H, W] target: Target labels [B] Returns: Dictionary containing: - 'loss': Combined training loss (task + distillation) - 'output': Student logits (for metrics) - 'task_loss': Classification loss component - 'kd_loss': Feature distillation loss component TrNr) rrfr4rr]rOrKrLrrmse_lossrjri) r8r?rrrrrteacher_featuresrrs r<rIFeatureDistillationTask.forwardCs$,0+@+@+G(NN>: ]]_||33E;L;LdN^N^_H#||HOO,=t|T **-/?@**Y69Q9QT[9[[ $"   _s AC C)rfrirjr]r) NNNNNNNNTr)r/rSrTrUrVr+r,rr'r rrrWr4rrrYr% staticmethodrrrrXrrIrZr[r\s@r<rrsp L.25937041515-1+/ `99`!bii1D!DE`  * ` &.c] ` "*% `'uo`"*#`"*#`U\\*`EKK(```D 299   "*.  # 2! <<! LL!  c5<< ! ! r>r)rVloggingtypingrrrrr4torch.nnr+torch.nn.functional functionalr timm.modelsr timm.utilsrtaskr getLoggerr/r(r,r r'rrrarcrrr#r>r<rs;// $#   H %uQ"))uQp"sBII'::;"yy""#"& "  $ "  "Jx Lx v(0(0VO lO r>