This presentation explores a theoretically grounded approach to knowledge distillation with emphasis on the control of dark knowledge during incremental learning. The study investigates how SoftMax temperature influences knowledge transfer, class separability, and catastrophic forgetting in large-scale classification tasks. Unlike heuristic-based approaches, the proposed method introduces a mathematically motivated strategy for selecting the distillation temperature, aiming to improve learning stability and model generalization. Experimental evaluations on benchmark datasets demonstrate the relationship between temperature scaling and the effectiveness of dark knowledge transfer, highlighting both the advantages and the practical limitations of knowledge distillation in incremental learning scenarios.
Short Bio:
I am currently enrolled in the fourth year of my PhD at the West University of Timișoara. I hold a bachelor’s degree in Computer Science and a master’s degree in Artificial Intelligence and Distributed Computing. My research focuses on Optimization of Knowledge Distillation Techniques for Incremental Learning in Deep Neural Networks.
