Saturday 22 March 2025
As deep learning models continue to grow in complexity and size, their deployment on resource-constrained devices like smartphones and embedded systems becomes increasingly challenging. The problem is exacerbated by the scarcity of high-quality training data, which can lead to significant accuracy drops when these models are compressed or pruned.
One solution to this issue is few-shot model compression, which involves compressing a pre-trained neural network using a small number of samples from the target dataset. However, this approach often suffers from class imbalance issues, where certain classes have significantly fewer samples than others. This can lead to biased models that perform poorly on minority classes.
Researchers have proposed various techniques to address class imbalance in few-shot model compression, including oversampling and undersampling methods. But these approaches are often ineffective or require significant modifications to the original dataset.
A new paper proposes an innovative framework called OOD-Enhanced Few-Sample Model Compression (OE-FSMC), which leverages out-of-distribution (OOD) data to mitigate class imbalance issues during few-shot model compression. The authors demonstrate that incorporating OOD data into both the compression and fine-tuning processes can significantly improve model accuracy on minority classes.
The OE-FSMC framework consists of three key components: OOD set handling, compression stage, and fine-tuning stage. During the OOD set handling phase, the authors sample a subset of the original dataset that is representative of the class imbalance issue at hand. This subset is then used to construct an auxiliary dataset for both the compression and fine-tuning stages.
In the compression stage, the pre-trained teacher model is compressed using a combination of pruning and knowledge distillation techniques. The pruned student model is trained on the original few-shot dataset with the OOD data incorporated as additional training samples. This ensures that the model learns to balance its attention across all classes, rather than focusing solely on majority classes.
During the fine-tuning stage, the compressed student model is further refined using a combination of regularization and joint distillation losses. The regularization term helps to prevent overfitting to the OOD data, while the joint distillation loss ensures that the model learns to mimic the behavior of both the teacher and original datasets.
Experimental results on three benchmark datasets demonstrate the effectiveness of OE-FSMC in mitigating class imbalance issues during few-shot model compression. The authors show that their framework can improve model accuracy on minority classes by up to 10 percentage points compared to baseline methods.
Cite this article: “Mitigating Class Imbalance in Few-Shot Model Compression using Out-of-Distribution Data”, The Science Archive, 2025.
Deep Learning, Few-Shot Model Compression, Class Imbalance, Out-Of-Distribution Data, Ood-Enhanced Few-Sample Model Compression, Pruning, Knowledge Distillation, Regularization, Joint Distillation Loss, Neural Networks.
