Wednesday 09 April 2025
Deep learning has revolutionized many areas of computer science, but one area where it still lags behind is active learning – a technique that allows machines to learn from human feedback in an efficient and cost-effective manner. Active learning typically involves selecting which samples to label next, based on some measure of uncertainty or relevance. But this approach can be limited by the quality and quantity of labeled data available.
Researchers have been working on developing more sophisticated active learning algorithms, but many of these methods rely on domain-specific knowledge or are not easily adaptable to different datasets. What’s needed is a unified framework that can learn effective representations from both labeled and unlabeled data, while also incorporating task-awareness into the selection process.
Enter ADROIT, a new approach that uses a combination of reconstruction, adversarial, self-supervised, knowledge-distillation, and classification losses in a unified VAE-based model. The key components of ADROIT are a representation generator (VAE), a state discriminator, and a proxy task-learner or classifier.
The VAE learns a latent code from both labeled and unlabeled data, incorporating task-awareness by leveraging labeled data with the proxy classifier. Unlike previous approaches, the proxy classifier also employs a self-supervised loss on unlabeled data and uses knowledge distillation to align with the target task-learner. The state discriminator distinguishes between labeled and unlabeled data, facilitating the selection of informative unlabeled samples.
The dynamic interaction between the VAE and the state discriminator creates a competitive environment, where the VAE attempts to deceive the discriminator, while the discriminator learns to differentiate between labeled and unlabeled inputs. This competition encourages the VAE to learn more robust and discriminative representations, which in turn improves the performance of the proxy classifier.
ADROIT was tested on several datasets, including CIFAR10, CIFAR100, TinyImageNet-200, ImageNet-100, modified imbalanced CIFAR10, and Caltech101. The results show that ADROIT outperforms state-of-the-art active learning baselines in terms of accuracy and efficiency.
One of the key advantages of ADROIT is its ability to adapt to different datasets without requiring domain-specific knowledge or extensive tuning. This makes it a versatile tool for a wide range of applications, from image classification to object detection.
In addition to its performance gains, ADROIT also provides insights into the importance of self-supervised learning in active learning.
Cite this article: “Boosting Active Learning with Self-Supervised Representation Learning”, The Science Archive, 2025.
Active Learning, Deep Learning, Vae, Self-Supervised Learning, Knowledge Distillation, Classification Losses, Reconstruction Loss, Adversarial Loss, State Discriminator, Proxy Task-Learner
