Friday 28 February 2025
As AI-powered computer vision models continue to dominate the field of image recognition, a new threat has emerged: adversarial attacks that can fool these models into misclassifying images. Researchers have been scrambling to develop methods to improve the robustness of these models against such attacks. A recent paper proposes a novel approach called SAFER (Sharpness-Aware Fine-tuning for Enhanced Robustness), which leverages sharpness-aware minimization to selectively fine-tune layers in vision transformers and boost their adversarial robustness.
The core idea behind SAFER is to identify the layers most prone to overfitting during adversarial training and focus on fine-tuning those specific layers using a sharpness-aware optimizer. This approach allows for more efficient adaptation of the model to the adversarial environment, leading to improved performance against various attacks.
To test the effectiveness of SAFER, the researchers trained several vision transformer models using their proposed method and compared them to baseline models trained with standard adversarial training techniques. The results showed that SAFER consistently outperformed the baselines across various datasets and attack methods, demonstrating its ability to improve the robustness of vision transformers against adversarial attacks.
One of the key advantages of SAFER is its ability to selectively fine-tune layers based on their sharpness, which allows for more targeted adaptation to the adversarial environment. This approach also avoids the need for early stopping, a common technique used to mitigate overfitting in adversarial training. By contrast, SAFER’s adaptive nature enables the model to continue learning and improving throughout the training process.
The researchers also conducted an ablation study to investigate the impact of dynamic layer selection and the number of layers chosen for fine-tuning on the performance of SAFER. The results showed that dynamically selecting the top-2 layers resulted in the best performance, while choosing additional layers led to a decline in performance.
To further evaluate the effectiveness of SAFER, the researchers also tested its performance against a range of attacks, including FAB, STADV, and ℓ∞-bounded PGD attacks. The results showed that SAFER consistently outperformed baseline models across all evaluated attacks, demonstrating its ability to improve the robustness of vision transformers against diverse adversarial threats.
Overall, the proposed SAFER approach offers a promising solution for improving the adversarial robustness of vision transformers.
Cite this article: “SAFER: A Novel Approach to Enhance Adversarial Robustness in Vision Transformers”, The Science Archive, 2025.
Vision Transformers, Adversarial Attacks, Robustness, Fine-Tuning, Sharpness-Aware, Minimization, Overfitting, Adversarial Training, Image Recognition, Computer Vision
