Thursday 13 March 2025
A comprehensive evaluation framework for deep learning-based software vulnerability detection has been proposed, aiming to ensure accurate representation of vulnerabilities that should be learned by these detectors. The framework, dubbed VIPer, employs feature-preserving and feature-eliminating perturbations to assess a detector’s performance.
The researchers behind VIPer have identified the importance of understanding the features relevant to vulnerability prediction, as well as those that may impede accurate detection. They’ve developed a taxonomy categorizing these features into two main categories: Vulnerability Features (VFs) and Spurious Features (SFs). VFs are characteristics of code that contribute to vulnerabilities, while SFs are irrelevant patterns that detectors may learn from.
To evaluate the performance of deep learning-based vulnerability detectors, VIPer employs feature-preserving perturbations, which ensure that a detector’s prediction remains unchanged while modifying the input code. Feature-eliminating perturbations, on the other hand, eliminate specific features from the input code to assess how they impact the detector’s predictions.
The researchers applied VIPer to five recent and representative vulnerability detectors, including DeepWukong, ReVeal, DeepDFA, LineVul, and SySeVR. Their study revealed that while most detectors performed well in detecting vulnerabilities, some were more susceptible to SFs than others. For example, they found that graph-based detectors like ReVeal and DeepDFA struggled with features related to system APIs, leading to reduced accuracy.
The findings of this study have significant implications for the development of deep learning-based vulnerability detection techniques. By understanding which features are most important for accurate detection and which may impede performance, researchers can design more effective models that better identify vulnerabilities. Additionally, VIPer provides a valuable evaluation framework for assessing the performance of these detectors, ensuring that they are reliable and trustworthy.
The researchers’ approach also highlights the importance of code property graphs (CPGs) in vulnerability detection. CPGs provide a comprehensive representation of code dependencies, allowing detectors to learn from complex relationships between code elements. By incorporating CPGs into their models, developers can create more accurate and effective vulnerability detection techniques.
Overall, VIPer represents a significant step forward in the development of deep learning-based software vulnerability detection. By providing a comprehensive evaluation framework and highlighting the importance of understanding VFs and SFs, this study has laid the groundwork for the creation of more reliable and trustworthy vulnerability detection models.
Cite this article: “Evaluating Deep Learning-Based Software Vulnerability Detection with VIPer”, The Science Archive, 2025.
Deep Learning, Software Vulnerability, Detection, Evaluation Framework, Viper, Feature Preserving, Feature Eliminating, Perturbations, Code Property Graphs, Cpgs
