Wednesday 26 February 2025
In recent years, deep learning models have become increasingly prevalent in a wide range of applications, from self-driving cars to medical diagnosis. However, as these models grow more complex and sophisticated, they also become more vulnerable to errors caused by soft faults, which are glitches that occur when a bit flip or other fault disrupts the normal functioning of a digital circuit.
Soft faults can have devastating consequences for deep learning models, causing them to produce incorrect results or even crash entirely. This is because the subtle changes introduced by these faults can easily propagate through the complex neural networks used in these models, leading to catastrophic failures.
One way to mitigate this problem is to use bounded activation functions, which are designed to limit the range of values that a neuron’s output can take on. By doing so, these functions help to reduce the impact of soft faults by preventing them from causing drastic changes in the model’s behavior.
In a recent study, researchers explored the effects of using bounded activation functions on deep learning models’ robustness against soft faults. They found that these functions significantly improved the models’ resistance to errors caused by bit flips and other faults.
The researchers used a type of neural network known as an encoder-decoder model, which is commonly used for tasks such as image segmentation. They trained several different versions of this model using different activation functions, including the standard ReLU (rectified linear unit) function, as well as two bounded variants: sigmoid and hard sigmoid.
The results showed that both sigmoid and hard sigmoid activation functions significantly improved the models’ robustness against soft faults compared to the standard ReLU function. The sigmoid function performed particularly well, reducing the error rate caused by bit flips by over 90%.
The researchers also found that pruning, a technique used to reduce the complexity of deep learning models by removing redundant or unnecessary connections, can have an unexpected impact on their robustness against soft faults. While pruning can improve the models’ speed and efficiency, it can also make them more vulnerable to errors caused by bit flips.
In addition to using bounded activation functions, the researchers explored the effects of quantization, a technique used to reduce the precision of deep learning model’s weights and activations in order to save memory and increase processing speed. They found that quantizing the models’ weights and activations can significantly improve their robustness against soft faults.
Overall, the study highlights the importance of considering the potential impact of soft faults on deep learning models, particularly as they become increasingly complex and sophisticated.
Cite this article: “Robustifying Deep Learning Models Against Soft Faults”, The Science Archive, 2025.
Deep Learning, Soft Faults, Bounded Activation Functions, Neural Networks, Encoder-Decoder Models, Image Segmentation, Relu Function, Sigmoid Function, Hard Sigmoid, Pruning, Quantization
