Wednesday 19 March 2025
Researchers have long struggled with a pesky problem in machine learning: plasticity loss. This phenomenon occurs when neural networks, which are designed to learn and adapt to new data, become unable to do so over time. The result is a model that’s stuck in its ways, struggling to perform well on new tasks or adjust to changes in the data it’s trained on.
A team of scientists has now proposed a novel solution to this problem: Activation by Interval-wise Dropout (AID). This technique uses a combination of traditional dropout and interval-wise activation functions to create subnetworks that are more robust to plasticity loss. The result is a model that can learn new tasks without forgetting old ones, and adapt to changes in the data it’s trained on.
The researchers tested AID on several benchmarks, including continual learning tasks such as Permuted MNIST and Random Label MNIST, as well as reinforcement learning environments like Atari 2600. In each case, AID outperformed traditional dropout and other regularization techniques, demonstrating its ability to mitigate plasticity loss and preserve model adaptability.
One key advantage of AID is its ability to create subnetworks that are more diverse and active than those created by traditional dropout. This is because AID applies dropout at different intervals during the training process, rather than randomly selecting units to drop out as in traditional dropout. By doing so, AID creates a network with multiple paths through which information can flow, making it more resilient to changes in the data.
Another advantage of AID is its ability to learn new tasks without forgetting old ones. This is particularly important in real-world applications where models may need to adapt to changing data distributions or new tasks. Traditional dropout and other regularization techniques have been shown to suffer from catastrophic forgetting, where a model forgets previously learned information when learning a new task.
The researchers also explored the effectiveness of AID on reinforcement learning environments like Atari 2600. They found that AID was able to learn complex behaviors in these environments with greater ease than traditional dropout and other regularization techniques. This is because AID creates a network that is more robust to changes in the data it’s trained on, allowing it to adapt more effectively to new situations.
Overall, AID presents a promising solution to the problem of plasticity loss.
Cite this article: “Mitigating Plasticity Loss with Activation by Interval-Wise Dropout (AID)”, The Science Archive, 2025.
Machine Learning, Neural Networks, Plasticity Loss, Activation Functions, Dropout, Interval-Wise, Subnetworks, Continual Learning, Reinforcement Learning, Atari 2600
