Thursday 20 March 2025
The pursuit of faster, more accurate, and memory-efficient machine learning has led researchers to explore novel optimization strategies. One such approach is zeroth-order (ZO) optimization, which relies solely on forward passes for gradient estimation, bypassing the need for backward propagation and significantly reducing storage requirements.
However, ZO methods have historically lagged behind traditional first-order (FO) fine-tuning in terms of convergence speed and accuracy. To bridge this gap, a team of researchers has introduced Divergence-Driven Zeroth-Order (DiZO) optimization, which leverages layer-wise divergence analysis to uncover the distinct update patterns of FO and ZO methods.
DiZO conducts divergence-driven layer adaptation by incorporating projections to ZO updates, generating diverse-magnitude updates precisely scaled to individual layer optimization needs. This approach allows DiZO to significantly reduce the needed iterations for convergence without sacrificing throughput, cutting training GPU hours by up to 48% on various datasets.
The researchers tested DiZO on a range of pre-trained language models, including RoBERTa-large and OPT-series, as well as Llama-series models. Results demonstrated that DiZO consistently outperformed representative ZO baselines in fine-tuning tasks, often surpassing memory-intensive FO fine-tuning.
To better understand the performance benefits of DiZO, the researchers conducted experiments on the SST-2 and SQuAD datasets, comparing DiZO to MeZO (a memory-efficient variant of traditional first-order optimization) and HiZOO (another ZO method that relies on second-order information estimation). The results showed that DiZO not only matched but often exceeded the performance of these methods while reducing training GPU hours.
The researchers also explored the application of DiZO to Llama-series models, which employ a Group Query Attention mechanism. While DiZO performed well on these models, it did struggle with some datasets, suggesting that further investigation is needed to fully understand the limitations and potential of this approach.
From a technical standpoint, DiZO relies on a novel projection learning strategy that leverages perturbations to estimate gradients. The researchers implemented this approach in PyTorch, demonstrating its feasibility and flexibility.
The development of DiZO represents an important step towards more efficient and effective machine learning optimization strategies. As the demand for AI-powered applications continues to grow, innovations like DiZO will play a crucial role in ensuring that these systems can be deployed and scaled without compromising performance or resource constraints.
Cite this article: “Accelerating Zeroth-Order Optimization with Divergence-Driven Layer Adaptation”, The Science Archive, 2025.
Machine Learning, Optimization, Zeroth-Order, First-Order, Fine-Tuning, Divergence-Driven, Layer Adaptation, Projection Learning, Pytorch, Ai-Powered Applications
