Monday 10 March 2025
The quest for efficient and cost-effective machine learning training has led researchers to a crucial breakthrough in large-scale Ads recommendation model training at Google. By tackling three critical challenges, scientists have successfully optimized the process, resulting in a 116% performance boost and an 18% reduction in training costs.
Firstly, input generation and ingestion pipelines are often identified as major bottlenecks in training performance. To address this issue, researchers developed a shared input generation service that amortizes the cost across several training pipelines. This approach enables efficient transformation of raw features into numerical inputs, reducing computational load and streaming them to Tensor Processing Units (TPUs) for processing.
Secondly, handling large embedding tables is a significant challenge in scaling up Ads recommendation models. The team introduced partitioning, pipelining, and Remote Procedure Call (RPC) coalescing techniques to optimize embedding table lookup operations on both TPUs and Central Processing Units (CPUs). By distributing the workload across multiple processing units, they achieved improved performance and reduced memory usage.
Lastly, interruptions and error handling are crucial aspects of maintaining efficiency at scale. To minimize resource wastage caused by preemptions from other jobs in shared datacenters, researchers implemented a novel preemption notice mechanism that anticipates and handles preemptions gracefully. Additionally, a training hold infrastructure was developed to intelligently pause models encountering permanent errors, preventing wastage of TPU resources.
The results are impressive: the optimized approach has significantly reduced training costs while improving performance by 116%. This achievement is particularly noteworthy given the complexities involved in scaling up Ads recommendation models for industry-scale applications. The innovations presented here have far-reaching implications for machine learning research and development, as they demonstrate the potential for efficient and cost-effective large-scale model training.
In this era of rapidly advancing artificial intelligence, the ability to train complex models at scale is critical for various industries, including online advertising. By addressing the challenges associated with input generation, embedding tables, and error handling, researchers have opened up new avenues for exploring the possibilities of machine learning in real-world applications. As the demand for efficient and cost-effective model training continues to grow, this breakthrough serves as a beacon of hope for the future of AI development.
Cite this article: “Optimizing Large-Scale Ads Recommendation Model Training at Google”, The Science Archive, 2025.
Machine Learning, Ads Recommendation, Model Training, Large-Scale, Efficient, Cost-Effective, Tensor Processing Units, Tpus, Central Processing Units, Cpus, Remote Procedure Call, Rpc
