Saturday 01 February 2025
As scientists and researchers increasingly rely on cloud computing to power their high-performance computing (HPC) workloads, a pressing challenge has emerged: how to optimize resource selection to minimize execution times and costs. With the sheer variety of virtual machines (VMs) available in the cloud, selecting the right combination of VM types, number of nodes, and processes per node can be a daunting task.
To tackle this problem, a team of researchers has developed an open-source tool called HPCAdvisor. This innovative solution automates the process of setting up a cloud environment, running benchmarking runs, handling output, and providing users with resource selection recommendations. By leveraging data analytics and optimization techniques, HPCAdvisor can help users make informed decisions about their HPC workloads.
The researchers used two well-known HPC applications – OpenFOAM and LAMMPS – to test the effectiveness of HPCAdvisor. They discovered that by considering application input parameters, such as mesh size or specific simulation parameters, they could reduce the number of scenarios required to generate resource recommendations. This approach not only saves time but also helps users avoid unnecessary costs.
HPCAdvisor’s algorithm uses a combination of known data points and linear interpolation to predict execution times for different VM types and node configurations. By scaling up or down from existing data points, the tool can provide accurate predictions without requiring extensive data collection. The researchers also explored machine learning techniques to further optimize resource selection, but this approach requires substantial historical data.
The results of the study are promising, with HPCAdvisor demonstrating the ability to reduce the number of scenarios required to generate resource recommendations by a significant percentage. This means that users can quickly find the optimal configuration for their calculations without incurring unnecessary costs or delays.
As cloud computing continues to evolve and new hardware emerges, HPCAdvisor has the potential to revolutionize the way researchers approach HPC workloads. By streamlining the process of resource selection, this tool can help scientists and engineers focus on what matters most – conducting high-quality research and advancing our understanding of the world.
Cite this article: “Optimizing Resource Selection for Cloud-Based HPC Workloads with HPCAdvisor”, The Science Archive, 2025.
Cloud Computing, High-Performance Computing, Hpcadvisor, Resource Selection, Optimization, Vms, Data Analytics, Open-Source Tool, Machine Learning, Linear Interpolation
