Evaluation of the performance of tightly coupled parallel solvers and MPI communications in IaaS from the public cloud

Abstract : IaaS from the public cloud is becoming a new option for organizations in need of HPC capabilities. IaaS offers greater flexibility in hardware choices and operational costs. Furthermore, IaaS enables small organizations to access resources previously reserved for organizations with larger capitals. This study uses the HPCG benchmark to assess the performance of parallel solvers, which are critical in computational engineering, and microbenchmarks to measure collective MPI operations. The benchmarks encompass IaaS from five cloud vendors (AWS, Google Cloud Platform, Azure, Oracle CIoud Infrastructure, Packet), and two architectures, ARM and x86_64. The benchmarks cover clusters with up to 4,500 cores and illustrate the benefits of higher network bandwidth when scaling up clusters. The results for some of the clusters are particularly promising as they exhibit good scalability and compare well to on-premises supercomputers. Additionally, the study includes a preliminary cost estimate based on on-demand prices for IaaS computational power and memory.

 EXISTING SYSTEM :

 ? These are organized via a taxonomy that considers the viability of HPC cloud, existing optimizations, and efforts to make this platform easier to be consumed. ? Most existing HPC software systems offer licenses to enable their usage on on-premise user environments, which can be single servers or computer clusters. ? Most of the services currently available in these clouds are meant for users to consume already-existing APIs of pre-trained models. ? This paper introduced a taxonomy and survey for the existing efforts in HPC cloud and a vision architecture to expand HPC cloud adoption with its respective research opportunities and challenges.

 DISADVANTAGE :

 ? To bridge the divide between HPC and clouds, we present the complementary approach of making HPC applications cloud-aware by optimizing an application’s computational granularity and problem size for cloud. ? The problem is even more challenging since different application react differently to different platforms. ? We will refer to this as Multi-Platform Application-Aware Online Job Scheduling (MP A2OJS). ? GrADS project addressed the problem of scheduling, monitoring, and adapting applications to heterogeneous and dynamic grid environment.

 PROPOSED SYSTEM :

 • In the latter, scheduling policies that are aware of application and platform characteristics were proposed. • To tackle such an issue, the authors proposed a method to enable the MPI runtime to detect processes that share the same physical host and use in-memory communication among them. • In the optimization of resource allocation for HPC cloud, we observe a series of efforts on platform selectors due to hybrid cloud and multiple cloud choices, and studies aligned with specific features in cloud environments such as spot instances and elasticity. • All of these optimizations benefit from resource usage and performance predictions.

 ADVANTAGE :

 ? We evaluate the performance of HPC applications on a range of platforms varying from supercomputer to cloud. ? The insights from performance evaluation, characterization, and multi-platform scheduling are useful for both – HPC users and cloud providers. ? To gain thorough insights into the performance of selected platform, we chose benchmarks and applications from different scientific domains and those which differ in the nature, amount, and pattern of inter-processor communication. ? Despite the poor sequential speed, Ranger’s performance crosses Open Cirrus, private cloud and public cloud for some applications at around 32 cores, yielding a much more linearly scalable parallel performance.