Enabling Cost-Effective, SLO-Aware Machine Learning Inference Serving on Public Cloud

Abstract : The remarkable advances of Machine Learning (ML) have spurred an increasing demand for ML-as-a-Service on public cloud: developers train and publish ML models as online services to provide low-latency inference for dynamic queries. The primary challenge of ML model serving is to meet the response-time Service-Level Objectives (SLOs) of inference workloads while minimizing serving cost. In this paper, we proposes MArk (Model Ark), a general-purpose inference serving system, to tackle the dual challenge of SLO compliance and cost effectiveness. MArk employs three design choices tailored to inference workload. First, MArk dynamically batches requests and opportunistically serves them using expensive hardware accelerators (e.g., GPU) for improved performance cost ratio. Second, instead of relying on feedback control scaling or over-provisioning to serve dynamic workload, which can be too slow or too expensive, MArk employs predictive auto scaling to hide the provisioning latency at low cost. Third, given the stateless nature of inference serving, MArk exploits the flexible, yet costly serverless instances to cover occasional load spikes that are hard to predict. We evaluated the performance of MArk using several state-of-the-art ML models trained in Tensor Flow, MXNet, and Keras. Compared withthe premier industrial ML serving platform SageMaker, MArk reduces the serving cost up to7.8×while achieving even better latency performance.

 EXISTING SYSTEM :

 The advances of Machine Learning (ML) have sparked growing demand of ML-as-a-Service: developers train ML models and publish them in the cloud as online services to provide low-latency inference at scale. The key challenge of ML model serving is to meet the response-time Service Level Objectives (SLOs) of inference workloads while minimizing the serving cost.

 DISADVANTAGE :

 The accuracy of prediction depends on the underlying workload, there is no such a universal method that works perfectly in all cases. The challenge is how to gracefully handle unavoidable prediction errors and unexpected load surges.

 PROPOSED SYSTEM :

 In this paper, we tackle the dual challenge ofSLO compliance and cost effectiveness with MArk (ModelArk), a general-purpose inference serving system built in Amazon Web Services (AWS). MArk employs three design choices tailor-made for inference workload. First, MArk dynamically batches requests and opportunistically serves them using expensive hardware accelerators (e.g., GPU) for im-proved performance-cost ratio. Second, instead of relying on feedback control scaling or over-provisioning to serve dynamic workload, which can be too slow or too expensive for inference serving, MArk employs predictive auto scaling to hide the provisioning latency at low cost. Third, given the stateless nature of inference serving, MArk exploits the flexible, yet costly server less instances to cover the occasional load spikes that are hard to predict. We evaluated the per-for mance of MArk using several state-of-the-art ML models trained in popular frameworks including Tens or Flow, MXNet,and Keras. Compared with the premier industrial ML serving platform Sage Maker, MArk reduces the serving cost up to7.8×while achieving even better latency performance.

 ADVANTAGE :

 ? Therefore, an ML model serving system should strive to meet the target SLOs while minimizing the cost of provisioning the serving instances in the cloud. ? Our measurements suggest that among the three options, IaaS offers the best performance-cost ratio for inference serving, but it incurs long instance provisioning latency and is hence unable to quickly adapt to the changing workload.