INFERALL: COORDINATED OPTIMIZATION FOR MACHINE LEARNING INFERENCE SERVING IN PUBLIC CLOUD

open_access
Open Access
Author:
Kumar, Pramod
Graduate Program:
Computer Science and Engineering
Degree:
Master of Science
Document Type:
Master Thesis
Date of Defense:
March 15, 2021
Committee Members:
Chitaranjan Das, Thesis Advisor/Co-Advisor
Mahmut Taylan Kandemir, Committee Member
Chitaranjan Das, Program Head/Chair
Bhuvan Urgaonkar, Committee Member
Keywords:
ML Inference system
Machine learning in public cloud
Machine-learning-as-a-service (MLaaS)
Inference
Serving
Batching
Cloud
Serverless
Service Level Objective (SLO)
Cost-effective
Optimization
Modeling
Prediction
Machine learning inference
Abstract:
Recently, a plethora of applications have started to include a variety of Machine Learning based inference systems for predictive analytics tasks across diverse domains such as, Computer Vision, NLP engines, Recommender systems etc. Typically these applications are time critical and hence, the inference serving system has to satisfy the stringent latency requirements while maintaining high model accuracy. Designing an inference serving system requires a trained model, which is diverse in terms of model architecture and (or) variants within the same architecture that results in a wide range of latencies and accuracies. Furthermore, these models differ in compute resource consumption, which gives an opportunity to exploit multiple models/variants to reduce the tail latency under bursty workloads. Besides model heterogeneity, batching multiple requests within a model can be used to exploit parallelism in a computation graph that can greatly improve throughput and cost. It is crucial to take into account the allocated hardware resources in terms of cores and memory to decide the optimal parallel jobs as they affect the latency of individual jobs. In addition, variants in compute resources also adversely affect the latency and cost of individual inference. For instance, recent cloud offerings like serverless functions incur higher latency, while guaranteeing instantaneous scalability for bursty workloads. In contrast, virtual machine (VM) offerings are less expensive and slow in reacting to request surges. This thesis presents "InferALL", a modular python based simulator, which jointly exploits heterogeneity in models, request-batching, compute resources and cloud offerings to provide a cost efficient and SLO-aware inference serving system. InferAll consists of two major components, an offline profiler and an online scheduler. For a given SLO, it selects an optimized configuration in a greedy fashion, which makes sure that tail latency is checked while utilizing available resources to its full extent. By using optimization opportunities across all four dimensions (models, compute resources, batch sizes and cloud offerings) and testing on real world public traces, InferALL reduces cost upto 95% (depending upon SLO), while satisfying the 99th percentile tail latency within SLO Limits, when compared to state-of-art inference serving systems. InferAll can gracefully support a wide-variety of workloads with high variance and bursty arrival patterns.