AWS Spot instances price prediction - towards cost optimization for Big Data

Analytical data processing has become the cornerstone of today's business success, and it is facilitated by Big Data platforms that offer virtually limitless scalability. However, minimizing the total cost of ownership (TCO) for the infrastructure can be challenging. By analyzing spot instance price history using ARIMA models, it is feasible to leverage the cloud spot market's discounted prices with a limited risk of analytical job termination. In particular, we evaluated savings opportunities when using Amazon EC2 spot instances comparing to on-demand resources. During the presentation, we show the evaluation of univariate spot price regression models to forecast future prices, and we confirm feasibility of short-term spot price prediction through real data from AWS. Our analysis identifies cost savings opportunities up to 80% compared to on-demand pricing model, and within 1% of the absolute minimum. Furthermore, we present ETL framework architecture, and we discuss how one may leverage DevOps techniques and optimize the total cost of operations of cloud infrastructure relying on forecasts of spot prices.

Link to meeting: https://meet.google.com/jbj-tdsr-aop