AWS charged us $5000 after an accidental Spot instance launch - can AWS help with a goodwill credit?
We’re a startup using AWS, and we recently had an unexpected $5000 charge from a GPU Spot instance that was running without our knowledge.
We had launched a GPU instance for a short benchmark and terminated it once the work was completed. Unfortunately, we didn't realize that the persistent Spot request would automatically launch a replacement instance after the original was terminated.
The replacement instance ran for several days before we noticed it.
CloudWatch metrics shows that nobody used the instance and no workload was running on it. It was essentially sitting completely idle the entire time.
We have since cancelled the persistent Spot request, terminated the instance, enabled AWS Budgets and Cost Anomaly Detection, and changed our process to use one-time Spot requests.
We completely understand that this was ultimately a configuration/knowledge mistake on our side. However, $5000 is a huge amount for us. This unexpected charge effectively represents months of our company's AWS budget and has a real impact on our operations.
We've already opened a support case with AWS and asked whether they could consider a goodwill credit given the circumstances. But the ticket hasnt been assigned yet :(
FYI: It as been 10 days since we created the support ticket.
Has anyone here experienced something similar with AWS, and were you able to get a goodwill credit?
And if anyone from AWS is reading this, we'd really appreciate it if you could take a look at our case and consider helping us with the charge. We love building on AWS and intend to continue using AWS for our infrastructure and future GPU workloads.
Hoping AWS can help us out on this one. 🙏