How To Protect Yourself From Unexpectedly High AWS Bills

gowebing

Introduction

This really happened and keep in mind that stories such as these here seem to happen with some frequency:

I used my personal AWS account for [an AWS] workshop and thought I had deleted the CF stacks at the end of the workshop, but hadn’t. At the end of the month I was shocked when I saw an unexpected $1500 charge on my AWS bill. [1]

Here’s another story — this time much worse:

So, as the old joke goes, an expert is someone that avoids all the small mistakes that amateurs make — to go on and make a really big mistake!

Well, I just discovered a $14k mistake…I had come upon a really nice job request. Automate execution of [AWS] Comprehend-Medical from some json file that ends up in an S3 bucket…Two mistakes were made. (1) As part of my usual flow, I use terraform to test the infrastructure in my own test account. When I ran the test, I underestimated the number of json tags that trigger the automation. (2) Comprehend-Medical is priced rather differently from Comprehend. It’s 20x the cost! [2]

Sadly these billing disasters may be avoidable had the user simply set up a budget with an alarm that sends an email once the bill reaches a certain threshold.

In this article we’re going to set up a budget with an alarm, and we’re going to briefly look at the Identity and Access Management (IAM) Security Status page and then we’re going to introduce AWS Single Sign-On (SSO) —  the conclusion along with a short quiz follows.

AWS SSO offers some powerful features that can be used to help mitigate the risk of an account being compromised and we recommend taking some time to read through this section of the article, including the references.

First, let’s look at what happens when your AWS usage skyrockets.

What Happens When Your AWS Usage Spikes?

Stories like those mentioned in the introduction occur with some frequency and when this happens the customer needs to speak with an AWS representative and see if a reprieve can be granted. There are no guarantees that mercy will be shown, however, so it will behoove you to take advantage of a few tools ahead of time to help prevent this kind of surprise from happening in the first place. The following question on Quora should be considered required reading for this article: My AWS account was hacked and I have a $50,000 bill, how can I reduce the amount I need to pay? — in particular, the response by Chris Clark is instructive:

I use AWS for my popular cocktail site. I was recently hit with a similarly nefarious (albeit not 50K) charge to my account when I added a new service to my stack, and unlike others here, my outcome was not positive. I’m here to represent the less-than-happy path that can occur when this sort of thing happens, offer some advice on how to approach the situation, as well as preventative measures you can take.

The not-so-happy path


Yes, you can file a ticket with billing, and while Amazon is sometimes “good on their customers”, they are not required to be. Ultimately, your claims will be at the mercy of a black-box accounting department and if you don’t like the verdict, you will be told to talk to a lawyer or the police (this is not a joke). Record everything you do and be very deliberate about your moves leading up the the case. They will make it seem like everything is gonna be fine…until it isn’t. AWS does not have the same approach to customer service as other companies. They will eventually just cut you off ice cold, so don’t think the noisy-customer approach that sometimes works with airline companies will work here.

The billing is complex. Don’t file a ticket without all of the data. I dug deep into the root of my charges, and found out I was being charged for 3,000 instance hours a month. How? Well it’s complicated, and it took a long time for me to even find that data, so long that it didn’t matter because my verdict had already been given. Make sure to know everything you can about how and why you were charged so much. They might also ask you to delete any instances related to the billing, which can make it difficult to dig up logs. Consider saving all of the log files before you delete the instances.


Be prepared for anything, because in the end the AWS ToS is a blank check. The AWS billing model is deliberately open-ended, and there is little you can do about it after a verdict has been assigned. You can accumulate a life changing amount of debt on AWS in a matter of days, and unlike your engineering mistakes, this bug can’t be fixed. For this reason, below I will offer advice on how to avoid this situation in the first place.

How you can protect yourself


Set up billing alerts. If you are using AWS, even for a small task, please please please set up billing alerts. They are not required during setup, but if you are a non-enterprise user, I would consider this step mandatory as AWS will not alert you to dramatic increases in charges unless they bypass 15K which is already an incredible amount of money.


Read the pricing table…carefully. If you are installing a new service, make sure to carefully read the pricing table. Amazon will sometimes set ridiculous defaults for container size which you might not see until the bill comes in. Do understand however that this might not be good enough, as bugs, loose API keys, and improper installations can do crazy things.


Consider using another service. If you are a non-business individual user or small-business user, you might want to consider using another service. AWS is built for enterprise customers, and as such an enterprise wallet. Yes, it can be very cheap, but consider this: after my little mix up, I could have payed $150 a month over all of the years I used AWS and still come out ahead. Yes, AWS might be cheap at first, but one mistake can make it very expensive. Currently, AWS does not offer an experience tailored to non-enterprise customers, which can be unbelievably dangerous to new users looking to learn the services. Consider opting for different services with overage models more tailored to protect the individual user.

Note that a feature which appears to be unavailable at the time of this writing, and which is mentioned in the quote above, is the ability to set a hard limit to your budget. This feature request has appeared with some frequency when people have been burned and had they set a budget they may have caught their mistake early and as the saying goes: an ounce of prevention is worth a pound of cure.

Common Mistakes

In addition to not having budgets configured, there are two other mistakes people tend to make which can cause problems: they are simply not paying attention to what they’re doing or their personal access tokens have been compromised. If you know of other problems that can result in unexpected budgetary overruns please add them in the comments.

Not Paying Attention

Not paying attention can bite in several ways, for example, the user runs a script that creates many expensive EC2 instances and walks away, a user requires a service and doesn’t check the price first, or a user forgets to delete some data and they learn later that they’re being charged for it and the fees are non-trivial.

Personal Access Keys Compromised

This is a very bad scenario — in this scenario an AWS user includes their personal access keys (PAKs) in a commit that ends up on the Internet. To be clear, we’re not really concerned about the keys so much as the values.

The suggestions here are fairly straightforward:

GitHub scans for secrets [27] but not everyone uses GitHub and if the scan misses the key/value pairs or if it’s lost by some other means, for example, someone loses a flash drive with the PAKs / PAK values stored as unencrypted text, then this can be a problem.

Create a Budget With an Alarm

In this section, we’ll create a budget of $250 and when 80% of the budget is reached an alarm will sound and an email will be sent.

The first image shows us that we can find the Budgets page under the “My Billing Dashboard” menu item.

In the following image, we can see where the “Create budget” button is located. Note that in this example we already have an existing budget which is set at $100.00 (USD).

The next page allows us to choose what kind of budget type this will be — in this example we’ll choose the “Cost budget”.


The following page facilitates the creation of the budget and we’ve gone into detail in the image regarding what the important aspects of this page include, along with an explanation pertaining to the options available in the “Aggregate costs by” option.

See also: Understanding your AWS Cost Datasets: A Cheat Sheet


Set your budget.

The next step in this process involves creating an alert, which is shown below.

Configure budget alerts.

Finally, we review the budget and then create it.

Confirm the budget and create it.

We can see that our new budget now appears in the “AWS Budgets” page.


The budget has been created.

Requires further research: If a budget is exceeded, when exactly is the notification sent? We should not assume that an alarm causes the notification to be sent immediately unless there’s evidence to support this. If you find this information please add it in a comment with the link to the source.

In the next section, we’ll briefly explore the IAM Security Status, we’ll then introduce AWS SSO, and finally, we’ll conclude the article and we hope that you’ll comment on the quiz that follows.

IAM Security Status

If you are new to AWS you should take a look at the Identity and Access Management (IAM)console and if it doesn’t look like what we have below then consider addressing whatever flags that have been raised.

Further instructions regarding how to achieve the aforementioned results can be found here:

In the next section, we’ll introduce AWS Single Sign-On (SSO).

AWS Single Sign-On (SSO)

AWS SSO is free [13, 22, 23] and requires AWS Organizations, which is also available at no additional charge, and which provides the AWS account owner with better control over access to AWS resources. We’ve included several resources for learning about AWS SSO and we’re not going to repeat that here however please see the references section of this article below, specifically numbers #13 through #24.

In the test setup that we configured, we set the session time for the Developers group to one hour and we also restricted the policy to AmazonEC2FullAccess. We authenticated via the URL provided to us by Amazon at which point we could select the environment variables from the command line or programmatic access link, see the image below.

We set these variables in the ~/.aws/credentials file and then executed the following via the command line:

aws ec2 run-instances --image-id ami-02354e95b39ca8dec --count 1 --instance-type t2.micro --region us-east-1 --profile 221025111467_AWS_1HR_AmazonEC2FullAccess_PS --subnet subnet-cm235nh7cc72969d2 --tag-specifications 'ResourceType=instance,Tags=[{Key=costCenter,Value=45678}, {Key=department,Value=Energy}]'

And the EC2 instance was created.

If we attempt to run the same command after an hour has elapsed, we’ll see the following message:

An error occurred (RequestExpired) when calling the RunInstances operation: Request has expired.

It is helpful to note a few things here:

The last point we’ll make about this involves AWS Systems Manager Session Manager (SMSM) and is left as an exercise for the reader — quoting from [25]:

“Session Manager provides secure and auditable instance management without the need to open inbound ports, maintain bastion hosts, or manage SSH keys…while still providing end users with simple one-click cross-platform access to your managed instances.”

The exercise we alluded to is as follows: Demonstrate the usage of SMSM to run a single command, such as “ls”, on a running EC2 instance.

Conclusion

While researching this article we found a new AWS service which is in preview at the moment called AWS Cost Anomaly Detection (CAD) — from [29]:

Starting today, customers can use AWS Cost Anomaly Detection to detect unexpected or unusual spend. AWS Cost Anomaly Detection uses a multi-layered state machine learning model that learns your unique spend patterns to adjust spend thresholds — this means you do not need to worry about determining appropriate thresholds (e.g. 10% increase in daily spend) and maintaining them as your usage changes over time. These machine learning layers allow Anomaly Detection to detect various types of anomalies, such as a one-time cost spike or gradual, consistent cost increases. Anomaly Detection is customizable to segment your spend data and provides tailored alerting preferences so that you are informed as soon as an anomaly is detected.

This service preview was released on the 24th of September 2020 and we’ve yet to experiment with it however this sounds like it should be on everyone’s radar.

We’ll close this article by suggesting a few things:

Hopefully, this article has been helpful. If you have any questions, please leave them in the comments section and we can discuss them. Take the quiz!

Happy coding!

Hat tip: Allan Webb, Timothy Ingalls

Quiz

Are you ready for a quiz? Place your answers in the comments section for discussion.

  1. You just opened your new AWS account for personal use, what should you do to help avoid unexpectedly high bills?

  2. Your budget is exceeded, an email is sent, and you ignore the email — what happens?

  3. Following the advice in the “Security Status” section of IAM you create a new user, one that has full access to EC2. You want to further mitigate the damage that could occur in the event that your access keys were stolen or mistakenly placed on the Internet, so describe one thing you could do which would achieve this?

  4. Is there currently a means by which to set a hard limit to your AWS account budget?

  5. For mistakes similar to those described in this article, AWS will help me? (yes, no, maybe)

  6. What can I do to help lower the chance of my account being compromised? [related to 3]. Hint: The dashboard for _____ includes a section entitled “Security Status”.

  7. Question #6 has to do with a compromised account, where can I find suggestions for improving my personal account security? See also [4, 5].

  8. For an AIM Policy, under “Create Policy”, Visual Editor, for EC2 we see a Request Condition including “MFA required” and “Source IP” — provide one example where setting the Source IP could be helpful? (see also Q3)

  9. Why would you not want to use a Source IP of 0.0.0.0/0 ?

References

  1. Anonymized conversation

  2. Anonymized conversation

  3. I was billed for 14k USD on Amazon Web Services (note this source is not the same as #1 even though the numbers are equal)

  4. Getting Started: Follow Security Best Practices as You Configure Your AWS Resources

  5. Top 10 security items to improve in your AWS account

  6. Ryan Hellyer’s AWS Nightmare: Leaked Access Keys Result in a $6,000 Bill Overnight

  7. My AWS account was hacked and I have a $50,000 bill, how can I reduce the amount I need to pay?

  8. AWS Cost Allocation Tags and Cost Reduction:
    Optimize your AWS services to pay less per month with cost allocation tags

  9. Understanding your AWS Cost Datasets: A Cheat Sheet

  10. Enabling a virtual multi-factor authentication (MFA) device (console)

  11. How can I use IAM roles to restrict API calls from specific IP addresses to the AWS Management Console?

  12. Storing Secrets

  13. AWS Single Sign-On (SSO)

  14. Introducing AWS Single Sign-On

  15. How to retrieve short-term credentials for CLI use with AWS Single Sign-on

  16. Configuring the AWS CLI to use AWS Single Sign-On

  17. Identity and Access Management for AWS SSO

  18. Overview of Managing Access Permissions to Your AWS SSO Resources

  19. Requesting temporary security credentials

  20. Amazon WorkSpaces Cost Optimizer

  21. AWS Single Sign-On Features

  22. AWS Single Sign-On FAQs

  23. AWS Organizations FAQs

  24. How to create and manage users within AWS Single Sign-On

  25. AWS Systems Manager Session Manager (SMSM)

  26. (SMSM) Start a session

  27. GitHub About secret scanning

  28. My AWS account was hacked and I have a $50,000 bill, how can I reduce the amount I need to pay?

  29. Preview: Anomaly Detection and alerting now available in AWS Cost Management

  30. AWS Cost Allocation Tags and Cost Reduction

 

 

 

 

Top