Background

We often want to check for EC2 instance pricing data. Since the volume of instance types are vast and there are also many additional dimensions like Region, Operation, UsageType, etc to define the pricing factors, it's been a time-consuming and terrible experience to find the exact pricing we want to know quickly.

Where to find pricing data

There are many places to find pricing data:

AWS EC2 Pricing Page this is the official page on showing prices, but it's kinda slow and hard to find the information you need unless you clicked a fleet of buttons and select boxes. And there is still no way you can search.
AWS Cost Calculator grab instance type and along with other costs to define the budget.
ec2instances.info A third-party site that you can query and search for pricing data on various dimensions.
AWS Price List API This is what I want to share in this post.

AWS Price List API

This API contains all services pricing data which Amazon offer.

To get the offer code and URLs for all supported services by accessing the Offer Index:

https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json

It returns like:


{
  "formatVersion" : "v1.0",
  "disclaimer" : "This pricing list is for informational purposes only. All prices are subject to the additional terms included in the pricing pages on http://aws.amazon.com. All Free Tier prices are also subject to the terms included at https://aws.amazon.com/free/",
  "publicationDate" : "2015-11-19T02:10:02Z",
 "offers" : {
    "AmazonS3" : {
      "offerCode" : "AmazonS3",
      "currentVersionUrl" : "/offers/v1.0/aws/AmazonS3/current/index.json"
    },
    "AmazonRedshift" : {
      "offerCode" : "AmazonRedshift",
      "currentVersionUrl" : "/offers/v1.0/aws/AmazonRedshift/current/index.json"
    },
    "AmazonEC2" : {
      "offerCode" : "AmazonEC2",
      "currentVersionUrl" : "/offers/v1.0/aws/AmazonEC2/current/index.json"
    },
    "AmazonCloudWatch" : {
      "offerCode" : "AmazonCloudWatch",
      "currentVersionUrl" : "/offers/v1.0/aws/AmazonCloudWatch/current/index.json"
    }
  }
}

To take EC2 price index as an example, it contains a huge list of around 400,000 items and a single API call won't return that all information within a body. Luckily, AWS provides all of the data in a CSV format (file size around 304MB), it can be download from this link (don't open it if you are running on a low spec laptop):

A screenshot of the CSV (opened on my Mac):

Screenshot_2018-08-17 10.12.09_glSMiB.png

As you can see, the count of items is more than 400,000! And why are there so many pricing rates data?

While I filtered the CSV by columns like

InstanceType,
Operation,
Location,
TermType,
UsageType,
Tenancy,
...

then I got few lines left which contain the pricing data in a column called PricePerUnit, and the number on this cell is presenting the pricing rate for that specific instance in a specific region with specific dimensions. For example:

After filtering Region and InstanceType, an m3.medium instance in Tokyo region contains 204 pricing rate items . And the 204 pricing rate items of m3.medium instance are with different rate because there are dimensions like Operation System (Windows, Linux/Unix, SUSE, Enterprise Linux, etc are with different prices), Reservation Terms (either 3-year or 1-year with upfront option like No-upfront, Partial-upfront and Full-upfront, under Convertible or Standard types), Pre-installed Softwares, etc to factor the pricing.

Dump CSV into dynamoDb

There are many big data solutions to visualize a huge CSV. Stuff like BigQuery, QuickSight, etc. But I decided to dump that into DynamoDb for the sake of fast querying. So after filtered some unused values, I got 309,365 items of EC2 offering data available globally:

Screenshot_2018-08-17 09.51.59_cFq5b1.png

Click on an Item, the pricing data of this specific m5.12xlarge rate info looks like this:

Screenshot_2018-08-17 10.11.06_RVlKUt.png

Now, I can easily query the pricing info against the instance I wish to search:

Screenshot_2018-08-17 10.35.30_cOefol.png

Interesting Dimensions

The pricing data are so vast because of dimensions. And when you are about to calculate the true unblended cost by yourself you gotta understand all of these dimensions. I picked some of the most important ones:

TermType: OnDemand, or Reserved
PriceDescription: The description of the pricing item
ProductFamily: Compute Instance, Data Transfer or EBS Storage, etc
serviceCode: AmazonEC2
Location: Region, eg: Asia Pacific (Tokyo)
InstanceType: The type of instance, eg: c1.large
UsageType: This is the dimension used by AWS officially to calculate your monthly invoices. By matching this dimension with the column found in your monthly CUR csv, aws applies the unit price and generate the total usage cost.
Tenancy: Shared, Dedicated, or Host
Operation: This defines what OS is the instance running, eg:
- RunInstances Amazon EC2 running Linux/UNIX
- RunInstances:000g Amazon EC2 running SUSE Linux
- RunInstances:0010 Amazon EC2 running Red Hat Enterprise Linux
- RunInstances:0002 Amazon EC2 running Windows
- RunInstances:0006 Amazon EC2 running Windows with SQL Server Standard
- RunInstances:0102 Amazon EC2 running Windows with SQL Server Enterprise
- RunInstances:0202 Amazon EC2 running Windows with SQL Server Web
- RunInstances:0800 Amazon EC2 running Windows (Bring your own license)
- RunInstances:SV006 Amazon EC2 running Linux/UNIX Spot Instance-hour in US East (Virginia) in VPC Zone #6 ...
- ...

About CUR & Self-calculation

CUR (Cost and Usage Report) is a CSV file which aws provides to our S3 bucket (via payer account) on an hourly/daily basis. This CSV file contains all of the services usage data in detail.

We can hence calculate the cost base on that CUR csv file and get detailed insight of what our AWS usage is allocated to. More detail on understanding CUR and consolidated billing on this AWS documentation page.

Self-calculation can be tough and human intensive. Especially when the RI blending happens across multiple aws accounts we will have to re-calculate the correct true usage.

Luckily there are SaaS vendors who are providing such insight services:

I'm working at Mobingi team, and we've been helping many aws customers to calculate their billings every month in a total of multi-million dollars. The getting started guide on Mobingi Wave is very simple.

I'll share more on the CUR csv format in another article.