25 March 2014

(You might want first see the introduction to this series of posts if you jumped in here randomly.)

In my previous post I stated that AWS has set minimum spot instance prices and incorrectly asserted that these minimums are can be seen in the

 1  price-too-low
error when submitting low bids. This is wrong (I’ve updated the earlier blog post slightly to avoid spreading the wrong fact), as the “minimum” bid price given is actually the current spot price. Oh, how could I miss that.

### How to find a minimum spot price

Thus there is no direct way to get the minimum spot price in any market. But it is possible to infer these indirectly from spot price history data? I looked at the minimum spot prices in all regions and instance types (picking the lowest of all zones) and plotted it getting the graph below (the dataset spans 2013-12-08 to 2014-03-09):

Take a look at the graph for a few seconds. Go ahead. Don’t skip ahead until you’ve taken a bit of time to look at the dots. Okay, let’s continue then. I think you have also noticed there are patterns. There is a distinct pattern for several instance types in the data —

 1  m3.xlarge
, 1 m3.2xlarge, 1 t1.micro for example, but also continuing over the 1 m1 and 1 m2 classes. There may be another pattern with some 1 c3 instance types and maybe yet another for 1 m3 too, but let’s stick with the most obvious one for now. I’ll label the set of “similar” minimum price pattern the “suspect” group. Below is a plot with all of the “suspect” instance types (minimum relative prices) plotted on top of each other:

This graph requires a bit of thought to understand so bear with me. These are relative minimum spot prices so although the absolute minimum spot price differs from region to region these should be comparable to each other. In the graph there are two things that you need to be considered:

• Similarity between regions. It is quite clear that

 1  ap-southeast-1
, 1 ap-southeast-2 and 1 us-west-1 are almost identical to each other. With eyes squinted 1 us-west-2 has a cousin-style similarity to these three but all of the others are definitely dissimilar.

• Levels of the relative prices. You can find the cheapest spot instances (for the instance types discussed) in

 1  us-east-1
whereas 1 ap-northeast-1 and 1 eu-west-1 are clearly more expensive (err… less cheap?). All of the others seem to have roughly similar average minimum level.

What I find very interesting is the identical levels and structure between the three topmost regions. Here’s the raw data from these three regions in a tabular format:

These values are in dollars e.g. they are not relative prices. First of all both

 1  ap-southeast
regions have exactly the same observed minimum spot prices. The 1 us-west-2 region has several instances where prices are identical to 1 ap-southeast regions, some where prices are slightly higher and some slightly lower. However it should be noted (as can be seen in the earlier graph with relative prices) these differences are very small compared to differences to some other regions.

Regardless how you slice and dice these numbers I find it exceedingly unlikely that these very similar relative and absolute minimum spot prices in multiple regions would be result of pure chance. At least from practical point of view there are minimum spot prices.

### How are minimum spot prices set?

If this really is so, it raises another question: How are these minimum spot prices determined? Have they been set by AWS itself, or are they an artifact of external bidders during the data period?

And if they are set by AWS, then by what policy does AWS set them?

One possibility is that they are calculated from region’s on-demand prices. Yet at least for the three regions considered above this does not hold. The

 1  ap-southeast
regions have same on-demand prices but 1 us-west-2 has substantially lower prices. Yet it has very similar minimum spot prices. For example the 1 c1.medium instance type has an observed minimum spot price of $0.028, yet its price in 1 ap-southeast regions is$0.183 compared to $0.145 in 1 us-west-2. Could the minimum spot prices be based on operational costs? This seems more plausible since there seems to be similarity in the minimum prices between regions for the same instance generation — that is, instances running on so-called “first generation” hardware (compared to “second generation” of m3 and “new generation” of c3 classes) such as  1  m1 , 1 c1, 1 m2 and 1 t1 had somewhat similar prices between regions. For 1 c3 class instance types this similarity is even more striking: Note that apart two outliers (more on these below) the relative prices for  1  c3 instances between all of the regions are almost exactly the same. Again although absolute minimum prices differ the relative minimums (percentage of the on-demand price) is almost exactly the same for almost all regions for almost all 1 c3 instance types. In plain English this means that for all  1  c3 instance types the minimum bid price is a fixed percentage of that region’s on-demand instance price. That’d be a mighty coincidence if there was no minimum and all of this similarity was result of different bidders all over the world? Okay. There’s the question of the two outliers above:  1  c3.large in 1 ap-southeast-2 at$0.001 and 1 c3.8xlarge in 1 ap-northeast-1 at $0.060. Getting 1 c3.8xlarge at 1/40 of the on-demand price seems like a great bargain. If AWS enforced minimum prices I’d expect them to be set in all regions. Was it left out accidentally for these two cases? If so, it was around for quite a while (the$0.060 price for 1 c3.8xlarge was hit in 2013-12-10, 2013-12-12, 2014-01-03 and 2014-01-04 for a total of 338 minutes at that price and the $0.001 for 1 c3.large on 16 distinct days between 2013-12-22 and 2014-02-03 for a total of 13.4 days.) For the outliers I have two explanations: • It’s an accident. For some reason the minimum was set incorrectly for these two instance types. • There is actually no minimum spot price. Reconciling this view with the observed very similar minimum relative prices in multiple regions is difficult, though. One possibility that comes to mind is that maybe — maybe someone itself is bidding in all spot markets for spare capacity at a very low price and these low prices reflect those bids when there is very little demand by other customers. Yet even this outlandish scenario raises new questions. Why would this hypothetical all-excess-capacity-sucking entity bid at different prices or even at different percentages of on-demand prices? Why not bid at absolute$0.001 or at relative 10% in all regions for all instance types? If someone was really doing this then it seems reasonable that more than one market would have seen such a lull in demand that such a consistent low bid would become visible. This hasn’t happened (at least not in the data I have).

What else could be gleaned from this data?

• Minimum relative prices for 1st generation instances are more spread out than for

 1  c3
instance types. Actually, the 1 c3 relative prices are very tightly packed.

I’m definitely not sure, but this might reflect the underlying hardware and its operational costs. I think AWS has learned a lesson or two from first generation instances and its hardware. It seems reasonable that they have better understanding on how to optimally pack instances into a single server and how to operate them.

The earlier generation instances run on multiple different types of hardware (it is know they have different CPU types, at least, see Is the Same Instance Type Created Equal? Exploiting Heterogeneity of Public Clouds by Ou, Zhuang et al, 2013) and it seems that the capacity progression in

 1  m1
, 1 m2 and 1 c1 classes is not always a simple 2x step from instance type to another (potentially leaving “unfilled gaps”). These might explain why older instance types have more “spread” in their relative minimum prices than the newer ones.

• Japan and Europe (

 1  ap-northeast-1
and 1 eu-west-1) have consistently higher minimum relative spot prices than most other regions. OTOH, 1 ap-southeast-1 is the odd one out since with old instance types it had identical relative prices with two other regions, but for 1 c3 instances it stands out as substantially more expensive (“less cheap”).

This might be due to relatively higher operational costs in these regions and that the on-demand prices are not entirely based on operational costs. That is, they take into account competition, willingness of the market to accept the given price level as well as any desire by AWS to establish a (dominant?) market presence. All of these may provide a rationale to price on-demand instances at relatively lower gross margin level than in other, more established regions.

• Also it is interesting that

 1  us-east-1
doesn’t stand out as “cheaper” for 1 c3 instances in the same way it did for 1st generation instance types.

### Summary

So as a conclusion to the question whether there are minimum spot prices and who sets them I think the answer is that from a practical point of view there are minimum spot prices. If you are bidding in these markets you have to understand that there appears to be a minimum bid price you have to use to have any chance of getting an instance, ever. Although not conclusive, I find the most plausible scenario for these minimum prices that AWS is setting minimum spot prices based on operational costs, but using different formulae for different instance generations.