Sunday, July 20, 2008

Amazon S3 7/20/2008: EPIC FAIL

I have a feeling this is one that will go down in history and will spark a few debates about the reliability of Services in the Cloud.

S3 - Great Idea, but was Amazon really the right provider?


This is the question that will probably be asked over and over again for the next several weeks.

When talking "shop talk" with co-workers, I have always come out on the side of Amazon. It's something I like to call Technology Recycling. They know how to keep their mammoth site up, and they have requirements that vastly eclipse most providers, so why not use their knowledge of hosting large amounts of data as a profit center?

The Fall Out of a Huge Outage


Working for a telecom, I'm very familiar with concepts surrounding disaster recovery and the incredible effort that goes into handling disasters. You can't avoid that hurricane that might blow through your office in Texas, or the earthquake in California, or (heaven forbid) an act of war that may interrupt services.

The cause is unimportant (until the problem is solved), the response is critical. Amazon's response has been pretty thorough communication, but through that communication, it's not exactly clear that they were ever prepared for this.

When this is over, Amazon will have learned something that enterprise telecommunications carriers know all too well: When you have an outage, you have no idea how much the impact of that outage will be. Enterprise carriers offer services to companies who turn around and use those services to sell services of their own. Sometimes the end-customer is several companies deep.

Case in point, this one hit me personally today. I sell products on eBay and use a provider called Auctiva. I'm not a power seller, I just don't like to throw things away that someone might be able to use, so I list old items on eBay or Craigslist, and if they don't sell, I put them on freecycle (or Craigslist "free" section). If nobody wants it, I put it to the curb.

I sell less than I give away (and throw away very little), but the amount that I sell is enough that I had to get a tax ID number here in Michigan.

One of the golden rules of small time sellers is closing your auctions on Sunday at 9:00 PM EST. It's probably half folk-lore about eager west coast buyers winding their Sundays down with a browsing of eBay, but I (and many others) stick to it.

Today is Sunday and I have an item that was getting great bidding right up until early this morning when all of my Auctiva hosted images vanished (lets face it, broken links to images don't exactly instill faith in the seller). Until now, I didn't know I was using Amazon's S3 service and at this point I'm strongly regretting using Auctiva's service. It's going to cost me about $30 (it's within $30 of my researched target price and nobody is bidding despite lots of activity last night).

It's not the end of the world for me, but I'm going to guess that it could spell some serious consequences for others. This is made especially more painful since the company who made the Fail Wail famous is among its victims.

Twitter and SmugMug have no excuses. They relied solely on the services of one individual company with no real back-up back plan.

Granted, Twitter and SmugMug aren't life and death services (well, most of the time).

The obvious loser here is S3. If they were hoping to attract the attention of companies who provide services that require five nine's reliability, they've already lost.

So, while I still think that S3 was a great idea from Amazon, it's beginning to feel like Sears Dental: While it's convenient to have oral surgery while the folks downstairs are replacing the tires on your truck, there are some things you want to leave to companies who "specialize", rather than diversify.

No comments: