Back to normal after data center power outage

The data center that houses Nirvana’s front-end web servers suffered a prolonged power outage last night. It took a lot longer than anyone expected (or wanted) before the thousands of VMs at their facility, including ours, were able to be brought back online. The high-level sequence of events is available here.

We’re back online now and everything is running normally.
Please accept our sincere apologies if you were unable to login this morning.

We don’t usually talk much about our infrastructure (as we figure it’s not that interesting to most GTD’ers), but given recent events I suppose this might be one of those times when people might like to know more.

Over the past few months we’ve been incrementally migrating our infrastructure to a geographically distributed and fault tolerant cloud architecture, hosted at Amazon AWS.  They are truly amazing, and we are in good company.

Our databases are already running as multi-AZ replicated RDS instances, so they were unaffected by the power outage.

In a frustrating twist of fate, we had planned on moving the remainder of our web servers to AWS yesterday, but decided to push the migration back a week (as we’ve been working a lot of Saturday nights lately and we kinda wanted a break), and Linode, where we’ve been happily hosted for years, winds up having a major outage — the exact type of event we have been working hard to mitigate by moving to our new architecture.  Arrrrgh.

Having our servers auto-scaling and load-balanced across multiple data centers will significantly reduce the chances of outages in the future.  In light of last night’s events, it can’t come soon enough.  Thanks for sticking with us, and sorry again for the unexpected downtime.

 

Comments

One Comment on Back to normal after data center power outage

  1. Longstreet on Mon, 14th Apr 2014 9:18 am
  2. This is all good, Elbert. Keep up the excellent work and dedication.

Nirvana is task management software that's 100% web-based. Based upon the well known Getting Things Done® (GTD®) method of keeping organized, it is fast, easy, available from anywhere.

Spend more time doing your tasks, not managing them. Sign up now!

What is GTD®?

GTD® rests on the principle that a person needs to move tasks out of the mind by recording them externally. That way, the mind is freed from the job of remembering everything that needs to be done, and can concentrate on actually performing those tasks.

Read more about GTD® on David Allen's website »