The data center that houses Nirvana’s front-end web servers suffered a prolonged power outage last night. It took a lot longer than anyone expected (or wanted) before the thousands of VMs at their facility, including ours, were able to be brought back online. The high-level sequence of events is available here.
We’re back online now and everything is running normally.
Please accept our sincere apologies if you were unable to login this morning.
We don’t usually talk much about our infrastructure (as we figure it’s not that interesting to most GTD’ers), but given recent events I suppose this might be one of those times when people might like to know more.
Over the past few months we’ve been incrementally migrating our infrastructure to a geographically distributed and fault tolerant cloud architecture, hosted at Amazon AWS. They are truly amazing, and we are in good company.
Our databases are already running as multi-AZ replicated RDS instances, so they were unaffected by the power outage.
In a frustrating twist of fate, we had planned on moving the remainder of our web servers to AWS yesterday, but decided to push the migration back a week (as we’ve been working a lot of Saturday nights lately and we kinda wanted a break), and Linode, where we’ve been happily hosted for years, winds up having a major outage — the exact type of event we have been working hard to mitigate by moving to our new architecture. Arrrrgh.
Having our servers auto-scaling and load-balanced across multiple data centers will significantly reduce the chances of outages in the future. In light of last night’s events, it can’t come soon enough. Thanks for sticking with us, and sorry again for the unexpected downtime.
You may have read in the news this week that a major bug, nick-named heartbleed, was reported in OpenSSL, the cryptographic library used by websites to encrypt and protect information transmitted over the internet.
Nirvana servers are not affected by this vulnerability. We run an earlier version of OpenSSL that is not vulnerable to the heartbleed bug. (version 1.0.0a to be precise)
That said, if you use the same email/password combination to login to Nirvana that you use on other affected sites, it might be wise to change your Nirvana password, as those sites may have been exploited to expose your passwords in transit. This will reset/rotate your authentication tokens throughout our systems, and you should be good to go.
Some food for thought:
“The Principle of Priority states (a) you must know the difference between what is urgent and what is important, and (b) you must do what’s important first.”
― Steven Pressfield, The War of Art: Break Through the Blocks & Win Your Inner Creative Battles
Tagging actions in Nirvana with contexts, and the time / energy required, helps you quickly whittle down your available actions based on where you are, how much time, and how much energy you have throughout the day.
Having a short list of actions to chose from helps ensure that you make progress on the important things that matter “right now.”
We will be bringing our main database offline for about 30 minutes tomorrow night starting around 10PM EST (03:00 GMT). You won’t be able to login to your account during this maintenance window. However, you’ll still be able to use the web app and the iOS/Andrdoid apps just fine, provided you’re already logged in. Your data will resume syncing with the cloud (and between your devices) once we bring the database back online.
Sorry for the interruption. We’ll post updates on twitter @nirvanahq to let you know when we start and when we’re done.
We will be upgrading some of our backend systems on Saturday, February 8 between 1:00am-1:30am EST (6:00-6:30 GMT). You likely won’t notice a thing if you are already signed in to your account (on the web or via mobile devices), but you may not be able to login (create a new session) for 2 brief periods during this maintenance window.
Our apologies in advance for the forshadowed blip, but the upside is that we’ll be better prepared for growth on the flipside.