Harmony Downtime
Post Information
Posted on October 19, 2011
By John Nunemaker
On Monday and Wednesday we experienced two outages due to emergency hardware issues with the primary server Harmony lives on.
The server issues this week were complicated due to our current database setup. For the geeks among us, we want to explain what prolonged the outages.
When journaling is not enabled, MongoDB recommends doing a repair when starting the server after an unclean shutdown. This repair takes a lot of time on a database the size of Harmony’s and was a large percentage of the time on both outages.
Following today’s outage we immediately enabled journaling to prevent this issue in the future.
We are still working with our hosting provider to understand the hardware issues that caused the failures, and we are already working on plans to ensure that Harmony is highly available, even through hardware failures in the future.
Uptime is one of our biggest concerns. We believe a web presence is one of the most important aspects of modern business and life. We built Harmony to make it easier to create and maintain that web presence, so it deeply pains us when we fall short of ensuring your site is available.
Thank you for your patience and encouragement.