Harmony Website Management Blog

HTTP, HTTPS, and Google Chrome warnings

2017-09-08T09:52:22-04:00

Google has begun sending emails to website administrators explaining an upcoming change to the Google Chrome browser. Chrome will start showing a “NOT SECURE” message on any web page hosted under HTTP that includes form elements like text fields. The browser currently shows this message for pages that have login forms or credit card information, but that logic is soon being expanded to encompass any web page that allows users to enter in and submit information of any sort (e.g. a Contact Us form or commenting on a blog post).

The email from Google looks like this:

It is true that HTTP sites are not secure. When submitting information to a page hosted under HTTP, it is possible for a malicious actor to see what was submitted. When a site is hosted under HTTPS, this kind of attack is far more difficult to perform. Further more, Google and other search engines have started taking into account HTTPS hosting when determining site rankings, preferring sites that are secure over those that may not be.

Now, Harmony does not itself support HTTPS hosting of customer sites for various reasons, but that doesn’t mean HTTPS is not available. There are services available that will sit in front of your Harmony site and provide HTTPS for you. We recommended the service CloudFlare, setting it up is easy and free. It should be noted that setting up a CloudFlare account requires changing your domain’s nameservers to CloudFlare, which may not work for some people.

For those who want to set up a CloudFlare account, after you sign up you’ll see a page similar to this one:

Follow the 4 step setup process as instructed by CloudFlare. Once you’ve completed all four steps and switched your domain’s nameservers your site will start using HTTPS in about 24 hours.

For those sites where CloudFlare will not work and you would like to get your site under HTTPS, please contact us at harmony@collectiveidea.com. While it is true that this version of Harmony does not have built-in HTTPS support, we’ve been working on a version that does!

February 28 Incident Post-Mortem

2017-03-01T10:54:01-05:00

Around 1pm Eastern on February 28, 2017, our monitoring tools notified us that all of Harmony, both the Administration Portal and Live Sites, was erroring and inaccessible. It took us around 20 minutes to track down the underlying issue and another 10 minutes to put together and deploy partial mitigation. The Administration Portal and Live Sites were partially back up after a 30 minute downtime, with assets and images disabled across all of Harmony. We were able to re-enable assets and images by 4:30pm Eastern, after which Harmony was completely functional once again.

Specifically, the reason Harmony went offline is because Amazon S3, where we store all Images and Files, had an outage such that any request for files would never receive a response. This led to all of our web servers to fill up waiting for these requests, bringing our ability to handle any traffic to zero. Once we realized where the error lay, we were able to deploy a change to Harmony that let us disable requests to S3 entirely, letting the application return to mostly normal operation.

Due to the wide-spread nature of the outage of Amazon S3, we quickly realized that our normal tool for customer communication, Intercom, was also affected by the outage and we found ourselves unable to notify our customers of the outage and the reason for image-less websites. Also, some customers noticed that our status page which was hosted by Pingdom erroneously said that Harmony was fully live instead of in a state of partial availability. We also then realized that we could not add other notices and information to this status board to inform users of our current status.

The outage of Amazon S3 was an access-only outage and there was no threat to any of your site’s content. However, seeing your site disappear with an error page, and then function without images, with no communication from us, is not a good customer experience. We apologize for not having adequate communication channels in place for this outage.

While there is little we can do if Amazon S3 has another similar outage, we should be able to communicate more effectively the current state of Harmony. In that vein we are going to do the following:

Move our status page to StatusPage.io which will let us add incident information and provide better communication to customers during an outage.
Review our other communication and informational tools and make changes to those tools which will let us get the word out more quickly.
Set up further monitoring such that we can catch a full outage of this nature quicker and make sure Harmony and customer live sites stay running.

Again we apologize for the lack of communication during this incident. We understand how frustrating it can be when your sites are not functional and you don’t know why, and we are working to make sure this doesn’t happen again.

Improved Account Security

2014-06-23T10:17:13-04:00

Harmony takes the security of your accounts and websites seriously. Since the introduction of our Security Hall of Fame last year we have made numerous improvements to ensure Harmony and its data are as secure as possible. Most of these changes are invisible to our users but today we’re deploying a new Account Security page, accessible via the “Security” link in your “Profile and Accounts” page:

This page shows all devices (computers and mobile) that are currently logged into your account, letting you log out any sessions you don’t recognize. If you see any sessions that look unfamiliar to you, click “Revoke” to log them out.

Harmony and Heartbleed

2014-04-15T11:49:52-04:00

On April 7th, security researchers publicly disclosed a serious security vulnerability now known as Heartbleed. This vulnerability affected a vast majority of the Internet and made it possible for attackers to potentially gain sensitive information from sites protected by SSL.

Unfortunately Harmony was vulnerable to Heartbleed. We immediately took steps to fix this issue, including resetting User sessions and replacing our SSL certificate. Harmony is no longer vulnerable, and through our investigations we have not seen any evidence that any information was stolen. That said, as a precaution we do recommend changing your account password, not only on Harmony but on other sites you visit across the Internet.

We would also like to recommend, if you aren’t already, using one of the many password management tools available today, such as 1Password, Keeper, PasswordBox, LastPass, or KeePass.

Latest Updates and Site Exports

2013-05-21T11:18:51-04:00

We have been hard at work making continuous changes and improvements to Harmony, and today we’re announcing two big updates: Site Exports and an improved Theme Editor.

Site Exports

It’s now possible to get a fully static HTML export of your site! On your site’s main Dashboard you’ll see an “Export” option under “Site Administration”:

This link will bring you to the Export Management page, where you can trigger an export of your site. Harmony will email you when the export is complete and a download link will show up on the Management page. Harmony keeps the five most recent exports.

Improved Theme Editor

We have also rewritten our theme editor. Now using the fantastic Ace editor, theme file editing should be faster, especially with large files, and also customizable. By clicking the gear at the top right of the page, you can change the color scheme, the font, and other options:

We have also updated our WYSIWIG content editor and made a number of performance improvements both in the site and in our architecture. As always, we love hearing your suggestions for making Harmony awesome, so leave a comment or contact Support with your best ideas.

Server Move Progress

2013-01-28T09:00:22-05:00

We’re excited to report that our transition to new servers is going smoothly. The new servers are fast and stable, using only a fraction of their capacity. We’ve had no outages during the move, and fixed a handful of small bugs in the process.

We still have around 100 sites that need to move DNS, and we’ll be contacting you again soon. We’re very pleased at how quickly many of you made the transition. We know DNS can be confusing, so don’t hesitate to reach out if you have any questions. We’re much more familiar with the admin panels of most popular DNS providers now, and are happy to help.

Throughout the process, we’ve talked to a number of customers about new features they’d like to see. We won’t push anything big out until after the move, but we’re always open to good ideas. Let us know what you’d like to see us work on next!

IPv6 for your site

2013-01-18T16:53:40-05:00

Did you know Harmony is available over IPv6?

If you’re lucky enough to have IPv6 connectivity (our office still isn’t…come on Comcast Business), then you’re connecting to Harmony over IPv6! For the rest of us, this doesn’t make any difference, but we’re ready for the future, and your sites can be too.

To enable IPv6 for your site, use our recommended domain setup which uses a CNAME DNS record instead of A Records. A Records are limited to IPv4, and while AAAA Records exist for IPv6, a single CNAME record takes care of both IP versions in one go.

Get out there and enjoy IPv6 on Harmony!

Moving to New Servers

2013-01-14T18:25:08-05:00

We’re ready to move fully to our new infrastructure. We’ve been moving sites over and are running about 20% of our traffic on the new setup. We’re extremely happy with how it has been going, and want to share it with everyone.

Depending on your DNS configuration, you may need to change your DNS settings but even if you do, we have set it up for a zero-downtime, seamless transition.

Quick Summary

Today we’re running both new and old servers concurrently. Both use the same database underneath, so there are no data consistency issues. We recommend changing your DNS with the instructions below as soon as you can.

February 1, we will switch the old server into “proxy mode” meaning any remaining traffic will be proxied to our new servers. If you haven’t updated your DNS by this date, your site will continue to work.

March 1, we will switch off our old infrastructure. Expect lots of emails from us if you haven’t switched your DNS by this date.

The Details

This week, we are switching DNS for *.harmonyapp.com to point to the new servers. We already have about half of this traffic pointing to the new servers, so you’re possibly reading this off the new setup.

simplified view of our servers today

If you’re using our recommended setup of pointing your www (or another) subdomain to harmonyapp.com, then you are already set! If you have your “apex” or “root” domain (example.com) pointed to our old IP address (216.180.248.111) then you need to update your DNS before March 1.

Instructions for setting your DNS can be found, as always, in our documentation. We recommend using CNAME records instead of A Records. CNAME records allow us to take proactive measures to keep your site up, running, and fast in adverse conditions. If you do need A Records, we currently have two static IP addresses, so you’ll need 2 A Records instead of 1: 54.235.208.208 and 54.235.208.220.

simplified view of our servers after February 1

Both server setups point to the same database, so the transition will be seamless. You can change your DNS at any time, and there will be no downtime or data inconsistency as your visitors transition. We’ve worked hard to make this as easy as possible.

When should I change my DNS?

We recommending you change your DNS today. Our new servers are faster, and we have more tools to prevent downtime. On February 1, we will change our old servers from serving pages to “proxy mode” which will simply proxy traffic to our new servers. If you haven’t moved your DNS your site will work, but may be slightly slowed as it grabs the data from the new servers. We plan to turn off the proxy on March 1.

If your DNS needs to be changed, expect an email from us (we know everyone doesn’t read this blog). As we get closer to both the February 1 and March 1 deadlines, you’ll get more and more. We’ll make sure everyone gets moved over.

We know changing DNS can be a pain, which is why we take this very seriously. We won’t ask you to make changes unless we have a good reason. Our new infrastructure is truly a step forward for Harmony, and we’re excited for you to join us!

New Infrastructure for 2013

2013-01-02T16:51:56-05:00

As we roll into a new year, we’re excited to also roll out our new infrastructure.

Our Pingdom-reported uptime was 99.89% for December but we know that’s not near good enough. Here’s what we have planned with a bit of background.

Harmony in 2012

We’ve long been limited by one non-redundant server that controls most of Harmony. If this server had any hiccups, we felt it across all of Harmony. We were also limited in the maintenance we could perform.

In August, we added a load balancer to give us more flexibility. This let us quickly re-route traffic to a slave server when our primary server had hiccups. While we still didn’t have an ideal setup, this was a definite improvement.

In early December, we moved our MongoDB database from a single server with a slave into a proper replica set. This not only allowed us faster disaster recovery, it also removed a major barrier in moving to a new server setup.

2013: Redundancy, Uptime, and Upgrades

Our new infrastructure brings us a new level of redundancy, allowing us, both automatically and manually, to re-route traffic around problems and perform maintenance without affecting customer sites. It also allows us to add capacity at any time, giving us much more flexibility in dealing with unexpected surges in traffic (like a DDoS attack).

We’re currently testing out our new production servers. This week, we’ll be moving over a few of our own sites as well as a few customer volunteers. When we move everyone over (we’ll announce before we do) you won’t notice any downtime and should notice faster response times.

Under the hood, we’ve been able to make other improvements including upgrades to Ruby 1.9.3 and Rails 3.2, as well as making IPv6 available for all sites.

What’s next?

New hardware means we’ll be comfortable pushing the bounds of Harmony (and not worried about servers going down). For example, we can better separate admin and content code. Or we could route all API traffic to certain servers, keeping our main ones clear for page traffic.

Finally, we’ve added checks for the new infrastructure to our Status Page. We’re striving for zero downtime, and we want to be as transparent as possible.

Want early access?

If you have a live site you’d like to get on the new infrastructure, use the Feedback link in the admin to volunteer!

Update on our Maintenance

2012-12-06T11:37:30-05:00

We’re happy to report that we came through today’s planned maintenance window with zero downtime for live sites. We did go over our hour window a bit while still in “read-only mode” for the admin, but we’re happy with the results overall.

Today’s work centered around moving our database to a more robust infrastructure, making it easier for us to do maintenance in the future, reducing unplanned downtime, and allowing us to recover faster in the event of a catastrophic failure.

Keeping your sites available is always our primary concern. As we continue to grow and improve Harmony, we want to be able to continue to do maintenance without experiencing downtime. I know that some of you were concerned about our “1 hour window” but we prefer to be very transparent about when we perform maintenance and what to expect as a worst-case. We’d love to hear your thoughts about how we communicated this window, and how we should in the future.

Harmony Maintenance: Thursday Dec 6

2012-12-04T14:56:14-05:00

We will be performing maintenance on Harmony’s infrastructure this Thursday December 6th from 8 AM to 9 AM EST (check your time zone). We expect a maximum of 10-15 minutes of downtime, but it may be spread throughout this 1 hour window.

Consequently, Harmony and some of your hosted websites may be periodically unavailable during this time. We will be in read-only mode, meaning access to the admin, blog comments, form posts, and other edits will be unavailable. Most pages will be served from our cache layer and still be available.

This maintenance is part of some significant upgrades which will make Harmony better for everyone. We hope that you don’t even notice our maintenance, but we want to be transparent and announce our window.

Harmony & Integrations: Sitting in a Tree

2012-11-19T10:12:38-05:00

Looking for a wedding website? We’ve been working with a local startup to build their wedding website platform atop Harmony.

Sitting in a Tree is building beautiful wedding websites, and we’re giving brides & grooms a great interface in Harmony. You simply sign up with Sitting in a Tree, and you get a site generated for you in Harmony.

To make it work, we had to do a few things on the Harmony side that we had been wanting to add anyway. Specifically we built the beginnings of our public API. Working with the Sitting in a Tree team has helped us walk through features that our API needs, and let us a create a platform to build on.

We’re not ready to open our API up to the world quite yet, but it is one of our goals. In the meantime, if you’d be interested in doing a custom integration of your own, let us know!

Fighting a DDoS Attack: A post-mortem

2012-08-31T15:51:27-04:00

We previously discussed our experience with downtime due to a DDoS attack, but since then it got worse, and we got better at fighting it.

The Attack

At the end of July, we starting getting hit by a distributed denial of service (DDoS) attack. The attackers are going after domains with a certain term in them. For example, pretend the term was “bird” and they were attacking birdwatchers.net and houses-for-birds.biz. The attack wasn’t targeted directly at Harmony or our clients, but our clients were instead part of a larger, broader attack.

It was a true DDoS, coming from more than a million unique IP addresses.

The traffic was a mix of legitimate-looking page requests, and obvious bad requests. The bad requests mostly took the form of HTTP POST requests where they shouldn’t be posting to, and GET requests with a strange query string.

Mitigation Strategy 1: Short-circuit bad requests

Harmony is built to scale up well; we regularly have sites featured on Reddit or Hacker News and it doesn’t stress our infrastructure at all. However, the attack was using cache-busting strategies, causing them to hit a slower code path in our app instead of our cache layer.

Our first response was to short-circuit invalid POSTs. We changed our routing layer to recognize the few types of POST that we actually support, and turned all other requests into HTTP errors.

This worked for a while and got Harmony back up, but the attack ramped up in the following weeks.

Mitigation Strategy 2: Moar¹ Hardware!

To deal with increased load, we added hardware. First it was simply beefing up our existing boxes, and that kept us ahead of the attack for a bit longer.

On August 17, we turned to a new load balancer to give us more flexibility. We did encounter some problems during the installation, and our maintenance window ballooned to over an hour. The downtime was extremely frustrating, but a necessary addition to give us more control over how requests are handled.

Mitigation Strategy 3: Request Filtering

Blue are requests received by the load balancer and orange are requests that make it to the app servers.

With our new load balancer we can now identify potential attackers, IP spoofing, and illegitimate requests, and block them before they get near our app servers. We’ve been really happy with the results, and we have a lot more room to respond to future attacks. Though we continue to see spikes on the load balancer, our application servers are back to their normal level of traffic.

Other downtime

On August 28, we experienced 12 minutes of downtime due to an upstream DDoS attack related to DNS. This attack wasn’t directed at us, but we felt its effects.

In addition, we’ve had blips of downtime due to maintenance and load testing, but not due to attacks. We are working on our infrastructure to eliminate downtime that we cause, and continuing to stay ahead of attacks.

We’ve been using our status page as a way to be transparent about our downtime. We want to be open about our failures, as we work to eliminate them.

Moving Forward

We really hate even a minute of down time and we know our customers hate it as much as we do. We’ve learned a lot over this last month about what bottlenecks Harmony faces and we are working hard to smooth them out. We have a number of infrastructure improvements planned for the coming months. Expect to hear more about them soon.

¹ I gave a talk on fighting this attack at BarCampGR last week, and the first question was, “Why did you spell ‘moar’ that way?”

Planned Maintenance - August 17

2012-08-16T16:32:17-04:00

We’re planning a maintenance window of up to 15 minutes at 8:00 AM Eastern (13:00 BST, 12:00 UTC) Friday August 17.

We will have to take the app down briefly, but expect a maximum 10 minutes of downtime.

We apologize for any downtime, but it is necessary to install a new load balancer. We’ve been fighting a DDoS attack and the new load balancer will give us many more tools to fight these attacks in the future.

Thanks for your understanding, and we are looking forward to continuing to increase speeds!

Downtime

2012-07-28T11:57:32-04:00

We’ve had our worst week yet for uptime in Harmony. We’ve long prided ourselves on our stability, but we failed this week. Here’s what happened.

First, we’ve been seeing a large increase in traffic over the last month. While many new sites are launching, and older sites are gaining in popularity, we can scale with growth easily. This was an increase in malicious spam traffic. We’ve had a few times where requests (mostly admin pages) would time out, or hang, while a burst of traffic was being dealt with.

On Monday, we got hit unexpectedly with DNS errors. Our DNS provider, Zerigo, was affected by a massive denial of service attack and we got caught realizing that 5 DNS hosts isn’t enough if they’re all under attack. At the same time, many of you use Zerigo for your own DNS, compounding the problem.

Update: Zerigo has posted their write-up, explaining the outage.

After Zerigo recovered, we setup redundant DNS so that we won’t get caught in this situation again.

For the rest of the week, we continued to see an uptick in traffic. We asked our hosting company to bump up our resources yesterday, and planned on a quick and easy upgrade today. Before they even got into the office to do an upgrade this morning, the traffic spiked again, and a simple reboot wasn’t enough to restore service. The fine crew at RailsMachine responded by upgrading right then, and we were back to normal.

Usual requests to Harmony pages scale very well because we heavily cache responses. These spam requests were mostly HTTP POSTs, which were able to use more resources than a GET request that hits our cache layer. We worked through a careful fix and finally deployed it this afternoon.

This graph shows the instant relief we saw after deploying, represented by the blue bar.

We’re not pleased that we had any downtime. You depend on Harmony and we let you down.

As a first step, we’ve launched a new public status page, powered by Pingdom, at status.harmonyapp.com

Eek, it doesn’t look very good today, but you can click into the details and see that we had 33 minutes of downtime on Live Sites today. Not good at all.

We also have a lot of monitoring tools in place, so we can respond within minutes. Our hope is to get our status page back to green. We also may iterate on our status to pull in data from other services, but we wanted to start being more transparent today.

We’re extremely sorry. We hope our new measures will keep us up and running, and transparency will help keep you informed. If you have any questions or concerns, don’t hesitate to contact us.