Containing the fallout from Heartbleed [UPDATED]

This afternoon I'm asking myself about the phone calls and emails I'll be getting next week. What questions will people be asking and what should I tell them?

Of course the first question will be: Is this really as serious as people say? Well, Bruce Schneier, probably the worlds most trusted authority on security, certainly thought so. A few days ago he published a blog entry in which he wrote:

"Catastrophic" is the right word. On the scale of 1 to 10, this is an 11. Half a million sites are vulnerable, including my own.

The next day Schneier blogged again on Heartbleed. This time he quoted researchers who found Heartbleed difficult to exploit.

In practice it has always been my experience that crackers are not as dilligent as we might expect - even in cases where a bug can be exploited easily. After all, if they just did the good work that they are able to do, they would prefer to have real jobs and make people smile instead of making people mad. While this is a good thing for us I must say that we handle it badly. Security is an important issue; having so many low wattage criminal minds allows too many of us to get away with being innappropriately lazy - and we pay for that when serious crackers decide to hit us.

Anyway, based on past experience, I don't really think the day-to-day operations of too many web sites will be much affected by crackers exploiting this bug. Still, the best answer to the above question is this: The publicity around the Heartbleed bug has sensitized the public to it. From that perspective, it is indeed a very serious issue. It most certainly is not the only security issue faced by many companies. However, Heartbleed may be the bug that forces your company to discuss security with your customers.

And, that, folks, turns it into an opportunity.

What is Heartbleed?

In communications software there are times when it's good to know if the party on the other end of the connection is still there or not. To perform this check a heartbeat message is sent between the two parties.

Late in 2011 a programmer working on the OpenSSL library, which is used by a vast array of software packages throughout the world, posted an update to a heartbeat function. In his code changes he forgot to include a check on a single piece of information and, unfortunately, the people who verified the code didn't notice the omission. The code was distributed throughout the world as part of OpenSSL v1.01 in early 2012. In early 2014 it was reported by a researcher that this error allowed the hearbeat code to leak chunks of information from otherwise protected memory - a very serious vulnerability. Initially called CVE-2014-0160, the bug has become popularly known as the Heartbleed bug because "it bleeds memory" out of the heartbeat function.

The OpenSSL library provides the foundation for much of the secure communications functionality that we use today. The vast majority of the servers in the world today use some variation on Linux or FreeBSD as their operating system and these systems rely on the OpenSSL library. They include popular server operating systems such as the Redhat/Fedora/CentOS distributions, the many Debian/Ubuntu distributions, along with distributions such as FreeBSD and their derivatives (including Apple OS/X.)

In addition most mobile devices also use a Linux-derived OS which depends on the OpenSSL library code. As such there are many Android & Tizen-based devices, as well as Apple iOS and QNX/Blackberry devices, which could be affected by this bug.

Fortunately, in practice, OpenSSL version 0.98 is very commonly found in many servers and devices. It seems that many organizations simply haven't gotten around to distributing version 1.01 yet. There are no reported problems with the 0.98 release of OpenSSL.

Note that the OpenSSL library is distributed under an open source license and, therefore, is often not used in Microsoft Windows - a closed-source commercial operating system. Due to the lack of source code there are fewer researchers reviewing commercial code. Of course there are researchers who disassemble the binary code and report any bugs that they find - as do crackers - and we see Windows Updates daily as a result - but the public rarely ever knows about the details of the Windows code bugs or the corresponding fixes.

What should we do about Heartbleed?

Of course this is the big question that everybody will be asking next week. The answer is that we need to respond to this issue from the two usual perspectives: Administrative and Technical.

From the Administrative Perspective, once again, Heartbleed is an opportunity to build trust between your customers and your organization. Once you proactively take technical action and report back to your customers they will recognize that you are an effective administrator that they can trust.

The technical perspective starts out as a small and surprisingly simple step: Start by patching your web server. If you host your web site on a service you will probably find that this has already been done for you. You can check to see if this has been done by using the form here:

Note: The test only takes a few seconds to complete. If it seems to be running for a long time: Try adding the port number to the end of your domain name, ie: if your domain name is you can try typing into the hostname box.

Going further

After that things get complicated. You may also have mail servers that need to be patched and, unfortunately, you probably have other equipment in your office that needs to be patched but is not easily patchable (ie: Modems and Routers, for example, not to mention all the mobile devices that are running Linux-based operating systems like Android and Tizen and, similarly, Apple iOS devices that are based on FreeBSD.)

This article is really about public services on the internet so I won't comment much about your office infrastructure. Call an expert next week and get started reviewing and responding to the issues you find in your review.

Non-profit community service organizations

Many of the people reading this note are running non-profit community service organizations. The vast majority of you:

  • have very little in the way of e-commerce functionality on your web sites.
  • do not have any need for the https:// protocol on your site,
  • occasionally do accept payments but pass those payments through a service such as Paypal, Skrill or SquareUp.

For you the technical work that is needed for your web services is being done by your Web Service Provider:

  • By now all service providers have applied all the security updates needed to remove buggy code from their web, mail and other communications servers.
  • If you do use a payment service you can be confident that they are always working hard to reduce every kind of risk that they face. They do a great job taking care of their infrastructure, keeping personal information and credit card data safe and secure.

Remember: building trust requires communication. It's easy: All you need to do, administratively, is send a note to your community:

  • Tell them about all the good stuff above;
  • tell them also that they can change their passwords, as a precaution, to protect against the possibility that a cracker may have gathered user login credentials by snooping on past sessions.
  • Remind them to keep tabs on their credit card transactions so that they will know if something goes wrong. Tell them to ask their banks and credit card companies about new email and SMS alert services that can report all transactions to them in real time (I use these services - they're fantastic for regular day-to-day issues, too.)

Don't forget to provide them with convenient contact information in case they have any questions, comments or concerns. After all: Communication can't be meaningfull unless it's two-ways between sincere and caring parties.

e-Commerce-enabled Businesses

If you have a for-profit business with e-Commerce capability you need to do all of the above - plus a bit more. To properly respond to the current opportunity:

  • Review your customer service history.

    In practice there will be few sites affected by crackers exploiting the Heartbleed bug, but, if you are hit, there may be warning signs:

    The exploits will usually result in unexplained problems that you suddenly started having at some point in the past year or so. For example: somebody placed an order through your system and had the order delivered to a new address - then the customer called to say that he never placed any order. If you find such warning signs in your service history you will know that you have been hit.

    At this point you need to seek help and advice from professionals. This is partly to ensure that you take action to protect your business and your customers, partly to ensure that you comply with reporting laws and generally demonstrate your good will.

    As you suffer through the exercise you may find it difficult to see "the Opportunity," but, trust me, from experience I assure you: you will learn, you will solidify your operations and you and your customers will benefit in the long run.

  • In some cases you will get hit but you will not see any warning signs:

    For example: If you keep customer information, with credit card numbers, in a database accessible to your web server a cracker may simply steal that info and sell it to others. In such cases your customers will stop placing orders with you as their lives collapse from the identity theft and financial calamities that they will experience - but you won't know about that. For this reason many companies simply refuse to keep any financial information about their customers on their web servers. In cases where they are forced to do so they keep their web services on a secured subnet and restrict access from the web services to the associated databases.

    Once again: seek help and advice from a security professional. The worst case is that you will go through a cycle of upgrades to your web infrastructure. In return: You will be able to proudly tell your customers that you are following best practices and properly securing their valuable data.

  • Demonstrate a willingness to exercise prudence.

    Again, in practice, there will be few sites affected by crackers exploiting the Heartbleed bug - yet we know that most e-commerce-enabled businesses are having their SSL certificates revoked. One estimate suggests as many as a half-million certificates will be revoked and re-issued next week. This is strictly because the cost of issuing a new certificate is negligable - but the cost of the fallout from customers visiting a spoofed web site ... let's just say there's no reasonable way to guess what that might be.

    What will this exercise cost? Probably little or nothing - other than the service charges associated with getting a replacement certificate and installing it on your web server.

    Certificates are really just small text files full of numbers. It takes a few seconds to calculate the numbers - that's about all. Certificates are available with different levels of Validation. Some $25 SSL certificates validate only that you paid your $25. Most Extended Validation (or EV) certificates, usually costing over $500, are issued after a verification of company registration papers and Yellow Pages telephone listings. At the high end of the scale there are EV certificates that sell for over $20k. When you order one of those you get a visit from a technical team that reviews your business processes and your operations.

    So the actual cost of revoking an existing certificate and issuing another will depend on the "Validation" it offers. For most companies the CA is not actually validating anything - the most basic certs serve only to secure the communications between the customer and the web server - so there are no serious legal issues to consider. Some Certificate Authorities might have some service charges that they apply. Many CA's will simply ask you to renew your certificate, in advance, for another year. This will get them some cash for their efforts while costing you, effectively, nothing at all (for the certificate.)

    In the rare cases involving more extensive validations the CA will probably try to make some kind of accomodation for the Heartbleed bug.


Remember: This is an opportunity for those who have the right attitude.

And, if you don't have the right attitude, now is a good time to get it.