Let's say you're completely on top of your SSL configuration: you're using strong ciphers, disabling obsolete protocols, and in particular, you're serving a certificate chain that's 100% SHA-2. SSL Labs gives you an A+ and shaaaaaaaaaaaaa.com shows no trace of SHA-1. Yet, for some reason, Chrome is displaying a red cross in your URL bar when you visit your site, suggesting your site is "affirmatively insecure" for serving a SHA-1 certificate:

Could this be possible? Unfortunately so. The certificates that your server sends might not be the certificates that your browser uses. This shouldn't be an issue when migrating to SHA-2, but due to bad practices by some certificate authorities and users running out-of-date software, it sometimes is. Read on to learn more.

Background: how certificate chains work

An SSL certificate must be signed by a certificate authority (CA) to be trusted. In the simplest case, a website's certificate (the "end-entity" certificate) is signed directly by a CA that is trusted by web browsers (a so-called "root" certificate).

It's straightforward for a web browser to validate a certificate that's signed directly by a root: the browser looks up the end-entity certificate's issuer in its root certificate store, and if found, it validates the signature of the end-entity certificate using the public key of the root certificate.

However, end-entity certificates are rarely signed directly by a root certificate. Instead, end-entity certificates are signed by an "intermediate" certificate (sometimes called a "subordinate CA"), which is in turn signed by a root:

Validating a certificate signed by an intermediate is tricky. A browser can't simply look up the issuer in its root store, since the certificate isn't signed by a root. Instead, the browser must find the intermediate certificate that signed the certificate, recurring as necessary (intermediates may be signed by other intermediates) until it has constructed a chain of certificates that ends at a root certificate. This is actually rather complicated, because there may be more than one possible chain from an end-entity certificate to a root.

The most basic way that browsers discover an intermediate certificate chain is by having the server tell them. This is why, when you deploy an SSL certificate, you have to configure not only your own certificate, but also the intermediate certificates. However, browsers often cache intermediate certificates, and might use a cached certificate instead of the one offered by the server. This is why Chrome might show a red cross even when the chain you serve is entirely SHA-2: it's not using your chain, but a cached chain which uses SHA-1.

In theory, this is avoidable, if CAs follow best practices and users run up-to-date software. Unfortunately, this isn't always the case, and there are two situations in particular that are known to cause issues.

(Note: to be precise, it's not Chrome that generates the chain, but the operating system's cryptographic library, to which Chrome outsources chain building. On Windows, it's CryptoAPI, and on Linux, NSS.)

Issue 1: Reusing a SHA-1 intermediate for SHA-2

When a certificate authority transitions to SHA-2, it can either reuse its existing intermediate certificates by re-signing the existing public keys with SHA-2 signatures, or it can generate brand new intermediate certificates with new public keys and subject names.

The first way is wrong. Although a CA can take an existing intermediate and re-sign it with SHA-2, browsers might have the intermediate cached with the old SHA-1 signature. As explained above, browsers can ignore the chain presented by the server, so even if the server sends the intermediate with the new SHA-2 signature, the client may construct the chain using the cached SHA-1 intermediate certificate instead. This is exactly what CryptoAPI does.

The second way avoids all problems with cached chains. Since the CA generates a brand new intermediate, with a new name and public key, browsers can't possibly have old versions cached with SHA-1 signatures. This is why Ballot 118 from the CA/Browser Forum states:

SHA-2 Subscriber certificates SHOULD NOT chain up to a SHA-1 Subordinate CA Certificate.

Unfortunately, some CAs ignored this advice at first, and one CA, StartCom, is still issuing SHA-2 end-entity certificates that are signed by a SHA-1 intermediate. While they provide a version of the intermediate signed with SHA-2, it does no good if a browser already has the SHA-1 version cached.

Fortunately, both of SSLMate's CAs correctly issue certificates from a brand new SHA-2 intermediate, so SSLMate customers need not worry about this issue.

Issue 2: Out-of-date NSS and cross-signed roots

Sometimes root certificates are themselves signed by other roots, a practice known as cross-signing. By providing an alternative path to a trusted root, cross-signing allows new roots to work in browsers that don't yet have the root in their trust store. Once browsers start trusting a root, the cross-signed certificate is no longer necessary. However, as with other intermediate certificates, cross-signed certificates can be cached, and a bug in NSS caused Chrome on Linux to use cached cross-signed roots even when a shorter and newer chain existed. If the cached cross-signed certificate happened to use SHA-1, Chrome would consider the chain weak and display the red cross, even if the chain sent by the server was entirely SHA-2.

This bug has been fixed as of NSS 3.17.4, released on January 28. Unfortunately, Debian has been extremely sluggish in shipping an updated NSS package. A bug was filed in Debian's bug tracker on December 30, 2014, but wasn't fixed in Debian Unstable until May 13. Meanwhile, Debian Stable (Jessie) continues to ship NSS 3.17.2. The Debian Security Team has ruled out fixing this through a security update, and it doesn't look like the package maintainer will respond quickly enough to get the fix into the upcoming point release of Debian Stable. Ubuntu, by contrast, treated this as a security issue and released updated packages for all their distributions on February 19.

Unfortunately, there's nothing certificate authorities can do about users with out-of-date versions of NSS. Until Debian releases an updated NSS package for their stable distribution, users of Chrome on Debian are going to see a lot of this:

Or, if the certificate expires in 2016:

Update: An updated NSS package, prepared by Andrew Ayer from SSLMate, shipped with the Debian Jessie 8.2 point release on September 5th, 2015!