In the coming weeks, Mozilla and Chrome will remove trust in the Entrust certificate authority due to a pattern of compliance issues. To minimize disruption, only newly-issued certificates will be distrusted: Chrome will reject Entrust certificates issued after November 11, 2024, and Firefox will reject Entrust certificates issued after November 30, 2024. Previously-issued certificates are supposed to continue working, relieving server operators of the burden of replacing already-deployed certificates.

However, the Mozilla trust store is used not only by Firefox, but by numerous non-browser TLS clients in the Linux and open source ecosystems. A survey of prominent non-browser consumers of the Mozilla trust store reveals that the vast majority will not handle the upcoming distrust correctly - most will fail open, continuing to accept newly-issued Entrust certificates, but some will fail closed, rejecting previously-issued Entrust certificates, potentially as early as November 30. Consequentially, the Entrust distrust may be more disruptive than intended, and site operators using Entrust certificates should consider replacing their certificates prior to November 30 to avoid issues with non-browser clients.

Technical Background

The list of root certificates trusted by Mozilla is stored in the mozilla-central repository in a text file named certdata.txt. Roots in this file can be tagged with attributes which restrict how the root is used. One such attribute, CKA_NSS_SERVER_DISTRUST_AFTER, specifies the latest allowable issuance date for certificates issued under the root. The list of roots and their attributes is shipped with Firefox, and the Firefox certificate validator rejects certificates whose Not Before date is after the root's Distrust After date. In the near future, Mozilla will tag Entrust's roots with a Distrust After date of November 30, 2024. The maximum certificate validity period is 398 days, so on January 2, 2026 (398 days later), all Entrust certificates issued before November 30, 2024 will be expired, allowing Mozilla to remove the Entrust roots entirely from certdata.txt.

Most open source TLS clients do not ship with a list of roots like Firefox. Instead, the operator of the software is typically required to provide a PEM-encoded root certificate bundle. There are several places for an operator to get this bundle:

• Operating system packages, typically called ca-certificates, install a bundle to a quasi-standard location like /etc/ssl/certs/ca-certificates.crt or /etc/pki/tls/certs/ca-bundle.crt, which TLS clients load automatically.

• Curl provides a bundle for download. This bundle is shipped in libwww-perl's Mozilla::CA module.

• The Certifi project provides bundles via libraries for several popular languages.

• mkcert.org (used by the Certifi project) provides bundles via an HTTPS API endpoint.

• The golang.org/x/crypto/x509roots/fallback Go module embeds a bundle that is used if the operating system does not provide a bundle.

Ultimately, all of these bundles are generated by parsing Mozilla's certdata.txt. Unfortunately, there is no standard way to include attributes like Distrust After in PEM bundles. What should bundle providers do with roots which have this attribute?

Option 1: Ignore Attribute and Include the Root

The following bundle providers simply ignore the attribute and include the root in the bundle:

• Debian and Ubuntu's ca-certificates package, which is also used as the source of Gentoo's ca-certificates package.

• Fedora's ca-certificates package, which is also used as the source of Arch's ca-certificates package.

• Slackware's ca-certificates package, which was forked from Debian's ca-certificates package in 2018.

If you use a bundle from one of the above providers, your TLS client will accept Entrust certificates issued after November 30.

Option 2: Exclude the Root

On the other hand, the following bundle providers exclude roots with Distrust After dates, if the date on which the bundle is generated is after the Distrust After date:

• Curl's bundle (Update: since publication, curl's bundle has been updated to include these roots!) (which is used by Alpine's ca-certificates package and libwww-perl's Mozilla:CA module)

• Certifi/mkcert.org (Update: since publication, a patch has been submitted to mkcert, and Certifi is blocking new releases until the patch is applied!)

• FreeBSD's ca_root_nss

The following bundle providers exclude roots with Distrust After dates, regardless of when the bundle is generated:

• Go's golang.org/x/crypto/x509roots/fallback module. (An earlier version of this post incorrectly stated that Go would include roots with Distrust After dates; I unfortunately had not reviewed all of the relevant code.)

Consequentially, if you use a bundle from one of these providers, then your TLS client will reject all Entrust certificates - even those issued before November 30, which are supposed to keep working.

Unfortunately, most information about the Entrust distrust doesn't mention this problem, or even discuss non-browser clients at all, so server operators who need to support non-browser clients aren't aware of the need to replace their certificates before November 30.

P11-Kit Persistence Files

Fedora/Arch's ca-certificates package also ships a root bundle encoded in the p11-kit persistence format. This format supports attributes, and the ca-certificates package includes the CKA_NSS_SERVER_DISTRUST_AFTER attribute in the persistence file under an attribute named nss-server-distrust-after. However, it's unclear what software currently makes use of this. GnuTLS recently gained support for properly enforcing Distrust After, but GnuTLS is not commonly used compared to OpenSSL. Indeed, I tested curl on Fedora and Arch, and on both platforms the attribute-less PEM bundle was used.

What Should Server Operators Do?

If you're using Entrust certificates, particularly on non-HTTP endpoints such as mail servers, or on API endpoints that are likely to be accessed by non-browser clients, you should replace your certificates with non-Entrust certificates prior to November 30, 2024.

What Should Root Bundle Providers Do?

Bundle providers should provide p11-kit persistence files with Distrust After attributes, so that TLS clients like GnuTLS can take advantage. This shouldn't be too hard, since the p11-kit persistence format is just PEM with some Explanatory Text between the CERTIFICATE blocks. In theory, existing PEM parsers should be able to parse p11-kit persistence files, though for whatever reason Fedora chose to ship separate files for each format in their ca-certificates package.

As for the PEM bundles, I think it's preferable to simply ignore the Distrust After date and include the roots. While this may be seen as the less secure option compared to omitting the roots, in practice the security value of Distrust After is virtually nonexistent, since CAs can trivially circumvent it by backdating the certificate's Not Before time. Although Mozilla threatens to fully distrust CAs caught doing this, I view the threat as pretty meaningless. Since Firefox doesn't require certificates (or their corresponding precertificates) to be logged in Certificate Transparency logs (yet), it's wishful thinking to believe backdated certificates would be discovered - the CA would simply not log the precertificates for backdated certificates. If an unlogged certificate did make its way into Certificate Transparency logs, it would be difficult to prove it was backdated, as it could have plausibly been logged by a third party long after it was issued.

Rather than provide security, the point of Distrust After is to gracefully sunset a CA that is not immediately dangerous but which has fallen short of the high standards expected of CAs. As long as the certificate validator rejects certificates that are valid for more than 398 days, the CA won't be able to set the Not After (expiration) date of a backdated certificate more than 398 days after the Distrust After date. This ensures that all certificates issued by the CA will eventually expire, allowing the root to be fully removed without breaking any sites.

I think this goal would have been much better accomplished by restricting the Not After date instead of the Not Before date. For example, instead of rejecting Entrust certificates issued after November 30, 2024, Firefox should reject Entrust certificates expiring after January 2, 2026 (398 days later, the same date on which it will be safe to fully remove Entrust roots). Doing it this way would make the security properties extremely clear, and it would be harder to misinterpret as curl and Certifi have.

I have opened bugs with curl (since resolved), with mkcert.org, and with Go about this issue. But even if they are fixed, there may be other certdata.txt consumers who have misinterpreted Distrust After who will begin rejecting all Entrust certificates beginning November 30, so it would still be prudent for server operators to replace their Entrust certificates.

Certificate Transparency: A Brief Overview

Certificate Transparency (CT) is a system for publishing SSL certificates in public logs. Certificate Transparency monitors like Cert Spotter download the logs to help you track certificate expiration and detect unauthorized certificates.

At a high level, Certificate Transparency works as follows:

1. Before issuing a certificate, the certificate authority (CA) creates a "precertificate" containing the details of the certificate it intends to issue.
2. The CA submits the precertificate to multiple Certificate Transparency logs.
3. Each log returns a receipt, called a Signed Certificate Timestamp (SCT), which confirms submission of the precertificate.
4. The CA embeds the SCTs in the certificate which it gives to the site operator.
5. When a browser loads a website, it makes sure the website's certificate has SCTs from a sufficient number of recognized logs. If it doesn't, the browser throws up an error page (ERR_CERTIFICATE_TRANSPARENCY_REQUIRED in Chrome) and refuses to load the website.

Note that only the precertificate needs to be submitted to logs; the certificate itself may or may not end up in logs. Some CAs submit them automatically, and crawlers such as the Googlebot submit certificates that they observe when crawling the Internet.

What Went Wrong

The incident began last Wednesday, March 15 at 15:13 UTC, when Sectigo, a CT log operator, began a migration of their "Sabre" log to newer software. There was about an hour and a half of planned downtime. At 16:30 UTC the log came back online. The migration was mostly successful, except for one rather significant problem: the new log software had been configured with the private key of a test log called "Dodo". The problem wasn't noticed and corrected until the next day, March 16, at 12:28 UTC. During this 20 hour window, all of the artifacts produced by Sabre, including the SCTs returned to CAs, were signed with Dodo's private key.

In theory, this should not have been a problem - CAs could have used Sabre's public key to validate the SCT signatures, rejected the ones with invalid signatures, and gotten SCTs from other logs.

Unfortunately, history shows that certificate authorities are prone to sloppiness and shortcut-taking. There was no doubt that at least some CAs would blindly take SCTs and embed them in certificates without validating the signature. Browsers, of course, do validate SCT signatures, and reject SCTs signed by Dodo, since it is a test log.

The only question was how many CAs were affected. To answer this question, SSLMate examined the approximately 164,000 precertificates which were logged to Sabre during the incident. For every precertificate, we looked for its corresponding certificate (in any log, not just Sabre), and evaluated the certificate's embedded SCTs for compliance with browser Certificate Transparency policies. Among the 35 distinct CAs logging to Sabre during the incident, we found:

• 12 CAs which had at least one non-compliant certificate, indicating that the CA does not validate SCT signatures: Amazon Trust Services (operated by DigiCert), Buypass, CFCA, certSIGN, Certum, DigiCert, Globaltrust, Microsoft, NAVER Cloud, Netlock, SwissSign, Viking Cloud
• 6 CAs with only complaint certificates, indicating that they correctly rejected SCTs from Dodo: Certainly, Chunghwa Telecom, Cybertrust Japan, IdenTrust, Let's Encrypt, Sectigo
• 17 CAs for which we could not find any certificates, making it indeterminate whether or not they validate SCT signatures: Atos Trustcenter, Certigna, D-Trust, Deutsche Telekom Security, Firmaprofesional, Google Trust Services, Government of Spain - ACCV, Government of Spain - FNMT, HARICA, Izenpe, Microsec, OISTE, QuoVadis, SSL.com, Shanghai Electronic Certification Authority, Taiwan-CA, Telia

Below are the total number of precertificates that were logged to Sabre during the incident by CAs which do not validate SCT signatures. Presumably all of the corresponding certificates contain Dodo SCTs that will be rejected by browsers:

Amazon Trust Services (operated by DigiCert)	42,363
Buypass	1
CFCA	19
certSIGN	2
Certum	878
DigiCert	27,367
Globaltrust	1
Microsoft	83,024
NAVER Cloud	1
Netlock	5
SwissSign	4
Viking Cloud	29
Total	153,694

Cloudflare for the Win

As a content distribution network, Cloudflare automatically obtains certificates for its customers' websites. Despite getting some of its certificates from DigiCert, Cloudflare-hosted websites were not affected by the incident. That's because Cloudflare evaluates every certificate for Certificate Transparency policy compliance, and if a certificate lacks a sufficient number of SCTs, Cloudflare submits the certificate to additional logs and serves the resulting SCTs via the TLS handshake.

We found several examples of Cloudflare-hosted websites that were serving a certificate with insufficient SCTs yet still worked fine thanks to the additional SCTs in the TLS handshake. Here's an example of the SCTs served by one such website:

Mechanism	Log	Signature	Log Status		Timestamp
			Apple	Chrome
Embedded	Google Argon 2024	Authentic	Usable	Usable	2023-03-16T00:32:03.504Z
Embedded	Sectigo Dodo	Authentic	Unrecognized	Unrecognized	2023-03-16T00:32:03.854Z
Embedded	Google Xenon 2024	Authentic	Usable	Usable	2023-03-16T00:32:03.59Z
TLS	Let's Encrypt Oak 2024h1	Authentic	Usable	Usable	2023-03-16T00:35:28.898Z
TLS	DigiCert Yeti 2024	Authentic	Usable	Usable	2023-03-16T00:35:29.071Z

This is great engineering by Cloudflare and helps improve the robustness of the WebPKI in the face of certificate authority blunders or catastrophic log failures. Unfortunately, Chrome is considering removing support for SCTs delivered via the TLS handshake. We believe this would be a mistake and hope they reconsider.

Final Thoughts

This isn't the first time that certificate authorities have botched Certificate Transparency compliance and issued nonfunctional certificates: in 2021, several CAs missed the memo that Apple had increased the minimum number of SCTs.

It's unfortunate that certificate authorities cannot be relied upon to validate the information they place in their certificates. This means that site owners need to vigilant about the mistakes made by CAs: both mistakes that lead to attackers getting certificates for your domains, and mistakes that cause your certificates to not work in browsers. You can learn more about how certificate monitoring by SSLMate can help.

macOS 11.4 and iOS 14.6 impose some new requirements on publicly-trusted SSL certificates which were issued on or after April 21 2021. In general, as a website owner you don't need to worry about this change, as it's your certificate authority's job to stay abreast of policy changes and provide you certificates which are compliant with web browser policies (all certificates issued through SSLMate are compliant).

Unfortunately, some certificate authorities, namely GoDaddy, GlobalSign, Certigna, and WidePoint, messed up and issued tens of thousands of non-compliant certificates between April 21 and April 27 that will not work in macOS 11.4 and iOS 14.6 (Safari will say "This Connection is Not Private"). If you got a certificate from one of these CAs during this time frame, you might need to replace your certificate. You can use our Certificate Transparency Policy Analyzer to check if your website is affected. If you're a Cert Spotter customer, Cert Spotter checks for this problem automatically and will alert you if you need to take action.

Background

Apple and Chrome require certificate authorities to publish information about their certificates in public Certificate Transparency logs so that misbehavior by certificate authorities can be detected. For example, Cert Spotter monitors Certificate Transparency logs and alerts you when a certificate is issued for your website that you didn't authorize.

Before issuing a certificate, the certificate authority publicly announces its intent to issue the certificate by publishing a precertificate in multiple Certificate Transparency Logs. Precertificates are similar to certificates and contain the same information that will be included in the certificate, such as the domain names that the certificate will be valid for.

Each Certificate Transparency log returns a receipt, called a Signed Certificate Timestamp or SCT, promising to publish the precertificate. The certificate authority embeds the SCTs in the certificate. When Chrome or Safari load a website, they examine the SCTs, and if the certificate lacks a sufficient number of SCTs, they reject the certificate and refuse to load the website.

The number of required SCTs depends on the certificate lifetime. For certificates issued before April 21, 2021, both Chrome and Apple used the following policy:

Lifetime	# of SCTs From Distinct Logs
< 15 months	2
15 - 27 months	3
27 - 39 months	4
> 39 months	5

But for certificates issued on or after April 21, Apple now requires:

Lifetime	# of SCTs From Distinct Logs
180 days or less	2
181 to 398 days	3

This change was communicated to certificate authorities ahead-of-time, and most of them got the memo. Unfortunately, certificate authorities often have a hard time complying with the rules.

Estimating the Amount of Non-Compliance

To estimate how many non-compliant certificates have been issued, Rob Stradling of crt.sh searched for precertificates which were issued on or after April 21, are valid for more than 180 days, and which were logged to fewer than three logs. This provides a lower bound on the number of non-compliant certificates, since if a precertificate is in fewer than 3 logs, its corresponding certificate can't possibly have SCTs from 3 or more logs. However, it's only a lower bound: a precertificate might be in 3 or more logs, but the corresponding certificate doesn't necessarily embed SCTs from all of them.

Here's a summary of the data collected by Rob:

Certificate Authority	# Non-Compliant Precertificates
GoDaddy	76,376
GlobalSign	183
Certigna	15
WidePoint	5

To get an alternative lower bound, SSLMate searched for certificates (not precertificates) issued between April 21 and May 3 which are valid for more than 180 days, embed at least one SCT, and which are not compliant with Apple's Certificate Transparency policy. (We skipped certificates containing zero SCTs, as such certificates were probably intentionally not logged, which is permissible as long as you don't need your certificate to work in Safari or Chrome.) Evaluating the certificate, rather than the precertificate, definitively tells us whether or not the certificate is compliant. However, most certificate authorities only log precertificates, and their certificates will only be logged if they're discovered and submitted by a third party, such as the Googlebot. Therefore, evaluating certificates doesn't provide us a full picture of a certificate authority's compliance.

Here are our results, broken down by issuance date so that we can determine when each CA fixed the problem:

Certificate Authority	Issuance Date	# Non-Compliant Certificates
GlobalSign	2021-04-21	39
GlobalSign	2021-04-22	72
GlobalSign	2021-04-23	31
GlobalSign	2021-04-24	14
GlobalSign	2021-04-25	10
GlobalSign	2021-04-26	338
GoDaddy	2021-04-21	67
GoDaddy	2021-04-22	72
GoDaddy	2021-04-23	50
GoDaddy	2021-04-24	47
GoDaddy	2021-04-25	32
GoDaddy	2021-04-26	74
GoDaddy	2021-04-27	39

Our data increases the lower bound for GlobalSign to 504 non-compliant certificates.

How Did This Happen?

Looking at the non-compliant certificates, some interesting patterns emerge.

Every single one of GlobalSign's non-compliant certificates has a validity period of 182 days, 14 hours, 54 minutes, and 36 seconds. This happens to be just under the number of days in the six month period between April and October. It appears that GlobalSign thought that Apple's new policy affected certificates with validity periods greater than six months rather than greater than 180 days.

As for GoDaddy, the validity periods of the non-compliant certificates range from 181 to 254 days. None of the certificates has a validity period of a full year, suggesting that only reissued certificates (which have the same expiration dates as the original certificate) were affected. It's possible that GoDaddy was consulting the issuance date of the original certificate, rather than the reissued certificate, when deciding how many SCTs the reissued certificate needed to have.

By default, email is insecure: any email sent to your domains is at risk of being intercepted and read by network attackers. Even if your domains support STARTTLS, an active attacker can downgrade the connection to plaintext. MTA-STS changes that. It's a new standard that lets you publish a policy requiring that inbound email to your domains be encrypted in transit using authenticated STARTTLS. Major providers like Gmail already support MTA-STS. If your domains receive email, whether via a third party provider or your own mail servers, you should consider deploying MTA-STS.

However, MTA-STS is tricky to deploy correctly, and a mistake can prevent email from getting through. MTA-STS imposes specific requirements on your domain's mail servers; if the mail servers fall out of compliance, due to problems such as an expired or incorrectly-installed certificate, you'll lose email. If your MTA-STS policy falls out of sync with your domain's DNS, you'll lose email. The antidotes are monitoring to detect mistakes, and automation to prevent mistakes in the first place. Cert Spotter now does both: monitor your domains for MTA-STS problems, and automate the publication of correct MTA-STS policies.

Monitoring

Cert Spotter continuously monitors your domains' MTA-STS policies and alerts you if a problem would prevent the delivery of mail. Cert Spotter automatically detects MTA-STS policies for any of your monitored endpoints; you don't need to enable MTA-STS monitoring separately.

Among the problems detected by Cert Spotter are:

• A mail server is not eligible for MTA-STS (doesn't support STARTTLS with TLSv1.2 or higher, or has an expired, mismatched, untrusted, or otherwise invalid certificate).
• The MTA-STS policy can't be retrieved or is invalid.
• The MX servers listed in the policy don't match your domain's MX records.
• The MTA-STS policy has changed without rotating the ID in the _mta-sts DNS record.
• The MTA-STS policy has been removed without following the proper procedure to flush policies cached by sending mail servers.

Additionally, Cert Spotter provides a dashboard which shows MTA-STS eligibility and deployment status in one place. You can use this dashboard to track MTA-STS deployment across your entire domain portfolio:

Automatic Policy Publication

Deploying an MTA-STS policy initially requires:

• Publishing a text file over HTTPS at mta-sts.YOURDOMAIN, which requires a valid certificate for mta-sts.YOURDOMAIN.
• Publishing a DNS TXT record at _mta-sts.YOURDOMAIN containing the ID of the policy.

Maintaining an MTA-STS policy on an ongoing basis requires:

• Renewing the HTTPS certificate for mta-sts.YOURDOMAIN.
• Changing the policy any time you change your domain's MX records.
• Changing the TXT record's ID any time you change the policy.

Cert Spotter can optionally automate the above steps. All you need to do is create DNS CNAME records pointing mta-sts.YOURDOMAIN and _mta-sts.YOURDOMAIN to domain names hosted by SSLMate. We'll automatically publish a policy matching your MX records. For transparency, we'll alert you any time your MX records change, so you can take corrective action if the change was unauthorized. To set up automated MTA-STS, visit the endpoint page for your domain to retrieve the necessary DNS records and configure policy settings like the enforcement mode and the maximum age:

Final Thoughts

Email is a critical communication tool, and despite its flaws, it's here to stay. It deserves the same level of encryption and authentication that we take for granted with web browsing. MTA-STS is a major step in that direction, and Cert Spotter is here to help you easily and fearlessly deploy MTA-STS on your domains. Sign up for Cert Spotter today

One of the most requested features for the new Cert Spotter has been for Cert Spotter to consult CAA records to decide if a certificate is authorized. This makes a lot of sense - many domain operators have already published CAA records, and it's redundant to duplicate the information in your Cert Spotter settings. Also, CAA records can apply different policies to different domain names, which is more flexible than the list of authorized CAs in your Cert Spotter settings, which applies to every domain you monitor.

However, implementing this feature without degrading the security of Cert Spotter was trickier than it sounds. DNS is, by and large, not authenticated, which means that when Cert Spotter goes to look up CAA records, a network-layer attacker could return bogus CAA records which trick Cert Spotter into thinking a certificate is authorized.

DNSSEC, if we set aside its low adoption rate and deployment challenges, would at least make CAA lookups authenticated, but they still wouldn't be transparent. A misbehaving parent zone, such as a TLD registry, could return validly-signed but malicious CAA records and no one would be the wiser. That's a problem for Cert Spotter, which takes great care not to blindly trust third parties. Cert Spotter verifies the Merkle Trees of all Certificate Transparency logs, and gossips information about them with other auditors, including Google's Certificate Transparency monitor and a network of "honeybees" located all over the Internet. This gives Cert Spotter a high degree of confidence that the logs which it relies upon are not hiding certificates. DNSSEC doesn't provide the same level of confidence that DNS servers are not misbehaving.

For this reason, using CAA was never a consideration during the development of Cert Spotter. But our customers have spoken, so we set out to find a way to mitigate the risks of CAA as much as possible, and settled on the following solution:

1. You can optionally enable the use of CAA records for certificate authorization in your settings.

2. At any time you can examine the list of CAA records that Cert Spotter is relying upon to make authorization decisions.

3. Cert Spotter will email you any time it detects that a CAA record has been added, modified, or removed.

While this solution won't stop Cert Spotter from being fooled by a malicious CAA record, you will at least know about it and be able to take action in response - much like Certificate Transparency doesn't stop malicious certificates, but makes sure you know about them.

Cert Spotter's use of CAA records is unique among Certificate Transparency monitors and is just one example of how SSLMate is innovating to bring you certificate monitoring that reduces alert fatigue by making it easy to automatically authorize (and elide notifications for) legitimate certificates - without reducing security. We have some other features in the pipeline to offer even more flexibility in auto-authorizing certificates. Get in touch if you are interested in being added to our beta tester list.

Do you operate a website? Do you like security and uptime? Cert Spotter monitors your website's security certificates to give you early warning about security and uptime problems like malicious certificates, DNS tampering, and certificate expiration - so you can act before your customers notice a problem, not after.