Fixing Let’s Encrypt Root CA X3 Expiry on Yealink Phones

So as many people are now (possibly painfully) aware, Let’s Encrypt root CA X3 certificate expired on September 30, 2021. For most users, the expiry of the old root certificate went unnoticed. This is because this was expected and planned for. Let’s Encrypt had long since been handing out certificates referring to their own new ISRG Root X1 certificate, which is valid until 2035.

The Chain of Problems

Unfortunately, many embedded systems (like Yealink phones) have built-in root certificates which don’t necessarily get updated that often. Essentially, they still have the expired X3 certificate as a root CA to validate the certificate chain. So in essence, the valid certificate presented by the domain is being validated by a root authority certificate stored in the device which it considers to be expired.

Many people have accepted this explanation and have been left frustrated, seeking alternatives. This, however, is far from the full story. It didn’t make sense that these certificates were not being accepted because many impacted devices have already had the new ISRG Root X1 certificate issued back in 2015 which itself expires in 2035. So after doing some digging, I discovered a hint of what was happening on the FreePBX Community Forums:

The problem is not the expired root CA DST cert although I know a lot of people think it is. The problem is that the intermediate R3 cert that is supplied by LetsEncrypt with even new certificates was deliberately cross-signed with the expired DST cert so that old antique Android clients that don’t validate expirations on their DST CA certs and don’t have the ISRG CA root in their cert store would continue to work.

– Ted Mittelstaedt on the FreePBX Community

This led me to a forum post post from March 31, 2021 where a Let’s Encrypt engineer confirmed the breaking change that led to this:

On May 4, we’ll update our API so that ACME clients will download and use a longer certificate chain. This longer chain will ensure our certificates remain compatible with almost all Android devices 102 even after DST Root CA X3 87 expires on September 30. Most subscribers don’t need to make any changes.

This chain will consist of three certificates, instead of the current two:

• End-entity certificate (aka leaf certificate), signed by R3
• R3, signed by ISRG Root X1
• ISRG Root X1, signed by DST Root CA X3

Our API will also offer an “alternate” chain, which you may configure your ACME client to select instead:

• End-entity certificate, signed by R3
• R3, signed by ISRG Root X1

We think the long chain is right for most websites. If you know you don’t have to support Android users, you may want to choose the short chain.

So essentially what is happening is that by default as of May 4th, Let’s Encrypt began handing out certificates that were, at the root of the chain, cross-signed by DST Root CA X3. The problem is that this is the certificate that just expired. Although the current issued DST X3 certificate is valid, the one stored in phones and other embedded systems has not been updated with the new one because it shouldn’t have been necessary. Vendors (in my opinion, rightly) assumed that since Let’s Encrypt has their own root X1 certificate now, it should not be necessary to update the DST Root CA X3 one. As a result, when a device reaches the end of the chain, they are hitting the expired cert they were preloaded with, so the validation ultimately fails!

Fixing Let’s Encrypt on Yealink Phones

Thankfully, the solution is fairly simple, if not a bit unnecessary thanks to Let’s Encrypt’s odd default of using the longer chain.

Essentially, when using your preferred ACME client such as Certbot or Dehydrated you need to issue the cert using the the follow argument added:

--preferred-chain "ISRG Root X1"

This will request the certificate with the shorter chain and will omit the DST Root CA X3 certificate which is causing authentication to fail. Don’t forget to restart nginx, apache or whatever server is using the newly issued certificate.

How to Check your Current Certificate

You can check your current SSL certificate being issued by your web server and view the chain of trust very clearly here:

https://www.sslshopper.com/ssl-checker.html

This is what it looks like when using the default long chain with the old X3 root:

After passing --preferred-chain "ISRG Root X1" and reloading, you should see a chain that looks like this:

Other Gotchas

• This may affect some ancient Android clients or any client which hasn’t added the ISRG Root X1 cert, but since that’s been issued since 2015 it shouldn’t be a problem for most users.
• If you are trying to issue you certificates from some clients, they may not support the –preferred-chain option. It appears that Dehydrated only supported this option as of version 0.7.0. People have also reported issues with certbot users on CentOS 7.
• Devices that are too old to support the ISRG Root X1 root CA will never work unless they are updated.

Hopefully this can help somebody else fix their issues with Let’s Encrypt certificates on their devices

Posted on November 2, 2021