Fixing Let’s Encrypt Root CA X3 Expiry on Yealink Phones

Let’s Encrypt’s DST Root CA X3 certificate expired on September 30, 2021. For most users, the expiry passed unnoticed. For embedded devices like Yealink phones, however, it can break TLS validation against servers that issue Let’s Encrypt certificates — provisioning fails, the phone refuses to connect to its config server, or HTTPS dialog boxes fail to validate. This article explains why, and how to fix it on the server side with a one-flag change to your ACME client.

The Chain of Problems

Many embedded systems (Yealink phones included) ship with a baked-in root certificate store that does not get updated often. They still carry the expired DST Root CA X3 root certificate and fall back to it for validation. When a server presents a valid Let’s Encrypt certificate that ultimately chains up to DST Root CA X3, the phone treats the chain as expired and the validation fails — even though the leaf certificate itself is fine.

What makes this confusing is that most affected devices already have the newer ISRG Root X1 certificate (issued in 2015, valid through 2035). The expectation was that they would simply use ISRG Root X1 to validate Let’s Encrypt certificates and ignore the older root. Unfortunately, Let’s Encrypt’s default chain still routes through DST Root CA X3 — and that’s the root our phones reject.

The problem is not the expired root CA DST cert although I know a lot of people think it is. The problem is that the intermediate R3 cert that is supplied by Let’s Encrypt with even new certificates was deliberately cross-signed with the expired DST cert so that old antique Android clients that don’t validate expirations on their DST CA certs and don’t have the ISRG CA root in their cert store would continue to work.
— Ted Mittelstaedt on the FreePBX Community

A March 31, 2021 forum post from a Let’s Encrypt engineer confirmed the change that caused this:

On May 4, we’ll update our API so that ACME clients will download and use a longer certificate chain. This longer chain will ensure our certificates remain compatible with almost all Android devices even after DST Root CA X3 expires on September 30. Most subscribers don’t need to make any changes.

This chain will consist of three certificates, instead of the current two:

• End-entity certificate (aka leaf certificate), signed by R3
• R3, signed by ISRG Root X1
• ISRG Root X1, signed by DST Root CA X3

Our API will also offer an “alternate” chain, which you may configure your ACME client to select instead:

• End-entity certificate, signed by R3
• R3, signed by ISRG Root X1

We think the long chain is right for most websites. If you know you don’t have to support Android users, you may want to choose the short chain.

In short: as of May 4, 2021, Let’s Encrypt began issuing certificates whose default chain ends at DST Root CA X3. That is the certificate Yealink phones and similar embedded devices reject as expired — so validation fails on the device side even though the leaf cert is valid.

The Fix: Use the Short ISRG Root X1 Chain

Switch your ACME client to request the alternate (short) chain rooted at ISRG Root X1 instead of DST Root CA X3. With Certbot, Dehydrated, and most modern ACME clients, this is a single flag:

--preferred-chain "ISRG Root X1"

Reissue your certificate with this flag, then reload the web server (nginx, Apache, etc.) so it serves the new chain. Yealink phones validate against ISRG Root X1 directly and the connection succeeds.

Verify the New Chain

You can confirm which chain your server is presenting using SSL Shopper’s chain checker:

https://www.sslshopper.com/ssl-checker.html

Default long chain (with the expired X3 root)

Short chain after --preferred-chain "ISRG Root X1"

Other Gotchas

• This change may affect very old Android clients or any client that has not added ISRG Root X1 — but since that root has been issued since 2015, this is not a concern for most modern devices.
• Some ACME clients do not support --preferred-chain. Dehydrated added support in version 0.7.0; some users have reported issues with Certbot on CentOS 7. Update your ACME client if necessary.
• Devices too old to have ISRG Root X1 in their trust store will never validate Let’s Encrypt certificates again unless their root store is updated. There is no workaround on the server side for that case.

FAQ

Will Yealink eventually update their root store?

Yealink ships firmware updates from time to time that include refreshed root CA stores, but rollout depends on which models are still supported and what version of firmware your phone is on. Check for a current firmware before assuming the X3 root is still in play. See Yealink phone user guides.

Does this affect EMAK Telecom services directly?

No — EMAK Telecom’s provisioning servers serve certificate chains that Yealink phones can validate. This article is for administrators running their own services that Yealink phones (or similar embedded clients) need to reach over TLS — for example, a self-hosted reporting dashboard or third-party integration server.

Why does Let’s Encrypt default to the longer chain if it breaks devices like these?

Because the long chain keeps very old Android devices working, and Android is by far the largest installed base. Let’s Encrypt explicitly recommends the short chain only when you know you do not need to support those legacy clients.

Does this apply to other embedded devices?

Yes — IP cameras, older printers, set-top boxes, and various IoT gear can hit the same issue for the same reason. The same --preferred-chain "ISRG Root X1" fix on the server side resolves it for all of them.

Is there a phone-side fix?

You could in principle upload an updated root CA store to the phone, but this is risky and rarely supported by the manufacturer. Fix it on the server side instead.