Important information about Certificates/CAs/OCSP Must-Staple

[Zoey2936]

Note: This is not a new release but an important information

Let's Encrypt has made an announcement today which has a huge impact on NPMplus, you can read it here: https://letsencrypt.org/2024/12/05/ending-ocsp

If you have any question/ideas etc. on this topic, please write a comment

What is OCSP/CRLs?

• first: OCSP, OCSP Stapling and OCSP Must-Staple are different things.

OCSP:

• With OCSP the client (Browser), asks the CA (Let's Encrypt) if the Certificate used by the web server was revoked
• This is a check done between CA and Client, NPMplus has no influence on this
• Revocation will be detected by the client if the client (re)checks for it (recheck because of cache which could exist)
• privacy problems since the CA knows things which it should not (can be disabled in Firefox/thunderbird settings, not sure about other clients, chrome doesn't even support this)
  => useful, but with a privacy problem, maybe takes some time to be detected because of cache
• https://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol

OCSP Stapling without Must-Staple:

• your 30 days cert is valid on its own, but it is additionally verified by a “second certificate”
• always enabled for all certbot certs of NPMplus (also for migrated instances and also if you disable must-staple)
• the “second certificate” is valid only for 7 days
• the “second certificate” is requested by the web server from the CA (since nginx implementation is not the best, NPMplus uses certbot-ocsp-fetcher for this)
• no privacy problem since browser only talks to the web sever and not to the CA
• BUT: if the cert is compromised, the cert can still be used WITHOUT OCSP Stapling and through that revocation is not be detected by the client (if the client doesn't detect trough other ways client-CA OSCP/CRLs)
  => useless without must staple, see below
• https://en.wikipedia.org/wiki/OCSP_stapling

OCSP Stapling with Must-Staple:

• same as above, but your 30 days cert is NOT valid on its own, it needs a “second certificate” to be valid
• this requirement is part of the cert itself (so not removable) and added while creating the cert (must-staple requirement is enabled by default with NPMplus for all certbot certs, but can be disabled via ENV)
• Revocation will be detected as soon as the “second certificate” expires and if the client support must-staple
  => useful, if supported by the client, maybe takes some time to be detected because of validity of “second certificate”
• https://en.wikipedia.org/wiki/OCSP_stapling

CRLs

• the older technology to detect revocation
• CAs publish huge lists containing information about all revoked (and by date still valid) certificates and chrome/Firefox/thunderbird download collections of these lists
• Problem: because of size, lists may not contain all revoked certs
  => depends: if the revocation information of your cert is not included, then it is useless, otherwise it is ok
• https://en.wikipedia.org/wiki/Certificate_revocation_list

My opinion on this (I mostly talk about Must-Staple)

• first I understand that they remove OCSP because of the costs they have through it and because of the privacy concern
• BUT I don't understand that they remove Must-Staple support, it is better then CRL
• They argue that most web servers have no good implementation for this, which is not fully true, like caddy which has good support for this or NPMplus itself through the certbot-ocsp-fetcher script ((free)nginx own implementation is not the best) ⇒ I don't think that this argument is big enough to revert to CRLs
• The argument with the client is sadly true, since chrome (and it forks) doesn't support must-staple, so the only big clients remaining are Firefox/thunderbird (not sure about WebKit/safari and other big mail clients like outlook)

What now?

• I must say, there is no good solution:
  • Moving to ZeroSSL would be an option, but they have no CRLs support, which is important for chromium (and forks)
  • Staying with Let's Encrypt would mean to lose Must Staple functionality
• But since a decision needs to be made and chrome is very important, I will stay with Let's Encrypt by default and instead change the default value of ACME_MUST_STAPLE, maybe stapling needs to be fully removed even if your custom CA supports it, but I will try to find a way to prevent this
• If ZeroSSL or any other public ACME supporting CA will have support for OCSP Stapling/Must-Staple and CRLs, then this will become the new default CA

When will the change happen?

• Before January 30, 2025 (the day Must-Staple will stop working for new instances)
• I have no date exactly, but I will try to have some releases until this change:
  • at least on release mentioning this change in its changelog, in a few days
  • in between (or maybe with the next release), I will sync the value of ACME_MUST_STAPLE with all renewal configs of certbot cert to make sure that certs which get renewed will have the same setting as the env (maybe I will also sync the ACME_SERVER env)
  • and at some point ACME_MUST_STAPLE will be changed to false by default (and stapling may be removed) - maybe still in December
• ENVs set by you will not be overridden

---

This discussion was created from the release Important information about Certificates/CAs/OCSP Must-Staple.

[Raxiel1987]

Thanks for this explanation,
I already added the must staple env to false (cause Firefox) so I'm assuming I'm ok, correct ?

[Zoey2936]

yes

[Raxiel1987]

yes

Thank you, I really appreciate your work, and your continuous help, I only need to make it work with Wordpress now 😅

[Zoey2936]

https://grapheneos.social/@GrapheneOS/113640802603196882

[Raxiel1987]

What this means ? Good or bad news ?

[Zoey2936]

both, good because short-lived certificates are a good replacement for must-staple, but bad because no idea when they will be released

[Raxiel1987]

so we'll wait for that then :D

[ShrinkWrapper]

Would Googles "newish" TLS service suffice as a substitute? https://pki.goog

Setup procedure: https://cloud.google.com/certificate-manager/docs/public-ca-tutorial

Keypoints:

• Free
• 90 days cert length
• ACME support
• Wildcard support
• OCSP with stapling
• CRL support
• DNS, HTTP and TLS-ALPN request challenge

The only CON I see with them is that you need to connect it to a google cloud project, then enable google CA API in the project and use that API key to request certificates. But it works for normal free google accounts or any random google account you create and they provide a cli tool so the process can be fully scripted.

gcloud init
gcloud projects create mycertificates
gcloud config set project mycertificates
gcloud projects add-iam-policy-binding mycertificates --member=user:ShrinkWrapper --role=roles/publicca.externalAccountKeyCreator
gcloud services enable publicca.googleapis.com
gcloud publicca external-account-keys create
    b64MacKey: aaaaaaaaaaaaaaxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    keyId: bbbbbbbbbbbxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

certbot register --email "[email protected]" --no-eff-email --server "https://dv.acme-v02.api.pki.goog/directory" --eab-kid "bbbbbbbbbbbxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" --eab-hmac-key "aaaaaaaaaaaaaaxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
certbot certonly --manual --must-staple --preferred-challenges "dns-01" --server "https://dv.acme-v02.api.pki.goog/directory" --domains "*.somerandomdomain.example"

I just tried the procedure and uploaded the certificate manually to NPMPlus and added it to this domain if you want to inspect the certificate "removed domain".

Edit: I ran certbot-ocsp-fetcher for it and updated the host manually to enable stapling for the custom certificate inside the container, SSLLabs is now reporting Stapling for it as well.

I couldnt get OCSP Stapling with Must-Staple to work, only OCSP Stapling without Must-Staple. Google might not support Must-Staple after further testing :(

[Zoey2936]

Thanks for the idea, but you wrote the problem yourself, people would need to create eab keys inside the google cloud console and at the same time, even if they support ocsp and crl, they don't support must-staple. I used it myself for some time, and will use it again, but missing must-staple is still a problem, having ocsp and crl is better than only one, but as written in my initial post stapling without must-staple is useless

[Zoey2936]

I thought about contacting and asking, but I don't know where I could contact them. But I don't think they will add it, when not even their browser supports it.

Contacting Let's encrypt makes no sense, since their decision is final.
I also wrote a mail to ZeroSSL, but the answer was short: Please note we do not have any plan to support CRLs for Chronium in the near future. => still must-staple is great but crls are important for chrome, at least currently, but I don't think chrome will add support in the near future.

[Zoey2936]

I just hope that Let's encrypt add support for short live certs very fast, but this would be, again, a breaking change in NPMplus

[ShrinkWrapper]

Yeah I was a bit quick on posting before I had checked if must-staple worked 😅
I spent some time yesterday looking at different certificate providers, but there doesnt seem to be lots of free options that have it all, lets encrypt was about the only one that checked all boxes. Crossing fingers lets encrypt manage to quickly implement their proposed changes.

[YaASlowpoke]

Ah, this explains why my applications and websites started behaving wonky recently. Reverted back to the version from early November which is working without problems.

I'll take a look in January to see what actions are expected of me, I'm using Unraid which is working with templates and the template that was added for NPMplus is missing the (newly added?) environment variables (there would be no problems adding them myself).

[Zoey2936]

What do you mean exactly with wonky?

[YaASlowpoke]

My server updates my docker containers on a weekly basis, (locally 15th) sunday night, and it seems the latest changes were pushed the day before. Ever since then I keep experiencing the errors below when trying to reach my subdomains I've setup in NPMplus.

With Plex (selfhosted mediaserver/player) my clients/users were getting an message that the server couldn't be found, they have to close/open the Plex app on their TV/Mediabox several times before it finally connected (gives the same feeling as having to F5 multiple times, same solution as on Firefox).

Now that I've reverted the update users no longer seem to be experiencing connection issues, though its been only a few hours since the revert so i might have to wait till i get a definitive conclusion (could be the application itself too, but having to wait before further troubleshooting).

Besides that I recently got errors (MOZILLA_PKIX_ERROR_REQUIRED_TLS_FEATURE_MISSING) with Firefox and accessing subdomains that worked perfectly fine before, but after some research that seems to have to do with "HSTS and security headers". Why this suddenly occurred I have no idea, but disabling this seem to solve this problem (as of now).

When I have more time I'll look through the logs to find the cause, just a bit too busy this week of the year..

[Zoey2936]

MOZILLA_PKIX_ERROR_REQUIRED_TLS_FEATURE_MISSING: there are many discussions just about this topic, just search for it