Plugins: security/acme-client: automated sftp export: no fullchain.pem

[thutex]

Describe the bug

When exporting certificates to a remote host over sftp, ca.pem, cert.pem and key.pem get exported.
fullchain.pem, however, does not seem to be exported.

To Reproduce

Steps to reproduce the behavior:

1. export certificates

Expected behavior

all parts of the certificate should be exported

Additional context

for my use-case, it is to use fullchain.pem with postfix to correctly enable tlsa records

Environment

OPNsense 20.1.2-amd64
FreeBSD 11.2-RELEASE-p17-HBSD
OpenSSL 1.1.1d 10 Sep 2019

[thomo5000]

Respectfully, is this really a feature? The export is clearly not transferring all the files required for a valid configuration. Unless I'm missing something the intended purpose of this script is to export the files required to create/deploy a valid configuration.

For my purposes at least sftp export is broken.

[fraenki]

Respectfully, is this really a feature?

The sftp export feature was contributed by a volunteer, so I don't know the intended scope of this feature. I haven't had enough time to look into this. Feel free to point at the actual code or provide a fix by submitting a pull request.

[thutex]

Respectfully, is this really a feature?

it is, and the feature is literally that: export the certificates.
i do miss the fullchain.pem file, but even without that one you can use the others to create a full chain, it's just a bit more configuration.

you probably are indeed missing the point, which is: export updated letsencrypt files to another host, which is configured to use those certificates.
for example, for me the LE plugin renews my certificates, and the sftp export exports the new certificate files to my postfix box. there i have a watcher that reloads postfix once the files change.
all the essentials are being transferred - fullchain.pem just makes some things a little easier, but not essential.

[thomo5000]

For my example I'm trying to update the certificates on a nginx host pre-configured to use them. So I think we are talking about the same thing. But unfortunately it is not fit for this purpose. Lets back up.

The real problem is not that we are not exporting the fullchain: but that we are not exporting the intermediate certificate.

We are exporting the CA and the host certificate, but _without the intermediate certificate_ there is no way to build a valid trust chain that will be trusted by clients.

Exporting the fullchain instead of separate host + intermediate certs is just a shortcut as that's the way letsencrypt ship them and that's the way nginx requires them (IIRC opnsense itself stores it as a fullchain for this reason).

Anyway you'd think I could just copy the intermediate certificate from the opnsense server manually and set them up that way. But I've really got no way of knowing if the intermediate ever changes, so this isn't satisfactory for production. The way letsencrypt works you really need the set of certificates issued together.

Your mail server sounds like it's not set up to use intermediate certificates?

[thutex]

if you (correctly) combine your ca.pem and cert.pem you will get a valid chain.
postfix does this fairly easily on its own (just specify your files), dont know about nginx.
but you can easily test this: create your own file with FIRST cert.pem and then ca.pem and you will have a valid aka fullchain.pem
change the order to ca.pem and then cert.pem and it will be invalid (missing intermediary)

you can also compare with the fullchain.cer on your opnsense box, it will contain the exact same.

this is why i say it would be easier if fullchain was exported along with the rest, but not essential.
(still would like for it to be exported though)

[thomo5000]

I don't think that's correct. I know a CA + host cert combined into a "fullchain.pem" will satisfy most clients but a real intermediate is supposed to be included or it's not a fullchain. Try running SSL Labs test against your domain and I bet you are capped at a "B" score because the intermediate had to be fetched as a separate download. Web browsers will be happy to accept this but it isn't accepted by, (for example) java applications in my enterprise.

But I do take your point that fullchain.cer produced by certbot (and used by opnsense) is missing the intermediate as well, I had not noticed that before.

Final: the fullchain.cer on the opnsense box contains Intermediate and CA. i.e. It is a vaild fullchain _for the intermediate_.

[thutex]

i will do you one better: https://ssl-tools.net/mailservers/2easy.be this is 1 of the domains using the mailserver behind opnsense, using the certificates from the export.
it has the DST root x3 and the LE authority x3, so it is a complete chain, no?
also, my sites are capped at A because i stopped using hsts
... what did i win from your bet ? ;)

additional: i just went to check out one of my older, generated by certbot on another box, fullchain.pem certificates. it has the exact same 2 certificate contents as combining ca and cert.
always been valid, both for mail and web, and i used to have a+ on ssllabs with them.

[thomo5000]

That's not quite the test I was proposing. With a SSL labs test your webserver will only get A if there are three certificates with no additional downloads. I'm curious if maybe your old webserver was Apache? Because IIRC certbot sets up Apache correctly with three separate certs instead of using fullchain (apache didn't know what to do with a full chain until recently).

Anyway you've convinced me that the problem of intermediate not being included is a distinct problem to the cerbot "fullchain" not being included so lets not get further side tracked.

unedit

[thutex]

i do always use apache, but i never let certbot do anything automatically until recently (now that opnsense is in front of the web and mail vm)
in any case, i have always used the fullchain and always had at least an A-grade rating.
also checking the fullchain (or combo ca/cert) on keycdn (https://tools.keycdn.com/ssl) does not produce errors.

i have no clue where exactly your desired certificate lives/is generated/can be copied, but it would indeed seem to be unrelated to my question of including the fullchain.pem in the export.

[thomo5000]

Please not the ACME-client documentation

fullchain.pem
All certificates, including server certificate (aka leaf certificate or end-entity certificate). The server certificate is the first one in this file, followed by any intermediates.
This is what Apache >= 2.4.8 needs for SSLCertificateFile, and what Nginx needs for ssl_certificate.

Or how about the TLS RFC

certificate_list
This is a sequence (chain) of X.509v3 certificates. The sender's
certificate must come first in the list. Each following
certificate must directly certify the one preceding it.

So the full chain is supposed to contain the intermediate, at least acording to the acme-client developers and the TLS RFC authors.

Call it feature or bug IDGAF but if the fullchain is to be included in the export is MUST contain the intermediate.

[thutex]

but the LE x3 IS the intermediary? you have the root (dst/isrg), LE x3 as intermediary, then your cert.
see chain of trust

[thomo5000]

"The sender's certificate must come first in the list. Each following certificate must directly certify the one preceding it."

So the pem file must contain Host certificate > Intermediate certificate(s) > CA certificate.

Yes LE x3 is the intermediary but your configuration "create your own file with FIRST cert.pem and then ca.pem and you will have a valid aka fullchain.pem" will make the client download the intermediate which will fail for many clients.

[thomo5000]

Maybe it will help to expand my use case. In a enterprise setup you have backend servers talking to each other using TLS. These servers are behind a DMZ (opnsense to the rescue) and do not have internet access.

The standard way this used to be done was 1) with real CA signed certs. Which is often waste of money for internal services or 2) with self-signed certificates, but management was a pain because you had to install your own self-signed CA on all servers so they would trust your host certs.

Certbot was always a tempting option because your hosts should trust the CA implicitly, but as the hosts do not have internet access to update their certs this was impossible (also when you have 30 hosts all asking for *.domain.com letsencrypt will start rate limiting you and messing up monitoring).

Then I notice this SFTP export functionality and I'm like "HA! somebody has finally solved the enterprise certificate problem". And indeed having a security appliance update domain certs once and distribute them to hosts protected behind it seems so perfect that I assumed it was the whole reason for for the functionality. Maybe I am wrong there.

Anwyay this clearly won't work as the backend hosts can not download the intermediate certificates they will not trust the certs. Distributing hosts certs and CA certs without the intermediate certs is actually useless so for enterprise purposes broken.

[mimugmail]

Can we stop this kind of conversation please and switch to professional level?

The feature was contributed by a volunteer, working in his/her spare time. Also, don't forget, 99% of the plugins are supported by community. This means, enterprises should not demand enterprise features from plugins. If they are in, good, if there's something missing, add a feature request and provide as such support as possible :)

This thread moves in the wrong direction, it should just stay to the initial author,
"hey we have here a cool feature, but I need for my purpose this and that in addition".

[thomo5000]

Of course you are right. Much respect to volunteer developers.

[mimugmail]

Ok, I took quick look at the script. It exports the files separate:

https://github.com/opnsense/plugins/blob/master/security/acme-client/src/opnsense/scripts/OPNsense/AcmeClient/upload_sftp.php#L571-L591

Maybe we can adjust the script to put all in one, but I have no system with an active certificate and intermediate. Can you send me your config.xml with changed keys?

[thutex]

thanks @mimugmail ! i actually was trying to solve the issue, but didn't compare the config.xml with the php script (my bad) so i didn't find the exact way to solve this issue.
thanks to your comment i actually figured it out and made a pull request: https://github.com/opnsense/plugins/pull/1753

[fraenki]

@thutex added support for exporting a fullchain.pem, which will be available in the upcoming version 1.30 of os-acme-client.

The sftp export feature was contributed by a volunteer, so I don't know the intended scope of this feature.

To follow-up on this: the scope of the sftp export feature was never to export a fullchain. So a missing fullchain.pem is, in fact, not a bug, but a missing feature. Glad @thutex found the time to contribute this new feature. :+1:

[jkellerer]

@thutex thanks for your addition, indeed for my main use case I didn't need a fullchain so far as the servers I used it for wanted the ca-chain separately. I'm glad that you added it (the volunteer).

@fraenki, as the upload names are customisable, the UI also needs a little adjustment after merging #1753. Complementary PR is #1776.

@thomo5000, I don't want to open a discussion, however my use case is quite similar to what you described, so it should also work for yours. In difference to what the let's encrypt client usually provides, "ca.pem" of the SFTP upload contains intermediate & root cert (in this order) to avoid the issues you had described. So glueing "cert.pem" & "ca.pem" together provides a valid full chain. For verification, I've just tested this on Apache and Nginx (based on changes from @thutex) using "openssl s_client ... -showcerts" which clearly shows 3 certs (server > inter-CA > root-CA) in the correct order for both servers.