Lego: Challenge fails with lego but succeeds with certbot and cert-manager

[linkvt]

Hi guys,

I'm using traefik and noticed that requesting certificates from our company internal ACME endpoint failed, but works when using cert-manager (within kubernetes), certbot and even acmesh-official/acme.sh according to my colleagues.
I then tried to request the certs manually with the latest lego CLI but got the same errors, which are unfortunately not really speaking.

$ sudo dist/lego --server https://acme....../directory --email [email protected] --domains load-balancer-... --http --key-type rsa2048 -a run


2020/11/10 14:32:46 [INFO] [load-balancer-...] acme: Obtaining bundled SAN certificate
2020/11/10 14:32:47 [INFO] [load-balancer-...] AuthURL: https://acme.../authorization/0zWbr_W6CRGoZv2qVNQD
2020/11/10 14:32:47 [INFO] [load-balancer-...] acme: use http-01 solver
2020/11/10 14:32:47 [INFO] [load-balancer-...] acme: Trying to solve HTTP-01
2020/11/10 14:32:47 [INFO] Deactivating auth: https://acme.../authorization/0zWbr_W6CRGoZv2qVNQD
2020/11/10 14:32:48 [INFO] Unable to deactivate the authorization: https://acme.../authorization/0zWbr_W6CRGoZv2qVNQD
2020/11/10 14:32:48 Could not obtain certificates:
    error: one or more domains had a problem:
[load-balancer-...] authorization[get]: empty URL

I'm currently trying to debug it myself but have not really a plan about lego, ACME and Go, so this is quite hard... :grin:
I would really appreciate some help here and am willing to provide all infos you need as I understand that it is quite hard for you to debug when I'm using the company internal ACME endpoint.

Thanks in advance,
Vincent

Vincent Link, vincent.[email protected], Mercedes-Benz AG on behalf of Daimler TSS GmbH.
Imprint

[linkvt]

Ok minimal progress, the authorization URL seems to be not detected by lego in https://github.com/go-acme/lego/blob/b04d00b8fe074abe5dc5a989c2358c36511167da/acme/api/service.go#L23

The header value looks like this and contains the rel="up" part so it seems at least somehow correct:
https://acme...:443/acme/ws/Acme.svc/authorization/GwsHD-0SRW-s6VZ-h8a-;rel="up"

Edit: I compiled lego with optional < and > in the regexes and it worked, certificates were generated. I try to find out why these are in the lego code.
Regex that worked for me: linkExpr := regexp.MustCompile(?;\srel="(.+?)")

Edit2:
OK after reading the RFC of the Link header the Regex is correct and the header returned by our ACME server isn't, see https://tools.ietf.org/html/rfc8288#section-3.1 that < and > are required around the URI.
I added a test case and was successful with all tests with the following regex: <?(\S+?)>?(?:;[^;]+)*?;\s*rel="(.+?)"

maintainers would you still be interested in a PR for this? I will check the tool we use but as certbot, cert-manager, etc work it might make sense to be less strict here, too, for compatibility reasons.

[ldez]

Hello,

<> seems optional, so feel free to open a PR.

FYI @maintainers is an organization, not an alias for the repo maintainers.

[linkvt]

Hi @ldez ,
thanks, my mistake regarding the maintainers org, just wanted to make you aware of the question :).
I opened the PR with test cases, I validated the version to be able to request certificates for me.

I also don't want to sound pushy but I'm obviously interested in getting both lego and Traefik released with a bugfix so that I can use this fix soon and noticed that you are a maintainer of both projects, is there anything I can do to help with that?

Thanks!

[ldez]

I'm sorry, I re-read the RFC and check the implementation in Pebble and Boulder (Let's Encrypt), and all those elements define and use <>:

Each link-value conveys one target IRI as a URI-Reference inside angle brackets ("<>")

https://github.com/letsencrypt/pebble/blob/2c26e5bac888b5418c53dbea203fbdf9d7516104/wfe/wfe.go#L2442-L2444

https://github.com/letsencrypt/boulder/blob/ebba443cad233111ee2b769ef09b32a13c3ba57e/wfe2/wfe.go#L538-L540

lego has to follow the RFC8288.

The problem seems related to your ACME server implementation that doesn't follow the RFC8288.

Could you give me more detail about the implementation of your company's internal ACME?

[ldez]

I checked the implementation of the link parsing in acme.sh and certbot.

In fact, the algorithms used are globally based on the same approach using splits and trims. But these algorithms do not seem to be sensitive to the absence of the constraint defined by the RFC.

Currently, if I follow the RFC, I would say that these algorithms have a bug. @cpu what's your opinion about that?

Certbot

https://github.com/certbot/certbot/blob/bf20f39ceb7297dfbde8bb1c2f6e9d330c2ceea6/acme/acme/client.py#L796-L808

parse_header_links: https://requests.readthedocs.io/en/master/_modules/requests/utils/

acme.sh

https://github.com/acmesh-official/acme.sh/blob/053f4a9a2e7f74aaec4493f5e9828f229088ab7c/acme.sh#L4789

I didn't find the "up" Link parsing.

[cpu]

The problem seems related to your ACME server implementation that doesn't follow the RFC8288.

Agreed. @linkvt Out of curiosity is your ACME server an off-the-shelf product or something custom built?

@cpu what's your opinion about that?

I'm just some guy these days without much stake in the ecosystem (so feel free to disregard my opinion!) but I would be in favour of pushing the ACME server to adhere to RFC 8288. I suspect it's probably a small fix on the server-side and it would be a shame to have to add this workaround to a whole ecosystem of ACME clients when it's a pretty straight-forward specification deviation on the part of one ACME server.

[ldez]

Thank you @cpu, even if you are "just some guy these days", your opinion still counts for me :smile_cat:

I'm also very curious about the implementation of the server used by @linkvt.

[linkvt]

Hi guys, thanks for the feedback so far, really appreciate your quick responses.

I wrote my central IT colleagues who offer this ACME endpoint a few hours ago regarding the deviation from the RFC asking about implementation details as it wasn't clear if this is an off-the-shelf product or not from our docs, as soon as I have more infos I will respond but this might take a day or two.
If this is custom built and caused by us I of course agree that this fix is to be done on our side, no questions asked.

Regarding the current regex, I agree that a workaround is ugly but the current regex is maybe not the most robust. Right now it doesn't really match for an URI but rather for not the stuff around the URI if you get what I mean.
Please don't get me wrong, the current solution works everytime the standard is followed and Regexes for URIs are ugly so this is quite elegant, but at the same time I'm still wondering if requests or acme.sh are programmed defensively on purpose or just by chance to work in this special case.

So as you said, let's wait for the feedback from my colleagues and decide then what to do, thanks again!

Edit on 12.11 Unfortunately no response yet, will try to reach him again on Monday.
Edit on 19.11 Talking with a colleague now, but not that many details yet.