If a client requests certificates for example.com and blog.example.com;
The client can:
example.com and blog.example.comexample.comexample.comblog.example.comHowever for the case of example.com and *.example.com, the client cannot follow the same steps.
In this case, the steps are;
example.com and *.example.com example.com*.example.comexample.com*.example.comThe differences between the two have led to some bugs in various client implementations, eg[4][5]
It would be nice if boulder would support the first flow for a case of example.com and *.example.com.
This would be possible if it validated dns records at
_acme-challenge.example.com and
_acme-challenge.*.example.com respectively
ref:
1 https://tools.ietf.org/html/draft-ietf-acme-acme#section-7.4
this is more of a proposal than a bug report, you can close this issue if you disagree substantially with it.
Hi @komuw I'm not sure I understand your proposal. Can you clarify for me?
If I'm understanding correctly there shouldn't be any reason why you can't fulfill the individual authorizations for a V2 order with a wildcard identifier individually one at a time so long as you ensure that you provision the correct TXT value for the specific challenge being POSTed.
Can you provide a detailed log of the ACME messages you're POSTing to the server and the messages you're receiving back in a case where you try to handle a wildcard order the same way as a non-wildcard order w.r.t challenge fulfillment?
hi @cpu
there shouldn't be any reason why you can't fulfill the individual authorizations for a V2 order with a wildcard identifier individually one at a time so long as you ensure that you provision the correct TXT value for the specific challenge being POSTed.
I have been unable to do that.
As an example,
Here's my client's user agent: 'User-Agent': 'python-requests/2.18.4 (Linux: x86_64) sewer 0.5.0.1 (https://github.com/komuw/sewer)'
I applied for a certificate for amqphosting.com and *.amqphosting.com and got
apply_for_cert_issuance_response. status_code=201.
response={'status': 'pending', 'identifiers': [{'type': 'dns', 'value': '*.amqphosting.com'}, {'type': 'dns', 'value': 'amqphosting.com'}], 'finalize': 'https://acme-staging-v02.api.letsencrypt.org/acme/finalize/5768653/97296', 'authorizations': ['https://acme-staging-v02.api.letsencrypt.org/acme/authz/Qd0dsrRh7WYI2NsotRkEPwDY4ePKhGUj48KuQPzA7HQ', 'https://acme-staging-v02.api.letsencrypt.org/acme/authz/G3C_FcfpkeyU1WMtQHDivH6cU-tHxDoBk5opRjsHTXI'], 'expires': '2018-03-26T16:09:25.389840288Z'}
I asked for challenges for wildcard domain(*.amqphosting.com) and got response:
get_challenge_response. status_code=200.
response={'status': 'pending', 'challenges': [{'type': 'dns-01', 'status': 'pending', 'url': 'https://acme-staging-v02.api.letsencrypt.org/acme/challenge/Qd0dsrRh7WYI2NsotRkEPwDY4ePKhGUj48KuQPzA7HQ/110509036', 'token': 'j_bA60oVjwEJVD-VbQOzrloupFncBubXw7wfgDxLWTI'}], 'expires': '2018-03-26T16:09:25Z', 'wildcard': True, 'identifier': {'type': 'dns', 'value': 'amqphosting.com'}}
responded to challenges for wildcard domain:
request_to_acme.
url=https://acme-staging-v02.api.letsencrypt.org/acme/challenge/Qd0dsrRh7WYI2NsotRkEPwDY4ePKhGUj48KuQPzA7HQ/110509036.
request_body={"signature": "UtnxC3J9tCKfd3f2tyNLSAtmsH1IGMb8cH8rfO6MwiVO0WMbqDg_D5waPd_160SUukexOzII-XNOmXc316u20t_5HidD0vbofY4xiw8wKEl_QlzgkD0Sco2_wSxdYSsPcb8PuwtWgsG5t4tOrB9hXCaxaJuEMrrPy0gnq4qRVMsREFVdGHwRij8WOFw02i3SPxMNOGpHzZXl0n9VR9rP_jpQpDxhT3Q_8p22OtpR6qt4n91P0XfohI2Zbz2Pvw5WwFCgC4e6Aa3brPUT3cNNWvXBJN1h2uU1he-2p4cOKbXRKDyNoP5exXn2DZeSahVa6AjF1sJYtlTJXXWaqnvHeg", "payload": "eyJrZXlBdXRob3JpemF0aW9uIjogImpfYkE2MG9WandFSlZELVZiUU96cmxvdXBGbmNCdWJYdzd3ZmdEeExXVEkuR2lEdmtWUlFWSzBVX0dTYS1oRE1oMXloUmxzMUY3TUZ5bG1jWlQtN3dXQSJ9", "protected": "eyJhbGciOiAiUlMyNTYiLCAidXJsIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvY2hhbGxlbmdlL1FkMGRzclJoN1dZSTJOc290UmtFUHdEWTRlUEtoR1VqNDhLdVFQekE3SFEvMTEwNTA5MDM2IiwgImtpZCI6ICJodHRwczovL2FjbWUtc3RhZ2luZy12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvNTc2ODY1MyIsICJub25jZSI6ICJLdlpyVnFHZGlQSzRuUV9BNDVmdTVQeE9LR2loTVMyRnYwbGk4d1NiUmdzIn0"}.
request_headers={'User-Agent': 'python-requests/2.18.4 (Linux: x86_64) sewer 0.5.0.1 (https://github.com/komuw/sewer)', 'Content-Type': 'application/jose+json'}
Acme replied with:
respond_to_challenge_response. status_code=200.
response={'type': 'dns-01', 'status': 'pending', 'url': 'https://acme-staging-v02.api.letsencrypt.org/acme/challenge/Qd0dsrRh7WYI2NsotRkEPwDY4ePKhGUj48KuQPzA7HQ/110509036', 'token': 'j_bA60oVjwEJVD-VbQOzrloupFncBubXw7wfgDxLWTI'}
````
I asked for challenges for NON wildcard domain(`amqphosting.com`) and got response:
```bash
get_challenge_response. status_code=200.
response={'status': 'pending', 'challenges': [{'type': 'dns-01', 'status': 'pending', 'url': 'https://acme-staging-v02.api.letsencrypt.org/acme/challenge/G3C_FcfpkeyU1WMtQHDivH6cU-tHxDoBk5opRjsHTXI/110509037', 'token': 'KHDvIKC0KoDVuoT0iVwvLlYFcaNfEUSaVFm5yuJSHJE'}, {'type': 'http-01', 'status': 'pending', 'url': 'https://acme-staging-v02.api.letsencrypt.org/acme/challenge/G3C_FcfpkeyU1WMtQHDivH6cU-tHxDoBk5opRjsHTXI/110509038', 'token': 'pDDOMkvqYPx3u9ZbZfVT2XibWwowMtJ6p4ot0g-bukc'}], 'expires': '2018-03-26T16:09:25Z', 'identifier': {'type': 'dns', 'value': 'amqphosting.com'}}
responded to challenges for NON wildcard domain:
request_to_acme.
url=https://acme-staging-v02.api.letsencrypt.org/acme/challenge/G3C_FcfpkeyU1WMtQHDivH6cU-tHxDoBk5opRjsHTXI/110509037.
request_body={"signature": "Dol4gHTWs70l8-HoYYtJvZar18Q_VQvxpc8qFQpIo-tFt8kyRqslkN-YqQsPiq2QvnyqKFn4jnt4lO5eBqea6ssmsQKIMeQOkqSV07rp-ctg0E38yJ0NBxkcNSIfndwneXy-incPz2_l_r6wvGILv32jas6NwgxyG0AurZg__T_OZo4CgrBRNp8V7ygzapXTfOOhZK5oGofq-1U1CciDaKgwM8QwmQWuFUefqGH5xvVnvRtwxHdgoP4k9k_8ZJzTpH4NQXOwSQ3fDetVC2th-NFL_bmOO7qB-UlQ5-FBwafdD6azG6iyOj-o2H3gg8cYrn7gLCbYNSjx699S9MZbLQ", "payload": "eyJrZXlBdXRob3JpemF0aW9uIjogIktIRHZJS0MwS29EVnVvVDBpVnd2TGxZRmNhTmZFVVNhVkZtNXl1SlNISkUuR2lEdmtWUlFWSzBVX0dTYS1oRE1oMXloUmxzMUY3TUZ5bG1jWlQtN3dXQSJ9", "protected": "eyJhbGciOiAiUlMyNTYiLCAidXJsIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvY2hhbGxlbmdlL0czQ19GY2Zwa2V5VTFXTXRRSERpdkg2Y1UtdEh4RG9CazVvcFJqc0hUWEkvMTEwNTA5MDM3IiwgImtpZCI6ICJodHRwczovL2FjbWUtc3RhZ2luZy12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvNTc2ODY1MyIsICJub25jZSI6ICJCOFNyTW8zLUMxbUVJYVRnMWc0VFhhRXJqWUFWTVM1QnRLQjM3LXZ3UDBZIn0"}.
request_headers={'User-Agent': 'python-requests/2.18.4 (Linux: x86_64) sewer 0.5.0.1 (https://github.com/komuw/sewer)', 'Content-Type': 'application/jose+json'}
````
Acme replied with:
```bash
respond_to_challenge_response.
status_code=200.
response={'type': 'dns-01', 'status': 'pending', 'url': 'https://acme-staging-v02.api.letsencrypt.org/acme/challenge/G3C_FcfpkeyU1WMtQHDivH6cU-tHxDoBk5opRjsHTXI/110509037', 'token': 'KHDvIKC0KoDVuoT0iVwvLlYFcaNfEUSaVFm5yuJSHJE'}
Finally sent a csr, acme replied with:
status_code=400. response={'type': 'urn:ietf:params:acme:error:malformed', 'status': 400, 'detail': 'Order\'s status ("invalid") was not pending'}
All this while the two DNS TXT records were available(and are still are):
dig TXT _acme-challenge.amqphosting.com @8.8.8.8
;; ANSWER SECTION:
_acme-challenge.amqphosting.com. 299 IN TXT "jWac_bdTRdpahPZCnN5WPyJeb1gDLWlimx9glw0zChM"
_acme-challenge.amqphosting.com. 299 IN TXT "HXVH9ikocBmYEjRPiDRUfPEon1e9xCV-Mk-Mcs__jKA"
I think you most likely have a bug in your client, since there are other implementations (like certbot-dns-route53) that put up two TXT records simultaneously, and are succesful at validating both challenges in parallel.
One possibility would be issues with propagation times and/or caching. For instance, if you pushed one TXT record, then waited a while before POSTing to the challenge, then pushed the other TXT records, then waited a while before POSTing the challenge, you might run into trouble. If you want to minimize problems with caching, you should set a very low TTL (like 1 second) on the TXT records you create.
Thanks,
I'll look at the TTL idea.
Thanks @komuw. I'm going to close this issue in the meantime since I agree with @jsha's assessment so far. There's nothing protocol or Boulder-implementation wise that I'm aware of that would prevent this from working. If in your further debugging you find otherwise we can always reopen the issue :-)
Thanks!
Hello @cpu @jsha ,
I have the same trouble as mentionned above.
Let me give you more details:
In Tr忙fik, we are using the library LEGO to allow our users to automatically generate ACME certificates.
To solve the DNS-01 challenge, LEGO gets the challenge values for all the domains (CN and SANs) from boulder and then it manages all the domains one per one.
For each of them, it follows this process:
Unfortunately, even if the TXT record exists with the right value, time to time, we detect that boulder returns an error when it have to check both a wildcard domain and its base domain name (Incorrect TXT record "$OLD_VALUE" found at _acme-challenge.notmyreal.site or No TXT record found at _acme-challenge.notmyreal.site).
After analyzing the problem, it seems to come from the TXT record propagation into the DNS Provider infrastructure.
After validating and cleaning the wildcard domain TXT record, even if LEGO detects that the base name domain TXT record has been created with the right value, boulder gets the older one or get no value.
In the way to understand the problem I made the tests described below (the domain is managed by an external DNS provider) :
name: _acme-challenge.notmyreal.site, value: foo, TTL: 1,dig -t txt _acme-challenge.notmyreal.site
; <<>> DiG 9.13.0 <<>> -t txt _acme-challenge.notmyreal.site
...
;; QUESTION SECTION:
;_acme-challenge.notmyreal.site. IN TXT
;; ANSWER SECTION:
_acme-challenge.notmyreal.site. 1 IN TXT "foo"
...
TEST OK
name: _acme-challenge.notmyreal.site, value: bar, TTL: 1,$ dig -t txt _acme-challenge.notmyreal.site
; <<>> DiG 9.13.0 <<>> -t _acme-challenge.notmyreal.site
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 19946
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;_acme-challenge.notmyreal.site. IN TXT
...
# Test KO
$ dig -t txt _acme-challenge.notmyreal.site
; <<>> DiG 9.13.0 <<>> -t txt _acme-challenge.notmyreal.site
...
;; QUESTION SECTION:
;_acme-challenge.notmyreal.site. IN TXT
;; ANSWER SECTION:
_acme-challenge.notmyreal.site. 1 IN TXT "foo"
_acme-challenge.notmyreal.site. 1 IN TXT "bar"
...
# TEST OK
$ dig -t txt _acme-challenge.notmyreal.site
...
;; QUESTION SECTION:
;_acme-challenge.notmyreal.site. IN TXT
;; ANSWER SECTION:
_acme-challenge.notmyreal.site. 1 IN TXT "bar"
_acme-challenge.notmyreal.site. 1 IN TXT "foo"
...
# TEST OK
$ dig -t txt _acme-challenge.notmyreal.site
...
;; QUESTION SECTION:
;_acme-challenge.notmyreal.site. IN TXT
;; ANSWER SECTION:
_acme-challenge.notmyreal.site. 1 IN TXT "foo"
...
# TEST KO
The results are flacky
I tried to fix the problem in LEGO but it's complicated: all the mechanism seems to be based on the one by one domain process.
That's why, I solved the problem in Tr忙fik by adding a retry mechanism when user want to validate both wildcard and base name domains.
But this solution is not perfect and I assume it can be solve if wildcard domains can have a specific name distinct from their own base name.
That's why I would like to know if there is a technical reason to use the same name or if it's possible to have different names (maybe adding the prefix _acme-challenge-wildcard instead of _acme-challenge).
Of course if you think we can change the behavior, it would be a pleasure to help you to implement the solution.
Many thanks for your awesome product!
re:
That's why, I solved the problem in Tr忙fik by adding a retry mechanism when user want to validate both wildcard and base name domains.
That is kind of what I had to implement in my Acme client[1] via Pull request[2]
And it seems like, that is what a number of clients also had to do[3]
I assume it can be solve if wildcard domains can have a specific name distinct from their own base name.
This would be ideal
Hi @nmengin
After analyzing the problem, it seems to come from the TXT record propagation into the DNS Provider infrastructure.
Indeed - that is usually the case. How are you verifying that both TXT records are present throughout all authoritative DNS servers before POSTing the challenge? Is there a chance one does not have both records yet? Does the DNS provider use anycast in a way that would make it difficult to externally validate that 100% of distinct servers have the record? Does the DNS provider off its own API for checking whether the records have been distributed?
I assume it can be solve if wildcard domains can have a specific name distinct from their own base name.
This isn't possible without changing the ACME protocol. Section 8.4 mandates the domain construction for DNS-01 validation:
The client constructs the validation domain name by prepending the label "_acme-challenge" to the domain name being validated
Boulder could allow one authorization for _acme-challenge.domain.com to be used for both the base identifier and the wildcard but that would require more special casing throughout the codebase that is avoided by universally requiring one authorization for one identifier. It's a big change that would touch core parts of Boulder, that seems to err on the side of performing less validation work for a powerful wildcard identifier, and would be primarily addressing problems that I believe can be addressed by DNS providers. It's a big ask and I'm not convinced its the right path forward.
Hello @cpu,
Many thanks for your answer.
How are you verifying that both TXT records are present throughout all authoritative DNS servers before POSTing the challenge?
I have launched a dig command and checked the answer. LEGO have checked the DNS propagation thanks to this function.
Does the DNS provider use anycast in a way that would make it difficult to externally validate that 100% of distinct servers have the record? Does the DNS provider off its own API for checking whether the records have been distributed?
Up to now, users have reported problems with DigitalOcean, DNSimple, Gandi and Gcloud.
I have not found an API entry to check the propagation but, at least Gandi, DNSimple and Gcloud have used anycast (not sure concerning DigitalOcean).
I understand that no perfect solution would be easy to implement, I hope the retry mechanism I created into Tr忙fik will be helpful for users.
Thank you for all the details.
I have launched a dig command and checked the answer. LEGO have checked the DNS propagation thanks to this function.
@nmengin Can you provide the full dig command(s) you are running with arguments? One dig command wouldn't be sufficient to check all of the authoritative DNS servers have the required records.
DigitalOcean uses anycast, yes. Google Cloud DNS has an API to check if a change has completed: https://cloud.google.com/dns/api/v1/changes/get
Hello @cpu ,
Please find below the dig commands and the entire results:
$ dig -t txt myreal.site
; <<>> DiG 9.13.0 <<>> -t txt myreal.site
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22486
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;myreal.site. IN TXT
;; ANSWER SECTION:
myreal.site. 1 IN TXT "foo"
myreal.site. 1 IN TXT "bar"
;; Query time: 100 msec
;; SERVER: MYSERVER_IP#53(MYSERVER_IP)
;; WHEN: today 08:52:47 CEST 2018
;; MSG SIZE rcvd: 71
$ dig -t txt myreal.site
; <<>> DiG 9.13.0 <<>> -t txt myreal.site
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31136
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;myreal.site. IN TXT
;; ANSWER SECTION:
myreal.site. 1 IN TXT "foo"
;; Query time: 83 msec
;; SERVER: MYSERVER_IP#53(MYSERVER_IP)
;; WHEN: today 08:52:47 CEST 2018
;; MSG SIZE rcvd: 54
$ dig -t txt myreal.site
; <<>> DiG 9.13.0 <<>> -t txt myreal.site
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60252
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;myreal.site. IN TXT
;; ANSWER SECTION:
myreal.site. 1 IN TXT "foo"
myreal.site. 1 IN TXT "bar"
;; Query time: 263 msec
;; SERVER: MYSERVER_IP#53(MYSERVER_IP)
;; WHEN: today 08:52:48 CEST 2018
;; MSG SIZE rcvd: 82
Hi @nmengin,
As mentioned those dig commands are not checking the authoritative DNS servers. You're talking to whatever the system's configured recursive resolver is. This isn't sufficient to know that all of the authoritative DNS servers have the required records. What is the actual domain name in use?
Going back to what you're describing happens in Lego:
For each of them, it follows this process:
Creating a TXT record with a custom TTL (value depends on the DNS provider used),
Validating the TXT record creation,
If the TXT record creation is validated in the allowed time, sending a request to boulder to make the challenge,
Cleaning the TXT record.
I would expect that the TXT records for all of the order's authorizations are provisioned ahead of POSTing the validation of one of the authorizations. If what you're describing is true the TTL of the first challenge record will be respected for up to 60s. Can you provision the records in parallel before POSTing the challenge or wait >60s between challenge authorization POSTs? Why does Lego use a per-DNS provider TTL? I would expect you'd want a very low ttl in all cases for DNS-01 challenge records.
Hello @cpu ,
As mentioned those dig commands are not checking the authoritative DNS servers.
If I add the option +nssearch into the dig command (dig -t txt _acmechallenge-myreal.site myreal.site +nssearch), I can see 3 servers descriptions.
What is the actual domain name in use?
I am sorry but I can not share the domain used for my test in a public comment, but I will be able to do it in a private discussion if necessary.
If what you're describing is true the TTL of the first challenge record will be respected for up to 60s
It is the current behavior (if I have understood correctly the algorithm).
Can you provision the records in parallel before POSTing the challenge or wait >60s between challenge authorization POSTs?
I guess this opened PR may implement this behavior. I tested it yesterday but nothing change for the specific use case wildcard + base name. I can may be work by modifying this PR.
Why does Lego use a per-DNS provider TTL?
Apparently, some DNS providers require a minimal TTL which can be very big...
If I add the option +nssearch into the dig command (dig -t txt _acmechallenge-myreal.site myreal.site +nssearch), I can see 3 servers descriptions.
I would still recommend digging directly against the authoritative DNS servers to rule out any effects from your system recursive resolver.
I guess this opened PR may implement this behavior. I tested it yesterday but nothing change for the specific use case wildcard + base name. I can may be work by modifying this PR.
I'm not sure what you mean by "nothing changes". The problem of mismatched TXT records persists? If both TXT records are present before any DNS-01 challenges are POSTed there should be no way for only one of the records to be returned and not the other and Boulder checks each value.
I would recommend you take the code in the Lego PR you linked and add a very long sleep or a pause for input on line 599. When the client is sleeping/paused after adding records but before posting challenges you should dig each of the authoritative nameservers for the domain in question for _acme-challenge.domain.com TXT. Every authoritative nameserver should return the two expected TXT values. If both records are present and proceeding with the validation fails then we can dig further with the concrete domain name in question.
Hello @cpu ,
The problem of mismatched TXT records persists?
Yes it is.
If both TXT records are present before any DNS-01 challenges are POSTed there should be no way for only one of the records to be returned and not the other and Boulder checks each value.
I agree with you and I did not understand why the problem persists.
I would recommend you take the code in the Lego PR you linked and add a very long sleep or a pause for input on line 599.
Once again, I agree with you but adding a sleep instruction seems to an ugly solution.
But I can test with it in a first time and propose a better solution in a second time.
Many thanks for all you explanations and the time you spent to analyze my question.
Once again, I agree with you but adding a sleep instruction seems to an ugly solution. But I can test with it in a first time and propose a better solution in a second time.
To be clear I'm not suggesting this is a fix, just a next debugging step.
Hello @cpu,
By analyzing deeply the Pull Request I saw that the problem came from the CleanUp implementation of one of the DNS provider I have used.
I just proposed a modification to the author: https://github.com/captncraig/lego/pull/1.
Thanks to it it works well for me :smiley:
@nmengin Great, glad to hear it. Thanks for reporting back.