Boulder: PA rejects domains ending in IDN TLD

[pfigel]

PA currently rejects domains ending in IDN TLDs (such as example.xn--fiqs8s). This appears to be due to publicsuffix-go's Find expecting suffixes to be in unicode rather than punycode and thus rejecting the domain with "Name does not end in a public suffix".

Originally reported here.

[ghost]

Log for reference:

[Sat Oct 22 02:49:03 EDT 2016] Getting new-authz for domain='wangqiliang.xn--fiqs8s'
[Sat Oct 22 02:49:03 EDT 2016] url='https://acme-v01.api.letsencrypt.org/acme/new-authz'
[Sat Oct 22 02:49:03 EDT 2016] payload='{"resource": "new-authz", "identifier": {"type": "dns", "value": "wangqiliang.xn--fiqs8s"}}'
[Sat Oct 22 02:49:03 EDT 2016] RSA key
[Sat Oct 22 02:49:04 EDT 2016] GET
[Sat Oct 22 02:49:04 EDT 2016] url='https://acme-v01.api.letsencrypt.org/directory'
[Sat Oct 22 02:49:04 EDT 2016] timeout
[Sat Oct 22 02:49:04 EDT 2016] _CURL='curl -L --silent --dump-header /root/.acme.sh/http.header '
[Sat Oct 22 02:49:05 EDT 2016] ret='0'
[Sat Oct 22 02:49:05 EDT 2016] POST
[Sat Oct 22 02:49:05 EDT 2016] url='https://acme-v01.api.letsencrypt.org/acme/new-authz'
[Sat Oct 22 02:49:05 EDT 2016] _CURL='curl -L --silent --dump-header /root/.acme.sh/http.header '
[Sat Oct 22 02:49:05 EDT 2016] _ret='0'
[Sat Oct 22 02:49:05 EDT 2016] code='400'
[Sat Oct 22 02:49:05 EDT 2016] new-authz error: {"type":"urn:acme:error:malformed","detail":"Name does not end in a public suffix","status": 400}

[Sat Oct 22 02:54:09 EDT 2016] Getting new-authz for domain='wangqiliang.中国'
[Sat Oct 22 02:54:09 EDT 2016] url='https://acme-v01.api.letsencrypt.org/acme/new-authz'
[Sat Oct 22 02:54:09 EDT 2016] payload='{"resource": "new-authz", "identifier": {"type": "dns", "value": "wangqiliang.中国"}}'
[Sat Oct 22 02:54:09 EDT 2016] RSA key
[Sat Oct 22 02:54:10 EDT 2016] GET
[Sat Oct 22 02:54:10 EDT 2016] url='https://acme-v01.api.letsencrypt.org/directory'
[Sat Oct 22 02:54:10 EDT 2016] timeout
[Sat Oct 22 02:54:10 EDT 2016] _CURL='curl -L --silent --dump-header /root/.acme.sh/http.header '
[Sat Oct 22 02:54:10 EDT 2016] ret='0'
[Sat Oct 22 02:54:10 EDT 2016] POST
[Sat Oct 22 02:54:10 EDT 2016] url='https://acme-v01.api.letsencrypt.org/acme/new-authz'
[Sat Oct 22 02:54:10 EDT 2016] _CURL='curl -L --silent --dump-header /root/.acme.sh/http.header '
[Sat Oct 22 02:54:11 EDT 2016] _ret='0'
[Sat Oct 22 02:54:11 EDT 2016] code='400'
[Sat Oct 22 02:54:11 EDT 2016] new-authz error: {"type":"urn:acme:error:malformed","detail":"Invalid character in DNS name","status": 400}

[kachkaev]

Same here for a domain in .рф zone defined by me as .xn--p1ai:

ACME server returned an error: urn:acme:error:malformed :: The request message was malformed :: Name does not end in a public suffix

[cpu]

This is blocked on investigation by the upstream publicsuffix-go dependency: https://github.com/weppos/publicsuffix-go/issues/31

[SDKiller]

Dependency of library which has only 8 stars is blocking you?
Or, if to look at the list itself - 182 stars, random updates, pending issues.

In present days number of new TLDs is increasing rapidly and issues will only accumulate.

Finally - what is the sense to check domain suffix - you are checking /.well-known/acme-challenge anyway? If domain is incorrect - it will fail.

[alexstep]

You have really bug in IDN domain zones.
You service require domain name in punnycode, but check domain zone by the unicode list from "eightstartslibrary".

Example:
publicsuffix-go - resolve "рф", but you reject "пример.рф"
you resolve xn--e1afmkfd.xn--p1ai , but publicsuffix-go reject xn--p1ai

[ghost]

@SDKiller I have abandoned the HTTP-01 method three months ago and I'm now using the DNS-01 Method to verify my domain name. So there will be no problem in /.well-known/acme-challenge anyway.
The issue cares about that the PSL was encoded in UTF-8 while the IDN domains is actually encoded in Punycode defined in IDNA2008. So the bug is still to be solved.

[kachkaev]

Great to finally see this issue fixed!

@rolandshoemaker do I understand correctly that we don't need to update anything in order to start getting certs for IDN domains? I've got docker-letsencrypt-nginx-proxy-companion on my server, which attempts to generate expiring or missing certs every hour. Should I just wait for some time and then magically see my website turning into an https one? :–)

[slavonnet]

When the update will appear on the production servers letsencrypt?

https://acme-v01.api.letsencrypt.org/acme/new-authz
is now still gives an error.We are very forward to updates.

[cpu]

@kachkaev - you're correct, nothing will have to change on your end once the update to Boulder is live in production.

@slavonnet No sooner than Dec 1st. There's no deploy this week due to the American Thanksgiving holidays. We do deploys to staging on Tuesdays and deploys to production on Thursdays.

[kachkaev]

@cpu did you mean 1 Dec? (that's next Thursday)

[cpu]

@kachkaev Oops! Yes I meant Dec 1st. I edited my comment above. Apologies for the confusion.

[rqou]

I just tried a domain ending in .xn--j6w193g (.香港) against the _staging_ server and I still get "Name does not end in a public suffix." Wasn't there supposed to be a deploy yesterday?

[cpu]

@rqou There were unrelated problems that resulted in the staging release being reverted. You can see the Boulder revision running in staging is +6e85092 which does not include the suffix fix.

[kachkaev]

@cpu could you please let us know if (or when) the patch is in prod? I checked one my domain in .рф a couple of times today and it was still unencrypted (I've got docker-letsencrypt-nginx-proxy-companion on the server, so https should become available automatically within one hour from a fix).

Excited about this change you're making and happy to be one of the Letsencrypt supporters. Suggest others to donate too so that guys could keep the certs free further! https://letsencrypt.org/donate/

[cpu]

@kachkaev Absolutely - I will update this thread when it reaches prod. We appreciate your patience! I promise we're almost there :-)

[rqou]

I just tried an IDN TLD on staging today and it works! Will this deploy to prod on Thursday?

[jsha]

That's the plan!

[ob-tk]

We are still waiting?

[cpu]

@TimKGS Yes - we will comment on the ticket when it's done. You can also watch status.letsencrypt.org

[cpu]

@kachkaev @slavonnet @TimKGS @rqou, others - the update in production is complete. Can you please verify you're now able to issue for IDN TLDs without error?

[magiruuvelvet]

Can verify that it finally works. Successfully got my certs for .コム and .닷컴 without problems :+1:

[zeeZ]

.みんな success (with some more punycode in the domain name, using webroot lego)

[ob-tk]

Domain .рф success (using ISP Manager, Let's Encrypt plugin)
Thanks

[rqou]

Worked for .xn--j6w193g (.香港) using a custom client.

[kachkaev]

As a user of popular docker-letsencrypt-nginx-proxy-companion I'm still unable to get a cert for .xn--p1ai (i.e. .рф). But that's likely to be this issue: https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion/issues/130

The error in my case says the following:

Creating/renewal xn--abcdef.xn--p1ai certificates... (xn--abcdef.xn--p1ai)
2016-12-08 23:09:07,317:INFO:simp_le:1211: Generating new account key
2016-12-08 23:09:08,186:INFO:requests.packages.urllib3.connectionpool:756: Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/simp_le.py", line 1401, in main
    return main_with_exceptions(cli_args)
  File "build/bdist.linux-x86_64/egg/simp_le.py", line 1386, in main_with_exceptions
    persist_new_data(args, existing_data)
  File "build/bdist.linux-x86_64/egg/simp_le.py", line 1282, in persist_new_data
    client = registered_client(args, existing_data.account_key)
  File "build/bdist.linux-x86_64/egg/simp_le.py", line 1224, in registered_client
    client = acme_client.Client(directory=args.server, key=key, net=net)
  File "build/bdist.linux-x86_64/egg/acme/client.py", line 63, in __init__
    self.net.get(directory).json())
  File "build/bdist.linux-x86_64/egg/acme/messages.py", line 169, in from_json
    raise jose.DeserializationError(str(error))
DeserializationError: Deserialization error: Wrong directory fields

Unhandled error has happened, traceback is above

Debugging tips: -v improves output verbosity. Help is available under --help.
Sleep for 3600s

If anyone knowledgable can help with a PR in https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion, that'd be fantastic!

Really exited the Letsencrypt is finally available for IDNs, according to other comments! That's a great day to celebrate! 🎉 🎉 🎉

[jsha]

@kachkaev your issue sounds like it's related to an out-of-date certbot version. It's unrelated to this issue, but if you file an issue on that other repo and tag me I can help out.

[schoen]

While I know there's nothing more technically difficult about supporting one writing system as opposed to another within IDNs, it's still heartwarming to see that people just commenting on this single issue succeeded in getting certs issued for names in five different scripts on the first day!

[ghost]

Worked with .xn--fiqs8s and .xn--fiqz9s (.中国 & .中國) with an acme.sh client.

[ghost]

Worked with .xn--fiqs8s and .xn--fiqz9s (.中国 & .中國) with an acme.sh client.