Nixpkgs: ACME/LetsEncrypt is broken on unstable, works on 16.09

[nh2]

Issue description

I use nginx's enableACME setting to enable LetEncrypt for nginx:

    services.nginx = {
      enable = true;
      virtualHosts."example.com" = {
        enableACME = true;
        forceSSL = true;
      };
    };

This works on the nixos-16.09 channel, but on nixpkgs-unstable, it breaks, in 2 different ways on my server (from journalctl):

1) The request to letsencrypt hangs forever at Starting new HTTPS connection:

Jan 26 20:42:33 myhostname acme-example.com-start[10289]: 2017-01-26 20:42:33,292:DEBUG:simp_le:1371: ['-v', '-d', 'example.com', '--default_root', '/var/lib/acme/acme-challenge', '--valid_min', '2592000', '-f', 'fullchain.pem', '-f', 'key.pem', '-f', 'account ...
Jan 26 20:42:33 myhostname acme-example.com-start[10289]: 2017-01-26 20:42:33,292:DEBUG:simp_le:367: Loading fullchain.pem
Jan 26 20:42:33 myhostname acme-example.com-start[10289]: 2017-01-26 20:42:33,293:DEBUG:simp_le:367: Loading key.pem
Jan 26 20:42:33 myhostname acme-example.com-start[10289]: 2017-01-26 20:42:33,293:DEBUG:simp_le:367: Loading account_key.json
Jan 26 20:42:33 myhostname acme-example.com-start[10289]: 2017-01-26 20:42:33,293:DEBUG:simp_le:1203: Existing SANs: [], new: ['example.com']
Jan 26 20:42:33 myhostname acme-example.com-start[10289]: 2017-01-26 20:42:33,293:DEBUG:simp_le:1280: Computed roots: {'example.com': '/var/lib/acme/acme-challenge'}
Jan 26 20:42:33 myhostname acme-example.com-start[10289]: 2017-01-26 20:42:33,293:INFO:simp_le:1211: Generating new account key
Jan 26 20:42:33 myhostname acme-example.com-start[10289]: 2017-01-26 20:42:33,788:DEBUG:root:605: Sending GET request to https://acme-v01.api.letsencrypt.org/directory. args: (), kwargs: {}
Jan 26 20:42:33 myhostname acme-example.com-start[10289]: 2017-01-26 20:42:33,796:INFO:requests.packages.urllib3.connectionpool:805: Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org

2) An explicit error (happened only once in the entire systemctl):

Jan 26 20:46:48 myhostname acme-example.com-start[10289]: 2017-01-26 20:46:48,640:DEBUG:requests.packages.urllib3.connectionpool:401: "GET /directory HTTP/1.1" 200 352
Jan 26 20:46:48 myhostname acme-example.com-start[10289]: 2017-01-26 20:46:48,641:DEBUG:root:611: Received <Response [200]>. Headers: {'Content-Length': '352', 'Expires': 'Thu, 26 Jan 2017 20:46:48 GMT', 'Boulder-Request-Id': ...
Jan 26 20:46:48 myhostname acme-example.com-start[10289]: 2017-01-26 20:46:48,641:DEBUG:acme.client:551: Received response <Response [200]> (headers: {'Content-Length': '352', 'Expires': 'Thu, 26 Jan 2017 20:46:48 GMT', 'Boulder-Request-Id': ...
Jan 26 20:46:48 myhostname acme-example.com-start[10289]: Traceback (most recent call last):
Jan 26 20:46:48 myhostname acme-example.com-start[10289]:   File "/nix/store/7vjpm9g24gx80v0qgz2r9k59d9r3k7h9-simp_le-2016-04-17/lib/python2.7/site-packages/simp_le.py", line 1401, in main
Jan 26 20:46:48 myhostname acme-example.com-start[10289]:     return main_with_exceptions(cli_args)
Jan 26 20:46:48 myhostname acme-example.com-start[10289]:   File "/nix/store/7vjpm9g24gx80v0qgz2r9k59d9r3k7h9-simp_le-2016-04-17/lib/python2.7/site-packages/simp_le.py", line 1386, in main_with_exceptions
Jan 26 20:46:48 myhostname acme-example.com-start[10289]:     persist_new_data(args, existing_data)
Jan 26 20:46:48 myhostname acme-example.com-start[10289]:   File "/nix/store/7vjpm9g24gx80v0qgz2r9k59d9r3k7h9-simp_le-2016-04-17/lib/python2.7/site-packages/simp_le.py", line 1282, in persist_new_data
Jan 26 20:46:48 myhostname acme-example.com-start[10289]:     client = registered_client(args, existing_data.account_key)
Jan 26 20:46:48 myhostname acme-example.com-start[10289]:   File "/nix/store/7vjpm9g24gx80v0qgz2r9k59d9r3k7h9-simp_le-2016-04-17/lib/python2.7/site-packages/simp_le.py", line 1224, in registered_client
Jan 26 20:46:48 myhostname acme-example.com-start[10289]:     client = acme_client.Client(directory=args.server, key=key, net=net)
Jan 26 20:46:48 myhostname acme-example.com-start[10289]:   File "/nix/store/71zbxyggk7pd125sphxk799k3s0x6547-python2.7-acme-0.5.0/lib/python2.7/site-packages/acme/client.py", line 63, in __init__
Jan 26 20:46:48 myhostname acme-example.com-start[10289]:     self.net.get(directory).json())
Jan 26 20:46:48 myhostname acme-example.com-start[10289]:   File "/nix/store/71zbxyggk7pd125sphxk799k3s0x6547-python2.7-acme-0.5.0/lib/python2.7/site-packages/acme/messages.py", line 169, in from_json
Jan 26 20:46:48 myhostname acme-example.com-start[10289]:     raise jose.DeserializationError(str(error))
Jan 26 20:46:48 myhostname acme-example.com-start[10289]: DeserializationError: Deserialization error: Wrong directory fields
Jan 26 20:46:48 myhostname acme-example.com-start[10289]: Unhandled error has happened, traceback is above
Jan 26 20:46:48 myhostname acme-example.com-start[10289]: Debugging tips: -v improves output verbosity. Help is available under --help.

I've found a couple of LetsEncrypt related results when googling DeserializationError: Deserialization error: Wrong directory fields, but it's not clear to me what the problem is.
Maybe a newer version of the acme Python library broke it?

[k0001]

I recently saw this issue when I was using nixpkgs version 210b3b3184b27be8597f320fc9f337d3997dce94 (nixos-16.09 from some weeks ago). Upgrading to a newer nixos-16.09 fixed it.

[rnhmjoj]

Uhm, I have just finished setting up a server with nginx and let's encrypt with no problem whatsoever. I'm at f47757aaf03dd43cc953bdcb30feb015e94bf22a though.

[fpletz]

We are also seeing these weird errors from time to time but are not sure why they are happening. But we believe that this is a problem with the Let's Encrypt service itself because these errors are transient.

[nh2]

What I saw was definitely not transient, and switching to 16.09 immediately fixed it.

[domenkozar]

Isn't this fixed since https://github.com/NixOS/nixpkgs/pull/21102

[fpletz]

Oh yeah, @nh2 is still at acme 0.5.0.

[nh2]

So my nixpkgs master was too old? That is possible.

+1 to the full-deterministic-build-path-with-version-in-strack-traces concept :)

[globin]

I guess this can be closed then :)