Docker-letsencrypt-nginx-proxy-companion: Recent changes to the nginx.tmpl in jwilder/nginx-proxy cause certificate validation to fail

Created on 16 Sep 2016 · 17Comments · Source: nginx-proxy/docker-letsencrypt-nginx-proxy-companion

Hey,

when I use the suggested nginx.tmpl file (https://github.com/jwilder/nginx-proxy/blob/master/nginx.tmpl) The certificate validation fails.

letsencrypt-companion                | Creating Diffie-Hellman group (can take several minutes...)
letsencrypt-companion                | Generating DH parameters, 2048 bit long safe prime, generator 2
letsencrypt-companion                | This is going to take a long time
letsencrypt-companion                | ............................................................................................................................................+.......................................................................................................................................................................................................+....+...........................................................................+...........................................................+.+...........................................+..................................................................................................................................................................+.........................................................................................................................+....+......................................................................+...........................................+........................................................................................+....................................................................................+...................+.........................................................................................................................+................................+..............................................................+........+........................................................................................................................................+....................+.........................................................................+.................+...+........+..................................................................+........+..............................................+.....................................................................................................................................................+.............................................................................................................................................................+......................................................................................................................................................................................................+..............................+...................+............................++*++*
letsencrypt-companion                | Sleep for 3600s
letsencrypt-companion                | 2016/09/16 08:53:14 Generated '/app/letsencrypt_service_data' from 4 containers
letsencrypt-companion                | 2016/09/16 08:53:14 Running '/app/update_certs'
letsencrypt-companion                | 2016/09/16 08:53:14 Watching docker events
letsencrypt-companion                | Reloading nginx proxy (using separate container nginx-gen)...
letsencrypt-companion                | Creating/renewal sub.domain.tld certificates... (sub.domain.tld)
letsencrypt-companion                | 2016/09/16 08:53:14 Contents of /app/letsencrypt_service_data did not change. Skipping notification '/app/update_certs'
letsencrypt-companion                | 2016-09-16 08:53:14,695:INFO:simp_le:1211: Generating new account key
letsencrypt-companion                | 2016-09-16 08:53:15,942:INFO:requests.packages.urllib3.connectionpool:756: Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
letsencrypt-companion                | 2016-09-16 08:53:16,150:INFO:requests.packages.urllib3.connectionpool:756: Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
letsencrypt-companion                | 2016-09-16 08:53:16,365:INFO:requests.packages.urllib3.connectionpool:756: Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
letsencrypt-companion                | 2016-09-16 08:53:17,319:INFO:requests.packages.urllib3.connectionpool:756: Starting new HTTPS connection (1): letsencrypt.org
letsencrypt-companion                | 2016-09-16 08:53:17,743:INFO:requests.packages.urllib3.connectionpool:756: Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
letsencrypt-companion                | 2016-09-16 08:53:17,981:INFO:requests.packages.urllib3.connectionpool:756: Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
letsencrypt-companion                | 2016-09-16 08:53:18,230:INFO:requests.packages.urllib3.connectionpool:207: Starting new HTTP connection (1): sub.domain.tld
letsencrypt-companion                | 2016-09-16 08:53:18,235:WARNING:simp_le:1303: sub.domain.tld was not successfully self-verified. CA is likely to fail as well!
letsencrypt-companion                | 2016-09-16 08:53:18,246:INFO:requests.packages.urllib3.connectionpool:756: Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
letsencrypt-companion                | 2016-09-16 08:53:18,484:INFO:simp_le:1313: Generating new certificate private key
letsencrypt-companion                | 2016-09-16 08:53:19,097:INFO:requests.packages.urllib3.connectionpool:756: Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
letsencrypt-companion                | 2016-09-16 08:53:19,310:ERROR:simp_le:1271: CA marked some of the authorizations as invalid, which likely means it could not access http://example.com/.well-known/acme-challenge/X. Did you set correct path in -d example.com:path or --default_root? Is there a warning log entry about unsuccessful self-verification? Are all your domains accessible from the internet? Failing authorizations: https://acme-v01.api.letsencrypt.org/acme/authz/9P0PeTuGE9OXnehkgR89IZB97m3Ubhf4Vwo-sRpb-GY
letsencrypt-companion                | Challenge validation has failed, see error log.
letsencrypt-companion                |
letsencrypt-companion                | Debugging tips: -v improves output verbosity. Help is available under --help.
letsencrypt-companion                | Sleep for 3600s

An older version of the template (e.g. https://gist.github.com/SnowMB/87c5360f9bf81925af26c31f6d71410e) works just fine.

Anybody an idea what causes the problem and how to fix it?

Source

SnowMB

Most helpful comment

@domdorn It finally works! But I have to change the line
root /usr/share/nginx/html/.well-known/;
to
root /usr/share/nginx/html/;

Nierrrrrrr on 8 Mar 2017

👍2

All 17 comments

If you use the _blob_ link to download the file, you get HTML. Try https://github.com/jwilder/nginx-proxy/raw/master/nginx.tmpl instead?

almereyda on 28 Sep 2016

Yeah sure ;).

A few additional Infos:

I tried using the "recommended" setup with nginx and docker-gen in two seperate containers.
I defined no external network since all relevant containers are in the same docker-file
I user docker-compose version 2

I made some changes to the template file and found out that these lines return no values. This leaves me with and empty _upstream_ block in the config.

In the older file these lines are not present.

SnowMB on 29 Sep 2016

You need to open an issue in nginx-proxy repository as is not relate to letsencrypt container.
Perhaps because of a specific docker version ?

JrCs on 30 Sep 2016

I also had a issue with the nginx.tmpl file, not sure if it was the same, but it was fixed with using the file on the docker-gen repo. So for example i used the latest (https://raw.githubusercontent.com/jwilder/docker-gen/master/templates/nginx.tmpl) as i am also using the latest version of the docker image (0.7.3).
Maybe the link in the documentation should be changed to the one on the docker-gen repo as the example works with that docker image and not with the nginx-proxy image.

FallenRiteMonk on 20 Oct 2016

Hey,

I think the template file in the docker-gen repo is a little outdated. Anyway I found a fix for the problem. See https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion/issues/88.

SnowMB on 20 Oct 2016

👍1

That didn't work for me!

FallenRiteMonk on 20 Oct 2016

I had to adjust my nginx.tmpl to don't proxy .well-known/acme-challenge through to the child-containers. Did it like this:
`
server {
server_name {{ $host }};
listen 80 {{ $default_server }};
access_log /var/log/nginx/access.log vhost;

    {{ if (exists (printf "/etc/nginx/vhost.d/%s" $host)) }}
    include {{ printf "/etc/nginx/vhost.d/%s" $host }};
    {{ else if (exists "/etc/nginx/vhost.d/default") }}
    include /etc/nginx/vhost.d/default;
    {{ end }}

added code here START

    location /.well-known/acme-challenge {
            root /usr/share/nginx/html/.well-known/;
            break;
    }

added code here END

    location / {
            proxy_pass {{ trim $proto }}://{{ trim $host }};
            {{ if (exists (printf "/etc/nginx/htpasswd/%s" $host)) }}
            auth_basic      "Restricted {{ $host }}";
            auth_basic_user_file    {{ (printf "/etc/nginx/htpasswd/%s" $host) }};
            {{ end }}
            {{ if (exists (printf "/etc/nginx/vhost.d/%s_location" $host)) }}
            include {{ printf "/etc/nginx/vhost.d/%s_location" $host}};
            {{ else if (exists "/etc/nginx/vhost.d/default_location") }}
            include /etc/nginx/vhost.d/default_location;
            {{ end }}
    }

}

domdorn on 14 Dec 2016

👍2 😕1

@domdorn It finally works! But I have to change the line
root /usr/share/nginx/html/.well-known/;
to
root /usr/share/nginx/html/;

Nierrrrrrr on 8 Mar 2017

👍2

@domdorn that solved failing certificate validations for me as well, many thanks.

buchdag on 26 Mar 2017

@domdorn Solution fixed me up as well. Seems to only affect port 80 services as mentioned in #182

Needed @Nierrrrrrr 's adjustment.

Routhinator on 27 Mar 2017

Hi, guys!

I still have problems with verifying certificate... I tried both options:

location /.well-known/acme-challenge - in the nginx.tmpl
location /.well-known/acme-challenge - in the vhost.d/sub.domain.tld

Files generated in the .well-known/acme-challenge can be accessed by me in the browser, but the script still has some problems :(

I include some logs (interesting case...):

# it works... file can be accessed in the browser
nginx-proxy               | nginx.1    | sub.domain.tld 192.168.1.1 - - [29/Mar/2017:19:55:33 +0000] "GET /.well-known/acme-challenge/smth HTTP/1.1" 200 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"

# new file is generated by docker-letsencrypt-nginx-proxy-companion
nginx_proxy-lets_encrypt  | 2017-03-29 19:57:01,007:DEBUG:simp_le:983: Saving validation (u'pfLLevTbGfHeMYx1yFuDQCCPaV-CjveX6-g_YJtvMPs.d0Sfyt3ze6Zkhpu4NDshs08QRwRaFxYxZGePeEK9irA') at /usr/share/nginx/html/.well-known/acme-challenge/pfLLevTbGfHeMYx1yFuDQCCPaV-CjveX6-g_YJtvMPs
nginx_proxy-lets_encrypt  | 2017-03-29 19:57:01,012:DEBUG:acme.challenges:252: Verifying http-01 at http://sub.domain.tld/.well-known/acme-challenge/pfLLevTbGfHeMYx1yFuDQCCPaV-CjveX6-g_YJtvMPs...
nginx_proxy-lets_encrypt  | 2017-03-29 19:57:01,014:INFO:requests.packages.urllib3.connectionpool:207: Starting new HTTP connection (1): sub.domain.tld

# new file can be accessed in the browser...
nginx-proxy               | nginx.1    | sub.domain.tld 192.168.1.1 - - [29/Mar/2017:19:57:21 +0000] "GET /.well-known/acme-challenge/pfLLevTbGfHeMYx1yFuDQCCPaV-CjveX6-g_YJtvMPs HTTP/1.1" 200 87 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"

# docker-letsencrypt-nginx-proxy-companion unfortunately cannot access the file :(
nginx_proxy-lets_encrypt  | 2017-03-29 19:59:08,420:ERROR:acme.challenges:256: Unable to reach http://sub.domain.tld/.well-known/acme-challenge/pfLLevTbGfHeMYx1yFuDQCCPaV-CjveX6-g_YJtvMPs: HTTPConnectionPool(host='sub.domain.tld', port=80): Max retries exceeded with url: /.well-known/acme-challenge/pfLLevTbGfHeMYx1yFuDQCCPaV-CjveX6-g_YJtvMPs (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7efe9b0bac50>: Failed to establish a new connection: [Errno 110] Operation timed out',))
nginx_proxy-lets_encrypt  | 2017-03-29 19:59:08,422:WARNING:simp_le:1303: sub.domain.tld was not successfully self-verified. CA is likely to fail as well!

# right after previous companion logs, this log appears in the nginx logs: (200 status...)
nginx-proxy               | nginx.1    | sub.domain.tld 192.168.1.1 - - [29/Mar/2017:19:59:09 +0000] "GET /.well-known/acme-challenge/pfLLevTbGfHeMYx1yFuDQCCPaV-CjveX6-g_YJtvMPs HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

I completely have no idea, what's wrong :/ I've spent already many hours trying to fix that...
Do you have any ideas? Thanks in advance for any clues.

here is my docker-compose.yml file: https://gist.github.com/anonymous/2083cc1d10fefc334cd8fc8dce6b0440
and here is my nginx.tmpl file: https://gist.github.com/anonymous/4b689be2ccf605e9cedabc58a27b1f56

tomajask on 29 Mar 2017

@tomajask > Hi ! jwilder/nginx-proxy and jwilder/docker-gen are not supposed to be used together, the later is meant to be used with the official nginx image : https://github.com/jwilder/nginx-proxy#separate-containers

@Routhinator > I had that issue with non port 80 services as well, maybe because i wasn't using the "VIRTUAL_PORT" environment variable (I believe I wasn't supposed to, as the container was exposing onyl one port) ? In thoses cases, @domdorn modification of nginx.tmpl solved it, but the one suggested by @Nierrrrrrr did not.

buchdag on 31 Mar 2017

We ran into the same problem while setting up the companion on our server. @domdorn 's modification fixed it for us with the .well-known removed from @domdorn 's modification suggestion as pointed out by @Nierrrrrrr - adding "VIRTUAL_PORT" on the other hand to the environment for all containers did NOT work for us.

(so we couldn't manage to get this to work without modifying the nginx.tmpl by docker-gen)

Is there any chance this issue will be looked at in the near future? It seems with the current jwilder/docker-gen nginx.tmpl combined with the regular latest nginx image combined with this let's encrypt companion in the current version, people are bound to run into this problem.

etc0de on 31 Mar 2017

i encountered a LE validation failure on renewal due to timeout on the http link (and on a renewal the http port is redirected to https port) and discovered this issue, that could be (or not) the same cause, so i share as this issue is still opened:
May be the same cause than for the issue #178 (and the PR #192 resolved the problem for me, and @JonasT , without modifying the nginx.tmpl file)

basically, the LE endpoint wasn't taking into account, and the LE request was forwarded to my webapp.