Docker-letsencrypt-nginx-proxy-companion: arm64 ACME Challenge Validation failure - nginx-proxy terminates anytime companion reloads nginx docker-gen and nginx

[phantomski77]

Hi,

I've spent a day trying to work out what I'm doing wrong and trying various builds and older versions, but it always ends up the same. Anytime docker-letsencrypt-nginx-proxy-companion detects a new container being started with the right env variables for hosts/port set (nextCloud in this case) it correctly initiates the new certificate generation, but that immediately causes nginx-proxy container to terminate, This subsequently leads to ACME challenge validation failure, as the companion container is not reachable by Let's Encrypt at 80/443 due to nginx-proxy not running.

Config:

• Raspberry Pi 4 8GB running Ubuntu 20.10 arm64 / kernel 5.8.0-1007-raspi
• Docker 19.03.13 (API: 1.40) with overlay2 storage driver
• Basic (2 container) setup
• nginx-proxy is traskit/ngix-proxy:latest (11152020) - arm64 build of jwilder/ngix-proxy. nginx is v1.18.0 and njs v0.4.4
• docker-letsencrypt-nginx-proxy-companion is jrcs/letsencrypt-nginx-proxy-companion:latest (1.13.1)
• both ports 80 and 443 are accessible internally and also from the internet

nginx-proxy is started first:

sudo docker run --detach --name nginx-proxy --publish 80:80 --publish 443:443 --volume nginx_data:/etc/nginx/certs --volume nginx_data:/etc/nginx/vhost.d --volume nginx_data:/usr/share/nginx/html --volume nginx_data:/etc/nginx/dhparam --volume /var/run/docker.sock:/tmp/docker.sock:ro traskit/nginx-proxy

I'll wait for DH to be generated, if I test ports 80/443 on local network and internet at this time, nginx-proxy correctly responds with 503, so ngix-proxy is running correctly and available on the network.

letsencrypt-nginx-proxy-companion is started second (with debug to observe log):

sudo docker run --detach --name nginx-proxy-letsencrypt --volumes-from nginx-proxy --volume /var/run/docker.sock:/var/run/docker.sock:ro --env "[email protected]" --env "NGINX_DOCKER_GEN_CONTAINER=nginx-proxy" --env "DEBUG=true" jrcs/letsencrypt-nginx-proxy-companion

At this point, the companion kills the nginx-proxy:

dockergen.1 | 2020/11/17 16:49:24 Received event die for container ad7da93077fc
dockergen.1 | 2020/11/17 16:49:24 Received event stop for container ad7da93077fc
dockergen.1 | 2020/11/17 16:49:24 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
dockergen.1 | 2020/11/17 16:49:25 Received event start for container ad7da93077fc
dockergen.1 | 2020/11/17 16:49:25 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
dockergen.1 | 2020/11/17 16:49:25 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
forego      | sending SIGTERM to dockergen.1
forego      | sending SIGTERM to nginx.1
dockergen.1 | 2020/11/17 16:49:27 Received signal: terminated
dockergen.1 | 2020/11/17 16:49:27 Received signal: terminated
forego      | Killing nginx.1

companion log shows (times irrelevant, older log):

Info: Custom Diffie-Hellman group found, generation skipped.,
Reloading nginx docker-gen (using separate container nginx-proxy)...,
Reloading nginx (using separate container nginx-proxy)...,
2020/11/17 15:34:49 Generated '/app/letsencrypt_service_data' from 2 containers,
2020/11/17 15:34:49 Running '/app/signal_le_service',
2020/11/17 15:34:49 Watching docker events,
2020/11/17 15:34:49 Contents of /app/letsencrypt_service_data did not change. Skipping notification '/app/signal_le_service',
2020/11/17 15:34:50 Error: nginx-proxy container nginx-proxy isn't running.,
Sleep for 3600s,

If I restart nginx-proxy at this point, it runs OK, together with the companion:

2020/11/17 15:35:15 Received event start for container 6d1c2fc43c97,
2020/11/17 15:35:30 Debounce minTimer fired,
2020/11/17 15:35:30 Contents of /app/letsencrypt_service_data did not change. Skipping notification '/app/signal_le_service',

Next, I'll start the nextcloud container:

sudo docker run -d -p 8001:80 --name nextcloud -v nextcloud_data:/var/www/html --env "VIRTUAL_HOST=xxx.xxx" --env "VIRTUAL_PORT=8001" --env "LETSENCRYPT_HOST=xxx.xxx" --env "[email protected]" nextcloud

The companion picks up the container startup correctly:

2020/11/17 15:35:43 Received event start for container 07cd31e97f33,
2020/11/17 15:35:58 Debounce minTimer fired,
2020/11/17 15:35:58 Generated '/app/letsencrypt_service_data' from 4 containers,
2020/11/17 15:35:58 Running '/app/signal_le_service',
Debug: checking /etc/nginx/certs/xxx.xxx ownership and permissions.,
Debug: numeric ID of user root is 0.,
Debug: numeric ID of group root is 0.,
/etc/nginx/certs/xxx.xxx /app,
Reloading nginx docker-gen (using separate container nginx-proxy)...,
Reloading nginx (using separate container nginx-proxy)...,
Creating/renewal xxx.xxx certificates... (xxx.xxx),

But the nginx-proxy dies again:

2020/11/17 15:36:00 Received event die for container 6d1c2fc43c97,

This is then followed by a correct Let's Encrypt exchange (keys generation OK, DNS resolved OK, Let's Encrypt responses OK, nonce generated OK) that all looks as it should, until it fails on the ACME Challenge Validation, because the nginx-proxy is not running:

2020-11-17 15:36:07,767:DEBUG:acme.client:1177: Storing nonce: xxxxxxxxxxx,
2020-11-17 15:36:07,769:ERROR:simp_le:1417: CA marked some of the authorizations as invalid, which likely means it could not access http://example.com/.well-known/acme-challenge/X. Did you set correct path in -d example.com:path or --default_root? Are all your domains accessible from the internet? Please check your domains' DNS entries, your host's network/firewall setup and your webserver config. If a domain's DNS entry has both A and AAAA fields set up, some CAs such as Let's Encrypt will perform the challenge validation over IPv6. If your DNS provider does not answer correctly to CAA records request, Let's Encrypt won't issue a certificate for your domain (see https://letsencrypt.org/docs/caa/). Failing authorizations: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8670690504,
2020-11-17 15:36:07,779:INFO:simp_le:401: Saving account_key.json,
2020-11-17 15:36:07,781:INFO:simp_le:401: Saving account_reg.json,
2020-11-17 15:36:07,782:DEBUG:simp_le:1134: Removing validation file at /usr/share/nginx/html/.well-known/acme-challenge/xxxxxxxxxxxxxxxxx,
Challenge validation has failed, see error log.,

Any ideas why this happens? No AAAA fields, DNS resolved OK,

[buchdag]

At this point, the companion kills the nginx-proxy:

The companion does not kill the nginx-proxy container, it uses the Docker API call "kill" to send a custom signal (in this case SIGTERM) to the nginx-proxy container.

Your nginx-proxy container should not exit when receiving SIGTERM. It really sounds like an issue with this container to me, not with the companion container.

Also, please not that the arm versions of this container while available from DockerHub are not yet officially tested, and that I can't provide support for setup that don't use the upstream nginx-proxy/nginx-proxy (meaning no support for traskit/nginx-proxy unfortunately).

[phantomski77]

Thanks @buchdag for your response and of course your great work putting all this together. I appreciate I'm a bit early with the request for arm64 support, Raspberry Pi 4 is only a recent addition (but works great as a server!).

In the meantime, I've found this PR:
https://github.com/nginx-proxy/nginx-proxy/pull/1470#issue-448200510
and I've tried to build nginx-proxy image from the provided Dockerfile, that's using sourcecode instead of binaries for jwilder/docker-gen which I think is probably the main problem. Unfortunately it still displays the same behaviour.

I suppose until @jwilder decides to build his great stuff for arm64, this will stay an issue which is a bit unfortunate, as your companion and his proxy are so far the only solution for Let's Encrypt SSL certificates for Docker run subdomains.

Unless you know of a different solution of course.

[buchdag]

I haven't updated this for a while but take a look at https://github.com/buchdag/multiarch-letsencrypt-nginx-proxy

[unicod3]

Hello @phantomski77 ,
I was able to get it working with arm64 on the repo https://github.com/buchdag/multiarch-letsencrypt-nginx-proxy
see PR

[phantomski77]

@buchdag you're a star!

Thank you very much for your kind help (and thanks @unicod3 for your reply as well), that worked like a charm, first try, subdomain certificates issued, validated, challenge responded no probs this time. Signalling between containers works, nothing dies unexpectedly.

Build time on rPi4 8GB with USB SSD drive was 10 minutes including downloads, DH took 30 secs.

A question from a Docker noob (I've only started using it last week, so apologies for asking about obvious). If I correctly tag the images, can I then upload them into my Docker Hub repository, so I (and others) can re-use them elsewhere without another build? Do they need to run in a stack as now, or if I setup the container with same ENVs as the "normal" containers, they would work and talk to each other?

Thanks again for your great work. I'm hoping arm64 would become an official build at some time ;) It's a great architecture with a lot of future potential (even with nVidia in the mix).

[buchdag]

If I correctly tag the images, can I then upload them into my Docker Hub repository, so I (and others) can re-use them elsewhere without another build?

Provided you already created your image's repository on DockerHub and logged into DockerHub using docker login : yes (with docker push repo/image:tag)

Do they need to run in a stack as now, or if I setup the container with same ENVs as the "normal" containers, they would work and talk to each other?

I'm not sure I understood the question correctly, but if it was about running everything with Docker compose: it's not mandatory and was just done this way for convenience and ease of use. If you correctly built your nginx-proxy / docker-gen and letsencrypt-nginx-proxy-companion images on an arm or arm64 host, you can then run them however you want.

All the doc in this repo and in nginx-proxy/nginx-proxy repo will apply the same to the images you built locally.

[phantomski77]

Perfect, thanks @buchdag. All worked well and pushed into repos phantomski/nginx-proxy and phantomski/letsencrypt-nginx-proxy-companion with new arm64 tags and bit of a description.

The question was purely about how the compose actually created a Stack, but now I understand it's just for convenience as one Docker compose with all the presets in one but both images are still built separately and all the original guidelines from your jrcs/letsencrypt-nginx-proxy-companion still work.

Thanks for your help!