Docker-letsencrypt-nginx-proxy-companion: Don't always create a new certificate on startup

[jsaalfeld]

Hi there,

I got the Issue in a project of mine, that the companion creates a new certificate on every docker-compose up even though valid certificates are already in place from a startup before. Is there a flag or a way to tell the companion, to first look if in the certs folder are already valid certificates for the VIRTUAL_HOST?

[buchdag]

Hi.

What version of the container are you using ?

Do you have any log output from the container ?

[jsaalfeld]

For development purposes i regulary deleted my images, so it is the latest version available thorugh the docker repository.

What kind of logs do you need? Do you look for something special? From what containers?

[buchdag]

Okay, before we start digging into logs, how did you notice / reach the conclusion that the container creates new certificates on every docker-compose up ?

[jsaalfeld]

In 2 ways:

1. The logs running through always stated something about certificate creation and i saw in the nginx logs, that the .well-known was reached out to
2. After a couple of docker-compose down and docker-compose up on a system, the companion stated, that we reached the maximum of certificate requests for that domain in that week

I should add: If I start everything, except the lets-encrypt service in the setup, I have the valid certificate from the startup before still in place and the SSL-encryption works like a charm; without a new certificate.

[buchdag]

Sounds like there is something fishy with your volumes. Is it possible that your cert volume is deleted or recreated on docker-compose down / docker-compose up ?

[jsaalfeld]

As you can see from the linked repo in the start of the thread, there are no additional scripts to prune volumes etc.

The interesting thing is: When i let the companion create the certificates in the first start and then scale the companion down to 0 in the next start, nginx will successfully recognize the certificates. So no: I don't think there is something fishy with the volumes.

[buchdag]

I'll give a shot at your repo as soon as possible but I didn't see anything out of the ordinary either.

[jwulf]

For development purposes i regulary deleted my images, so it is the latest version available thorugh the docker repository.

If you also delete your volumes, there will be no cached certificate, so it will request a new one.

[jwulf]

I actually ran into this issue, without deleting volumes.

To get around it, I created two separate docker-compose files: one for the nginx/lets-encrypt combo, and another one for my webapp. I set my CD system to bring the webapp up and down, and not touch the proxy docker instance.

That got around it.

[alextousss]

Hello,
I'm having the same problem, which is kind of problematic as certificate creations create a few minutes of downtime.
Is there any fix ?

[buchdag]

Is there any fix ?

No clear and reproducible issue has been identified, so there is unfortunately nothing to fix at this point.

Under normal operation the container does not recreate existing certificate on restart and I haven't observed this behaviour on any of my own servers.

If your compose file don't use external volumes, the volume creation / deletion is handled by docker-compose itself and you might be deleting or recreating your volumes without realising it. Please check docker-compose doc, watch your Docker volumes and the docker-compose command output.

Is docker-compose down deleting your volumes ?
Is docker-compose up creating new volumes ?

[alextousss]

Is there any fix ?

No clear and reproducible issue has been identified, so there is unfortunately nothing to fix at this point.

Under normal operation the container does not recreate existing certificate on restart and I haven't observed this behaviour on any of my own servers.

If your compose file don't use external volumes, the volume creation / deletion is handled by docker-compose itself and you might be deleting or recreating your volumes without realising it. Please check docker-compose doc, watch your Docker volumes and the docker-compose command output.

Is docker-compose down deleting your volumes ?
Is docker-compose up creating new volumes ?

I guess docker-compose isn't deleting / creating volumes. Is there any way to be sure ?

version: 2.1
services:
    nginx-proxy:
        restart: always
        image: jwilder/nginx-proxy
        ports:
            - "80:80"
            - "443:443"
        volumes:
            - /srv/docker/nginx/certs:/etc/nginx/certs:ro
            - /etc/nginx/vhost.d
            - /usr/share/nginx/html
            - /var/run/docker.sock:/tmp/docker.sock:ro
            - ./http_proxy.conf:/etc/nginx/conf.d/my_proxy.conf:ro

    nginx-proxy-companion:
        image: jrcs/letsencrypt-nginx-proxy-companion
        volumes:
            - /srv/docker/nginx/certs:/etc/nginx/certs:rw
            - /var/run/docker.sock:/var/run/docker.sock
        volumes_from:
            - nginx-proxy

Is there anything wrong with my docker-compose configuration ?

Nevertheless, each time I'm restarting my server (docker-compose down, then docker-compose up), openssl uses 100% of the CPU during ~2 minutes, locking my whole server.

Thanks in advance for any help.

Alex

[buchdag]

The heavy CPU consuming openssl process at start is probably the 2048 bits DH parameters generation, but like the rest it should only happen once. It's also heavily nice'd and should not lock anything beside extremely small single core servers (the only thing I run that have issue with it is a Google Cloud f1-micro instance, and even that become slow but not locked), what hardware or VM are you running this on ?

Your container appears to be acting exactly as if its /etc/nginx/certs folder was wipped on each restart.

What happen to the files inside your /srv/docker/nginx/certs folder after a docker-compose down ?

Could you try using a Docker named volume for /etc/nginx/certs instead of a bind mount ?

version: 2.1
services:
    nginx-proxy:
        restart: always
        image: jwilder/nginx-proxy
        ports:
            - "80:80"
            - "443:443"
        volumes:
            - certs:/etc/nginx/certs:ro
            - /etc/nginx/vhost.d
            - /usr/share/nginx/html
            - /var/run/docker.sock:/tmp/docker.sock:ro
            - ./http_proxy.conf:/etc/nginx/conf.d/my_proxy.conf:ro

    nginx-proxy-companion:
        image: jrcs/letsencrypt-nginx-proxy-companion
        volumes:
            - certs:/etc/nginx/certs:rw
            - /var/run/docker.sock:/var/run/docker.sock:ro
        volumes_from:
            - nginx-proxy

volumes:
  certs:

[alextousss]

Hello,

Sorry if I was misunderstood. It doesn't lock the system completely, but the website is inaccessible (which is normal).

After a docker-compose down, the files stay the same.

I'll try with the bind mount, I'll keep you informed.

[dm17]

@jwulf Can I see an example of how you did that? Would be nice to leave the existing docker-compose.yaml alone and have a separate one that acts on it for the nginx/letsencrypt functionality.

[jwulf]

Here you go: https://github.com/jwulf/letsencrypt-nginx-sidecar

[buchdag]

This issue has been inactive for a while, closing.