Docker-letsencrypt-nginx-proxy-companion: Does not restore original vhosts configuration

[Redsandro]

When using Per-VIRTUAL_HOST configuration, for example to set rewrite or redirect rules, the docker-letsencrypt-nginx-proxy-companion prepends some custom code, and doesn't clean up nor restore the original file afterwards.

The files are left in /etc/nginx/vhost.d with the temporary code prepended, and the entire nginx server won't work.

For example, /etc/nginx/vhost.d/windowsremix.com before starting this companion:

return 301 https://www.windowsremix.com$request_uri;

/etc/nginx/vhost.d/windowsremix.com after this companion:

## Start of configuration add by letsencrypt container
location ^~ /.well-known/acme-challenge/ {
    auth_basic off;
    allow all;
    root /usr/share/nginx/html;
    try_files $uri =404;
    break;
}
## End of configuration add by letsencrypt container
return 301 https://www.windowsremix.com$request_uri;

Removing all custom vhost configuration and restarting the stack brings up the server again. Although of course without custom rewrite or redirect rules.

[buchdag]

Hi. I'll try to take a look at this soon.

[Faerbit]

I had the same problem and fiddled a bit with it.
My setup is something like this

# docker-compose.yml
volumes:
    vhostd:
services:
    nginx-proxy:
        image: jwilder/nginx-proxy
        volumes:
            - vhostd:/etc/nginx/vhost.d
            - /path/to/config/nginx.conf:/etc/nginx/example.com
    letsencrypt:
        image: JrCs/docker-letsencrypt-nginx-proxy-companion
        volumes:
            - vhostd:/etc/nginx/vhost.d

(Shortended to the relevant bits)
I modified /app/functions.sh a bit to be able to look at the original nginx.conf mounted as example.com, before it is modified by the letsencrypt_service (via functions.sh:add_location_configuration) script.
In this configuration the file example.com exists but is empty, which leads to the behaviour described in the OP.
I am not sure if this intendended behaviour or a bug, and if the bug is within docker or docker-compose, so I am unsure how one would proceed with this problem.
Simply moving the volume declaration like this:

# docker-compose.yml
volumes:
    vhostd:
services:
    nginx-proxy:
        image: jwilder/nginx-proxy
        volumes:
            - vhostd:/etc/nginx/vhost.d

    letsencrypt:
        image: JrCs/docker-letsencrypt-nginx-proxy-companion
        volumes:
            - vhostd:/etc/nginx/vhost.d
            - /path/to/config/nginx.conf:/etc/nginx/example.com

Produces /etc/nginx/vhost.d/example.com.new which looks like it should, but the mv, which copies the example.com.new to example.com fails.
For the moment I settled to a manually created file which contains my configuration directive and the ones from this container, and mounted this file to both containers.

[Redsandro]

Thank you @Faerbit for your insights!

For the time being as long as I can get away with it, I'm using https://hub.docker.com/r/morbz/docker-web-redirect/ to redirect things. This is only a workaround for a specific use case. I think something needs to be rewritten (or documented), but I'm not sure where. I'm hoping someone more knowledgeable than me will remedy this before it's time for me to require more complex configurations.

[Redsandro]

[Reverting accidental close]

[buchdag]

@Faerbit when using Docker bind mount (ie mounting a single file or directory inside a container), you have to configure the bind mount on every container that will need access to this file, either by using the volumes-from (CLI) / volumes_from (compose) option or by mounting it explicitly inside each container.

So the correct configuration snippets would either be :

# docker-compose.yml
volumes:
    vhostd:
services:
    nginx-proxy:
        image: jwilder/nginx-proxy
        container_name: nginx-proxy
        volumes:
            - vhostd:/etc/nginx/vhost.d
            - /path/to/config/nginx.conf:/etc/nginx/example.com

    letsencrypt:
        image: jrcs/letsencrypt-nginx-proxy-companion
        volumes_from:
            - nginx-proxy

or :

# docker-compose.yml
volumes:
    vhostd:
services:
    nginx-proxy:
        image: jwilder/nginx-proxy
        volumes:
            - vhostd:/etc/nginx/vhost.d
            - /path/to/config/nginx.conf:/etc/nginx/example.com

    letsencrypt:
        image: jrcs/letsencrypt-nginx-proxy-companion
         volumes:
            - vhostd:/etc/nginx/vhost.d
            - /path/to/config/nginx.conf:/etc/nginx/example.com

note that volumes_from is only available in compose file version 2.

[Redsandro]

@buchdag thank you for the update. I think to recall I did this already. I should try this out, but in the meantime I have moved to a different solution with a redirect container.

[buchdag]

I just did a test with the default vhost file and a per-VIRTUAL_HOST vhost file, both worked as intended by the code right now (which might or might not be the right way).

I edited both file to add ## This is a test comment from within the container, then created a container that triggered a certificate creation.

user@vps:~$ docker exec nginx-proxy echo '## This is a test comment' > /etc/nginx/vhost.d/default
user@vps:~$ docker exec nginx-proxy cat /etc/nginx/vhost.d/default
## This is a test comment
user@vps:~$ docker run --name nginx-test --detach --env VIRTUAL_HOST=my.domain.tld --env LETSENCRYPT_HOST=my.domain.tld --env LETSENCRYPT_TEST=true nginx:alpine
f43622abcc4b610437cc1936231e9e4fd41e0ac1ee644ba1304e73ca65e34da5

[waiting a bit ...]

user@vps:~$ docker exec nginx-proxy cat /etc/nginx/vhost.d/default
## Start of configuration add by letsencrypt container
location ^~ /.well-known/acme-challenge/ {
    auth_basic off;
    allow all;
    root /usr/share/nginx/html;
    try_files $uri =404;
    break;
}
## End of configuration add by letsencrypt container
## This is a test comment
user@vps:~$ docker stop nginx-proxy-letsencrypt
nginx-proxy-letsencrypt
user@vps:~$ docker exec nginx-proxy cat /etc/nginx/vhost.d/default
## This is a test comment

[second test]

user@vps:~$ docker stop nginx-test
nginx-test
user@vps:~$ docker exec nginx-proxy echo '## This is a test comment' > /etc/nginx/vhost.d/my.domain.tld
user@vps:~$ docker exec nginx-proxy cat /etc/nginx/vhost.d/my.domain.tld
## This is a test comment
user@vps:~$ docker exec nginx-proxy rm -rf /etc/nginx/certs/_test_my.domain.tld
user@vps:~$ docker start nginx-proxy-letsencrypt
nginx-proxy-letsencrypt
user@vps:~$ docker start nginx-test
nginx-test

[waiting a bit ...]

user@vps:~$ docker exec nginx-proxy cat /etc/nginx/vhost.d/default
## This is a test comment
user@vps:~$ docker exec nginx-proxy cat /etc/nginx/vhost.d/my.domain.tld
## Start of configuration add by letsencrypt container
location ^~ /.well-known/acme-challenge/ {
    auth_basic off;
    allow all;
    root /usr/share/nginx/html;
    try_files $uri =404;
    break;
}
## End of configuration add by letsencrypt container
## This is a test comment
user@vps:~$ docker stop nginx-proxy-letsencrypt
nginx-proxy-letsencrypt
user@vps:~$ docker exec nginx-proxy cat /etc/nginx/vhost.d/my.domain.tld
## This is a test comment

If I get it right, with some specific configuration leaving the /.well-known/acme-challenge/ location configuration prepended to existing configuration causes a configuration issue.

Sorry in advance if this is a stupid question, but does your custom vhosts configuration work before the companion prepends its code ?

Does the nginx process running in nginx-proxy complains about a bad configuration ?

Does the certificate creation works anyway ?

If yes I guess the container should clean the /.well-known/acme-challenge/ location configuration and reload nginx right after it obtained the certificate, right ?

[Redsandro]

@buchdag thank you for looking into this. Yes, everything worked except the custom configuration part. There were no complaints about bad configuration prior. I removed per-VHOST configuration (which I attempted to use for a rewrite configuration) and used the redirect container in stead. I cannot test exactly the config I used, since I don't have it anymore (it's been a while). But I will keep this in mind for the next time I want to use a custom configuration. I remember the need arose now and then, but could not work it out. I can update my experience after next time I want to set something like this up.

Not sure I should close this issue in the meanwhile. Since I cannot currently verify this, I think it's sane to wait for at least one other person to verify that this is no longer an issue. Keep in mind that both @Faerbit's experience and my own might indicate that this is a file permission issue. Not sure how, but a file being created but not being (re)moved tastes like a permission thingie.

[buchdag]

I tried to mimic your setup with one container redirecting to another using a custom per-vhost config with return 301 https://my.domaint.tld$request_uri; bind mounted to the nginx-proxy container and ran into an issue : because of the bind mount, the file inode can't be changed from within the container, so neither mv nor sed -i will work. This result in the following error:

mv: cannot move '/etc/nginx/vhost.d/my.domaint.tld.new' to '/etc/nginx/vhost.d/my.domaint.tld': Resource busy

Using cp then rm instead of mv fix the issue.

I'll have to come up with a test unit for this before merging the fix.

Oddly enough, even if the container couldn't overwrite the per-vhost config with the one that included the location configuration, the certificate was still issued and everything worked as intended, redirection included.

[Redsandro]

That's an interesting find @buchdag. I observed the same with the exception of redirection not working (which alerted me to this issue).

It's plausible small differences in outcome have the same root cause.

[buchdag]

@Redsandro according to my tests this should now be fixed on latest.

Let me know if it does indeed solves your configuration issue and if does not, do not hesitate to reopen the issue.

The original behaviour wasn't changed : the additional location code is only cleaned once, when the letsencrypt-nginx-proxy-companion container exits. It should not conflict with any other existing Per-VIRTUAL_HOST configuration it's prepended to, but I'm not expert enough in nginx configuration to guarantee than no conflict can happen ever. If anyone finds one, please report it as separate issue.

[Redsandro]

@buchdag thank you for looking into this. :+1: