Docker-letsencrypt-nginx-proxy-companion: 503 + upstream missing ip

[xar]

Hello. I have followed installation instructions and successfully started containers. But when i try to open url i get 503 unaval and on https i didn't get any response.

Here is conf.d/defautl.conf

# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
  default $http_x_forwarded_proto;
  ''      $scheme;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
  default upgrade;
  '' close;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent"';
access_log off;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
    server_name _; # This is just an invalid value which will never trigger on a real hostname.
    listen 80;
    access_log /var/log/nginx/access.log vhost;
    return 503;
}
upstream autobazaris.sk {
}
server {
    server_name autobazaris.sk;
    listen 80 ;
    access_log /var/log/nginx/access.log vhost;
    return 301 https://$host$request_uri;
}
server {
    server_name autobazaris.sk;
    listen 443 ssl http2 ;
    access_log /var/log/nginx/access.log vhost;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers xxx;
    ssl_prefer_server_ciphers on;
    ssl_session_timeout 5m;
    ssl_session_cache shared:SSL:50m;
    ssl_certificate /etc/nginx/certs/autobazaris.sk.crt;
    ssl_certificate_key /etc/nginx/certs/autobazaris.sk.key;
    ssl_dhparam /etc/nginx/certs/autobazaris.sk.dhparam.pem;
    add_header Strict-Transport-Security "max-age=31536000";
    include /etc/nginx/vhost.d/default;
    location / {
        proxy_pass http://autobazaris.sk;
    }
}

[JrCs]

Are the certs for autobazaris.sk generated correctly by letsencrypt ?
If yes your problem is from ngnix-proxy (or docker-gen container) and the nginx.tmpl file.
Tips: is ssl_ciphers xxx; in your configuration is correct ?

[pdemro]

I am seeing a similar issue. I believe I ran through all the steps correctly. At the end my site resolves with a 503. A few things probably worth mentioning

1. If stopped, the nginx container will refuse to restart. I have to start the process over again
2. /etc/nginx/certs directory has two items, dhparam.pem and a directory for my URL. The directory is empty, I am assuming this means that it failed to generate certs?

[JrCs]

Please give me the logs of your containers

[xar]

i'm sorry for later answer. Here are my logs:
nginx:

2016/08/16 19:27:52 [emerg] 1#1: no servers are inside upstream in /etc/nginx/conf.d/default.conf:36
nginx: [emerg] no servers are inside upstream in /etc/nginx/conf.d/default.conf:36
2016/08/16 19:32:06 [emerg] 1#1: no servers are inside upstream in /etc/nginx/conf.d/default.conf:36
nginx: [emerg] no servers are inside upstream in /etc/nginx/conf.d/default.conf:36

docker-gen

2016/08/16 19:19:03 Generated '/etc/nginx/conf.d/default.conf' from 17 containers
2016/08/16 19:19:03 Sending container 'nginx' signal '1'
2016/08/16 19:19:03 Error sending signal to container: API error (500): Cannot kill container nginx: Container 841adc3170366128438386fdd6793b4cf94e487a88966f2a1684a61e55f1e10f is not running
2016/08/16 19:19:03 Watching docker events
2016/08/16 19:19:03 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
2016/08/16 19:20:33 Received event start for container 5e96b30035d7
2016/08/16 19:20:38 Debounce minTimer fired
2016/08/16 19:20:38 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
2016/08/16 19:27:49 Received event die for container 24d7b6c5ca1b
2016/08/16 19:27:49 Received event stop for container 24d7b6c5ca1b
2016/08/16 19:27:52 Received event start for container 24d7b6c5ca1b
2016/08/16 19:27:52 Received event die for container 24d7b6c5ca1b
2016/08/16 19:27:57 Debounce minTimer fired
2016/08/16 19:27:58 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
2016/08/16 19:32:06 Received event start for container 24d7b6c5ca1b
2016/08/16 19:32:06 Received event die for container 24d7b6c5ca1b
2016/08/16 19:32:11 Debounce minTimer fired
2016/08/16 19:32:11 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''

letsencrypt:

Creating Diffie-Hellman group (can take several minutes...)
Generating DH parameters, 2048 bit long safe prime, generator 2
This is going to take a long time
....
.....
.....
2016/08/16 19:24:51 Generated '/app/letsencrypt_service_data' from 17 containers
2016/08/16 19:24:51 Running '/app/update_certs'
2016/08/16 19:24:51 Watching docker events
Sleep for 3600s
2016/08/16 19:24:51 Contents of /app/letsencrypt_service_data did not change. Skipping notification '/app/update_certs'
2016/08/16 19:27:49 Received event die for container 24d7b6c5ca1b
2016/08/16 19:27:49 Received event stop for container 24d7b6c5ca1b
2016/08/16 19:27:52 Received event start for container 24d7b6c5ca1b
2016/08/16 19:27:52 Received event die for container 24d7b6c5ca1b
2016/08/16 19:28:07 Debounce minTimer fired
2016/08/16 19:28:08 Contents of /app/letsencrypt_service_data did not change. Skipping notification '/app/update_certs'
2016/08/16 19:32:06 Received event start for container 24d7b6c5ca1b
2016/08/16 19:32:06 Received event die for container 24d7b6c5ca1b
2016/08/16 19:32:21 Debounce minTimer fired
2016/08/16 19:32:21 Contents of /app/letsencrypt_service_data did not change. Skipping notification '/app/update_certs'

[ryneeverett]

It looks like you didn't start the nginx container first and therefore docker-gen and letsencrypt are mounting volumes from an old (stopped) nginx container.

[brycejacobs]

This same problem is happening to me, after following your example off the README.

Might be worth mentioning somewhere.

[SnowMB]

Hey,

are you using the seperated container method? The link of the nginx.tmpl file belongs to the jwilder/nginx-proxy container and wont work with seperate containers (at least it didn't for me).

Since the readme was written there where several changes made to the template file. You could use this older version which will work.

[brycejacobs]

@SnowMB You're right, it is stemming from the changes in the nginx.tmpl. With the old files, the upstream ip addresses are resolved.

Did you open an issue on jwilder/nginx-proxy?

[ryneeverett]

I've never had any trouble with the separate container method and the
latest nginx-proxy release (0.4.0). Are yall using the template on master?

On Tue, Oct 11, 2016 at 11:45 AM, Bryce Jacobs [email protected]
wrote:

@SnowMB https://github.com/SnowMB You're right, it is stemming from the
changes in the nginx.tmpl. With the old files, the upstream ip addresses
are resolved.

Did you open an issue on jwilder/nginx-proxy?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion/issues/88#issuecomment-252957005,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADINmJAI5OXQU0JCfyBjWg5oxHkugec6ks5qy67-gaJpZM4JWczy
.

[SnowMB]

@ryneeverett yeah i did.

I played around with the template file and found the lines, that give the problems. (see #107)
But indeed it seems to be a problem of the docker-gen container and has nothing to do with the letsencrypt companion.

There is already an open issue on the docker-gen container https://github.com/jwilder/docker-gen/issues/196.

It links to https://github.com/jwilder/nginx-proxy/issues/479 and there is suggsted fix I can't try out right now.

@NoahO and remove -only-exposed for nginx-gen

Can someone try this?

Edit: I just double checked the readme for seperated containers in https://github.com/jwilder/nginx-proxy:

$ docker run --volumes-from nginx \
    -v /var/run/docker.sock:/tmp/docker.sock:ro \
    -v $(pwd):/etc/docker-gen/templates \
    -t jwilder/docker-gen -notify-sighup nginx -watch /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf

The code in the readme here:

docker run -d \
    --name nginx-gen \
    --volumes-from nginx \
    -v /path/to/nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro \
    -v /var/run/docker.sock:/tmp/docker.sock:ro \
    jwilder/docker-gen \
    -notify-sighup nginx -watch -only-exposed -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf

[JrCs]

@SnowMB can you try with -only-exposed option removed ?

[SnowMB]

Just tried it out, doesn't work for me either :disappointed:

[SnowMB]

Ok, I take that back.

It actually helps! I've just made another mistake that did result in the same error. :smile:.

So what I did is the following. I used docker-compose v2 and the example you linked in your readme. https://github.com/fatk/docker-letsencrypt-nginx-proxy-companion-examples

There is one mistake that shows the same empty upstream block. The docker-gen container MUST be added to the proxy-tier network.
But if I just add the container to the network, it still shows an empty upstream. But removing -only-exposed now makes the difference! I don't really know whats going on behind the scenes but it works for me.

Correct docker-gen block:

version: '2'

services:
  nginx:
    image: nginx
    container_name: nginx
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./proxy/conf.d:/etc/nginx/conf.d
      - ./proxy/vhost.d:/etc/nginx/vhost.d
      - ./proxy/html:/usr/share/nginx/html
      - ./proxy/certs:/etc/nginx/certs:ro
    networks:
      - proxy-tier

  nginx-gen:
    image: jwilder/docker-gen
    container_name: nginx-gen
    volumes_from:
      - nginx
    volumes:
      - ./nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro
      - /var/run/docker.sock:/tmp/docker.sock:ro
#    entrypoint: /usr/local/bin/docker-gen -notify-sighup nginx -watch -only-exposed -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/def$
    entrypoint: /usr/local/bin/docker-gen -notify-sighup nginx -watch /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
    networks:
      - proxy-tier

  letsencrypt-companion:
    image: jrcs/letsencrypt-nginx-proxy-companion
    container_name: letsencrypt-companion
    volumes_from:
      - nginx
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./proxy/certs:/etc/nginx/certs:rw
    environment:
      - LETSENCRYPT_TEST=TRUE
      - NGINX_DOCKER_GEN_CONTAINER=docker-gen


...
...
...


networks:
  proxy-tier:
    external:
      name: nginx-proxy

I also don't know what -wait 5s:30s is for. It works with and without.

Hope it helps anyone

[1beb]

I was also able to bypass this issue using @SnowMB solution. Examples here:

https://gist.githubusercontent.com/SnowMB/758bb6c993372111629f4e86c0e1de1a/raw/b1e6d4454ef4af302eca19061bebbd573725d96f/docker-compose.yml
https://gist.githubusercontent.com/SnowMB/87c5360f9bf81925af26c31f6d71410e/raw/fc8784d1cbc8ad56047b10630b68c1830859bf63/nginx.tmpl

[krivochenko]

Hi! I'm not sure that problem in docker-letsencrypt-nginx-proxy-companion, but I googled this issue so decided to write here.

I have a docker-compose.yml:

version: '2'

services:
    nginx:
        image: nginx
        container_name: nginx
        ports:
            - "80:80"
            - "443:443"
        volumes:
            - /var/www/proxy/conf.d:/etc/nginx/conf.d
            - /var/www/proxy/vhost.d:/etc/nginx/vhost.d
            - /var/www/proxy/html:/usr/share/nginx/html
            - /var/www/proxy/certs:/etc/nginx/certs:ro
        networks:
            - portainer_default

    nginx-gen:
        image: jwilder/docker-gen
        container_name: nginx-gen
        volumes:
            - /var/www/proxy/nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro
            - /var/run/docker.sock:/tmp/docker.sock:ro
        volumes_from:
          - nginx            
        entrypoint: /usr/local/bin/docker-gen -notify-sighup nginx -watch -only-exposed -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
        networks:
            - portainer_default

    letsencrypt:
        image: jrcs/letsencrypt-nginx-proxy-companion
        container_name: nginx-letsencrypt
        environment:
            - NGINX_DOCKER_GEN_CONTAINER=nginx-gen
        volumes:  
            - /var/www/proxy/certs:/etc/nginx/certs:rw
            - /var/run/docker.sock:/var/run/docker.sock:ro
        volumes_from:
          - nginx
        networks:
            - portainer_default  

networks:
    portainer_default:
        external: true 

I use https://github.com/jwilder/nginx-proxy/blob/master/nginx.tmpl as template. Also I have another one docker-compose.yml:

version: '2'

services:
    portainer:
        image: portainer/portainer
        container_name: portainer
        volumes: 
            - /var/run/docker.sock:/var/run/docker.sock
        environment:
            - VIRTUAL_HOST=portainer.my-domain.com
            - VIRTUAL_PORT=9000
            - LETSENCRYPT_HOST=portainer.my-domain.com
            - [email protected]

As a result I have next generated config:

# portainer.my-domain.com
upstream portainer.my-domain.com {
}
server {
    server_name portainer.my-domain.com;
    listen 80 ;
    access_log /var/log/nginx/access.log vhost;
    include /etc/nginx/vhost.d/portainer.my-domain.com;
    location / {
        proxy_pass http://portainer.my-domain.com;
                include /etc/nginx/vhost.d/portainer.my-domain.com_location;
    }
}

Of cource it calls error: "no servers are inside upstream".

Which additional information I can give you?

Thanks for help!

[thomasleveil]

don't use -only-exposed

[krivochenko]

Thaknks!

[jrbenito]

Hi,
I am facing the same issue while running on AWS ECS cluster with ecs-cli compose. I tried all network modes available (bridge, host and awsvpc) and problem remains the same, no server in upstream.

I am running the companion image with jwilder/nginx-proxy because limitations on AWS container names but tried to run docker-gen separated but config file ended up without upstream server too.

Any idea?

Thanks