Docker-letsencrypt-nginx-proxy-companion: Connection refused on 443

[martinhpunkt]

Hi all,

tried my best for the last two days and do not get the clue. Hopefully I haven't made any basic mistakes which I should have identified myself, however, if you could provide me assistance I'd be thankful.

My desired setup is as follows:
1) one nginx proxy serving for all services as reverse proxy
2) letsencrypt support with this project
3) one separate container for each service

My current design approach is to have separated docker compose files. The first defines the first two requirements and for each container from 3) there will be a separate compose file.

My compose file for nginx and nginx-companion looks like this:

version: '2'
services:
    proxy:
        image: jwilder/nginx-proxy:alpine
        container_name: proxy
        ports:
            - "80:80"
            - "443:443"
        environment:
            DEFAULT_HOST: domain.com
            DOCKER_HOST: unix:///tmp/docker.sock
        volumes:
            - certs:/etc/nginx/certs:ro
            - vhost.d:/etc/nginx/vhost.d
            - html:/usr/share/nginx/html
            - /var/run/docker.sock:/tmp/docker.sock:ro
            - /root/docker-setup/nginx-letsencrypt/sync.domain.com.conf:/etc/nginx/vhost.d/sync.domain.com
        networks:
            - nw
    proxy-companion:
        image: jrcs/letsencrypt-nginx-proxy-companion
        container_name: proxy-companion
        environment:
            - DEBUG=true
        volumes:
            - certs:/etc/nginx/certs
            - /var/run/docker.sock:/var/run/docker.sock:ro
        volumes_from:
            - proxy
        networks:
            - nw
volumes:
    certs:
    vhost.d:
    html:
networks:
    nw:

I added an additional configuration file to overcome Issue #254 from jwilder's project, i.e. I want to have multiple containers running on one subdomain.

The content of the additional file is very simple:

location /owncloud/ {
        proxy_pass http://sync.domain.com/owncloud/;
}

Which finally leads to the compose file of 3) which is owncloud:

version: '2'
services:
    mariadb:
        image: mariadb:latest
        container_name: owncloud-mariadb
        environment:
            - MYSQL_ROOT_PASSWORD_FILE=/tmp/initpw/mariadbpw-root
            - MYSQL_DATABASE=ocdb
            - MYSQL_USER=ocuser
            - MYSQL_PASSWORD_FILE=/tmp/initpw/mariadbpw-ocuser
        volumes:
            - /root/docker-setup/owncloud/:/tmp/initpw/
            - volume-owncloud-mariadb:/var/lib/mysql
            - /root/docker-setup/owncloud/init-data:/docker-entrypoint-initdb.d/
        networks:
            nw_owncloud:
    owncloud:
        image: owncloud:9.1
        container_name: owncloud
        environment:
            - VIRTUAL_HOST=sync.domain.com
#            - VIRTUAL_PATH="/owncloud/"
            - VIRTUAL_PORT=80
            - LETSENCRYPT_HOST=sync.domain.com
            - [email protected]
            - LETSENCRYPT_TEST=true
#        ports:
#            - 8180:80
        expose:
            - 80
        volumes:
            - /ocdata:/var/www/html
        networks:
            nw_owncloud:
volumes:
    volume-owncloud-mariadb:
#        external: true
networks:
    nw_owncloud:
#         external: true

Here comes the issue. The owncloud container with http and the ip works well so I can assume that this setup is working. When it comes to nginx and nginx-companion I don't even get log messages from nginx-proxy.

Example:

curl -v --ipv4 -I https://sync.domain.com/owncloud
*   Trying XXX.XXX.141.136...
* connect to XXX.XXX.141.136 port 443 failed: Verbindungsaufbau abgelehnt
* Failed to connect to sync.domain.com port 443: Verbindungsaufbau abgelehnt
* Closing connection 0
curl: (7) Failed to connect to sync.domain.com port 443: Verbindungsaufbau abgelehnt`

(translated: connection refused).

What I tried/verified

• the certificates are there
• iptables do not seem to me as if there was something wrong
• curl with normal http works and I'm getting logs
• issue317 from jwilders did not work either, I even tried to "compose" the containers on the command line.

Do you _please_ have any hint I can follow? Any help would be appreciated!

Kind regards,
Martin

[buchdag]

Are you certain your host is actually listening on port 443 and if yes that it is reachable from outside on this port ? Try netstat -lt for the former and nmap XXX.XXX.141.136 for the latter.

[martinhpunkt]

Hi,

thanks for your input.

netstat -lt
Aktive Internetverbindungen (Nur Server)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN     
tcp6       0      0 [::]:http               [::]:*                  LISTEN     
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN     
tcp6       0      0 [::]:https              [::]:*                  LISTEN 

This seems about right. I should've mentioned that I also checked this. However, haven't checked for nmap.

From my PC nmap returns:

nmap xxx.xxx.141.136

Starting Nmap 7.01 ( https://nmap.org ) at 2017-07-26 21:05 CEST
Nmap scan report for virtualhost.somehoster.com (xxx.xxx.141.136)
Host is up (0.070s latency).
Not shown: 995 closed ports
PORT    STATE    SERVICE
22/tcp  open     ssh
80/tcp  open     http
135/tcp filtered msrpc
139/tcp filtered netbios-ssn
445/tcp filtered microsoft-ds

Nmap done: 1 IP address (1 host up) scanned in 2.10 seconds

And this really bothers me now. How can that be?

docker ps
CONTAINER ID        IMAGE                                    COMMAND                  CREATED             STATUS              PORTS                                      NAMES
9684b9785224        jrcs/letsencrypt-nginx-proxy-companion   "/bin/bash /app/en..."   46 hours ago        Up 46 hours                                                    proxy-companion
99f0daf50d9b        jwilder/nginx-proxy:alpine               "/app/docker-entry..."   46 hours ago        Up 46 hours         0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   proxy
bfca61f50066        owncloud:9.1                             "docker-entrypoint..."   46 hours ago        Up 46 hours         80/tcp                                     owncloud
11c1544be1be        mariadb:latest                           "docker-entrypoint..."   47 hours ago        Up 47 hours         3306/tcp                                   owncloud-mariadb

Also:

iptables -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-ISOLATION  all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (3 references)
target     prot opt source               destination         
ACCEPT     tcp  --  0.0.0.0/0            172.18.0.2           tcp dpt:443
ACCEPT     tcp  --  0.0.0.0/0            172.18.0.2           tcp dpt:80

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination         
DROP       all  --  0.0.0.0/0            0.0.0.0/0           
DROP       all  --  0.0.0.0/0            0.0.0.0/0           
DROP       all  --  0.0.0.0/0            0.0.0.0/0           
DROP       all  --  0.0.0.0/0            0.0.0.0/0           
DROP       all  --  0.0.0.0/0            0.0.0.0/0           
DROP       all  --  0.0.0.0/0            0.0.0.0/0           
RETURN     all  --  0.0.0.0/0            0.0.0.0/0           

Chain l (0 references)
target     prot opt source               destination

...which makes sense to me, because

docker network inspect nw_letsencrypt-nginx
[
    {
        "Name": "nw_letsencrypt-nginx",
        "Id": "489f1344b82f5de88e71d84b52399db61edd1d8be5533bd7738c37685b7076c3",
        "Created": "2017-07-24T23:21:07.721003414+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "Containers": {
            "9684b97852245c781b9fb1fab89d8f022ec2f764160c015ca0ef5bbf8054ba5f": {
                "Name": "proxy-companion",
                "EndpointID": "35a7ce4d7630ffc51430fbaf1e05740568aa37251e95604876a495f69db16cb4",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            },
            "99f0daf50d9b2b55285645b8417642544a04db2d6a1a740ce4ced659b8b57268": {
                "Name": "proxy",
                "EndpointID": "53039348503c73700d9d4287414f437eb48ae193cac48baf6f7b3e891383a513",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

Any ideas? What am I missing?

Thanks you all again in advance!

Kind regards,
Martin

[buchdag]

Could you have a firewall / security group or something similar at the host provider level ?

Do you have se linux or another firewall like nftables running on this host ?

[martinhpunkt]

Hi,

sorry for the late response. Nothing at all. I haven't changed the iptables so only the rules from iptables are set. Because I needed the services I made a classical "non-dockerized" setup and everything works fine, so I can exclude anything from the hosting side. However, if I could switch over to Docker, it would still be great.

[alberto56]

It might be worth noting that it can take a few minutes for the port 443 to be available: I had to wait 5 minutes and everything works fine.

[grebois]

@martinhpunkt could you share the output of:

docker exec -it 99f0daf50d9b cat /etc/nginx/conf.d/default.conf

Thanks!

[bswinnerton]

👋 Hi @grebois, I'm having a very similar problem, and don't have any firewalls in place that would be causing problems.

@martinhpunkt could you share the output of:

docker exec -it 99f0daf50d9b cat /etc/nginx/conf.d/default.conf

Thanks!

When I do this, I notice nginx isn't wired up to 443. Could that be the problem?

# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
  default $http_x_forwarded_proto;
  ''      $scheme;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
  default $http_x_forwarded_port;
  ''      $server_port;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
  default upgrade;
  '' close;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Default dhparam
# Set appropriate X-Forwarded-Ssl header
map $scheme $proxy_x_forwarded_ssl {
  default off;
  https on;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent"';
access_log off;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
        server_name _; # This is just an invalid value which will never trigger on a real hostname.
        listen 80;
        access_log /var/log/nginx/access.log vhost;
        return 503;
}
# graphs.foo.bar
upstream graphs.foo.bar {
                                ## Can be connect with "bridge" network
                        # grafana
                        server 172.17.0.3:3000;
}
server {
        server_name graphs.foo.bar;
        listen 80 ;
        access_log /var/log/nginx/access.log vhost;
        include /etc/nginx/vhost.d/default;
        location / {
                proxy_pass http://graphs.foo.bar;
        }
}

I'm using the nginx.tmpl from https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion#separate-containers-recommended-method, specifically https://github.com/jwilder/nginx-proxy/blob/a6e8fae7f5959d1f7a4751f495d876ed174b612f/nginx.tmpl.

Update: After running docker exec -it 99f0daf50d9b ls -l /etc/nginx/certs, and saw that none existed, I realized that I had butchered the -v command of docker run. Recreating the containers fixed the problem for me 🎉 .

[rjsteinert]

Hi folks. I'm having a similar problem where port 443 connections are refused, there is no firewall in the way, but I am seeing the app on port 80. The way I know port 443 connections are not firewalled is that when I bind some http service to port 443, browsers and curl start complaining about handshake issues.

Getting to the point here, this is a very vanilla hello world example. Perhaps someone could point out my folly?

DOMAIN="p2p.earth"

docker run -d -p 80:80 -p 443:443 \
    --name nginx-proxy \
    -v $(pwd)/certs:/etc/nginx/certs:ro \
    -v /etc/nginx/vhost.d \
    -v /usr/share/nginx/html \
    -v /var/run/docker.sock:/tmp/docker.sock:ro \
    --label com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy \
    jwilder/nginx-proxy
docker run -d \
    -v $(pwd)/certs:/etc/nginx/certs:rw \
    -v /var/run/docker.sock:/var/run/docker.sock:ro \
    --volumes-from nginx-proxy \
    jrcs/letsencrypt-nginx-proxy-companion
docker run -d -e "VIRTUAL_HOST=$DOMAIN" nginx

[rjsteinert]

Any clues here?

➜  ssl-hello-world docker logs distracted_saha
Creating Diffie-Hellman group (can take several minutes...)
Generating DH parameters, 2048 bit long safe prime, generator 2
This is going to take a long time
➜  ssl-hello-world docker logs -f distracted_saha
Creating Diffie-Hellman group (can take several minutes...)
Generating DH parameters, 2048 bit long safe prime, generator 2
This is going to take a long time
..................................................................................................................................................................................................+..............................................................................................................+.............................................................+.................................................+...................................................+............................................+..............................................+................+................................+..+.......................................................................................................................................................................+.............................................................................................................................+.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+............................................................+.............................................................................+...........................................+...............................................................................................................................................................................+.....................+....................................................................................................................+..........................................................................................+...............................................................................................................................................................................................+.........................................................+................................................+...................................................................................................................................................................+.....................................................................................................................................................................................................................................+............................................................................+...............+..........................................................................................................................................+.........................................................+...................+..................................................................................+................+.........................................................................................................................................................................................................................................................................................................................................................................................................................................+..............................................................................................................+...............+.......................+....................+.............................................................................................................+............................................+....................+..............................................+...................................+.........................................................................................................................................................................................+...........+..................................................+...........................................+...............................................................+...........................+.............................+...........+.............................................................+.....................................+..........................................................................................+......................+...............................................................+.........................+........+............................................................................................+......................................................................................................................................................................+.................................................................................................................................................................................................................................................+....................+........................................................................................+..............................+..............................................+............................................................+.............+...................................................................................................+.........................+............................................................................................................................................................................................................................................................................+......................................................................................+.......................................+............................................+...................................................................................+.............................................................................................................................................+...........................................................................................................................................................+......................................................................................................+.........................................+.+.........................................................+.....................+..................................+..........................................................+...........................+.......................+......................................+.............+........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+....................................................................................+..........+.............................................................+............................................................................................................................................................................................................................+.............................................................................................................................................................+........................................+...........................................................................................+...........................................................................................................+..........+...............................................................................................................+.............+...............................+.......................................................................................................................................................................................................................................................+............+.....................................................................+.....................................................................................................................................+...............................................+..............+........................................................+.................................+...............................................................+.............................................................................................................................................................................................+.................................................................................................+...........................................................................................................................................................................+...............................................................................................+........................................................................................+........................+.............................................................................+.....+..................................................................+.................................................................................................................+.........................................++*++*
Sleep for 3600s
2017/12/25 15:42:36 Generated '/app/letsencrypt_service_data' from 3 containers
2017/12/25 15:42:36 Running '/app/update_certs'
2017/12/25 15:42:37 Watching docker events
Sleep for 3600s
2017/12/25 15:42:37 Contents of /app/letsencrypt_service_data did not change. Skipping notification '/app/update_certs'

[rjsteinert]

I added an extra environment variable -e "LETSENCRYPT_HOST=$DOMAIN" and it worked! You can see the current version of the hello world script here -> https://gist.github.com/rjsteinert/e819c547d2a37e76bb8735843f44f6bd

Does this log output look alright?

Sleep for 3600s
2017/12/25 15:54:54 Generated '/app/letsencrypt_service_data' from 3 containers
2017/12/25 15:54:54 Running '/app/update_certs'
2017/12/25 15:54:54 Watching docker events
/etc/nginx/certs/p2p.earth /app
Reloading nginx proxy (nginx-proxy)...
2017/12/25 15:54:54 Contents of /app/letsencrypt_service_data did not change. Skipping notification '/app/update_certs'
�2017/12/25 15:54:54 Generated '/etc/nginx/conf.d/default.conf' from 3 containers
2017/12/25 15:54:54 [notice] 55#55: signal process started
Creating/renewal p2p.earth certificates... (p2p.earth)
2017-12-25 15:54:55,805:INFO:simp_le:1538: Retrieving Let's Encrypt latest Terms of Service.
2017-12-25 15:54:56,817:INFO:simp_le:1356: Generating new account key
2017-12-25 15:54:58,926:WARNING:simp_le:1371: --email was not provided; ACME CA will have no way of contacting you.
2017-12-25 15:55:01,734:INFO:simp_le:1455: Generating new certificate private key
2017-12-25 15:55:03,460:INFO:simp_le:463: Saving account_key.json
2017-12-25 15:55:03,461:INFO:simp_le:463: Saving key.pem
2017-12-25 15:55:03,461:INFO:simp_le:463: Saving chain.pem
2017-12-25 15:55:03,462:INFO:simp_le:463: Saving fullchain.pem
2017-12-25 15:55:03,462:INFO:simp_le:463: Saving cert.pem
/app
Reloading nginx proxy (nginx-proxy)...
Q2017/12/25 15:55:03 Generated '/etc/nginx/conf.d/default.conf' from 3 containers
;2017/12/25 15:55:03 [notice] 64#64: signal process started
Sleep for 3600s

[buchdag]

@rjsteinert I think you should read the Let's Encrypt section of the doc.

[rjsteinert]

Thanks @buchdag. I went back and reread that. I noticed my hello world was missing the LETSENCRYPT_EMAIL variable.

FWIW, this is the hello world example I wish I had seen a long time ago, it might help out some other folks who come from where I'm coming from.

#!/bin/sh

DOMAIN="p2p.earth"
EMAIL="[email protected]"

echo ""
echo "Starting proxy and ssl companion."
echo ""

docker run -d -p 80:80 -p 443:443 \
    --name nginx-proxy \
    -v $(pwd)/certs:/etc/nginx/certs:ro \
    -v /etc/nginx/vhost.d \
    -v /usr/share/nginx/html \
    -v /var/run/docker.sock:/tmp/docker.sock:ro \
    --label com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy \
    jwilder/nginx-proxy

docker run -d \
    --name letsencrypt-nginx-proxy-companion \
    -v $(pwd)/certs:/etc/nginx/certs:rw \
    -v /var/run/docker.sock:/var/run/docker.sock:ro \
    --volumes-from nginx-proxy \
    jrcs/letsencrypt-nginx-proxy-companion

echo ""
echo "Sleeping while things start up..."
echo ""
sleep 60

echo ""
echo "Starting the app."
echo ""
docker run -d \
  --name app \
  -e "LETSENCRYPT_HOST=$DOMAIN" \
  -e "VIRTUAL_HOST=$DOMAIN" \
  -e "LETSENCRYPT_EMAIL=$EMAIL" \
  nginx

[weeger]

Thanks, for me the example usage of sleep 60 saved me.

[buchdag]

As the doc state pretty early:

NOTE: The first time this container is launched it generates a new Diffie-Hellman group file. This process can take several minutes to complete (be patient).

[buchdag]

Closing due to age and info about DH generation being outdated.