Compose: [1.7.0] Compose fails to execute tls, even when docker succeeds
Docker client handles certs without issue. Compose refuses to schedule any containers.
$ export DOCKER_HOST=tcp://leader.me.priv:2375
$ export DOCKER_TLS_VERIFY=1
$ export DOCKER_CERT_PATH=/home/ubuntu/.docker
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ docker-compose up -d service
ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
$ docker-compose version
docker-compose version 1.7.0, build 0d7bf73
docker-py version: 1.8.0
CPython version: 2.7.10
OpenSSL version: OpenSSL 1.0.2d 9 Jul 2015
$ docker version
Client:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Thu Mar 10 15:59:07 2016
OS/Arch: linux/amd64
Server:
Version: swarm/1.2.0
API version: 1.22
Go version: go1.5.4
Git commit: a6c1f14
Built: Wed Apr 13 05:58:31 UTC 2016
OS/Arch: linux/amd64
mattgiles
All 31 comments
cc @shin- is this a duplicate of the verify hostname issue?
dnephin
on 21 Apr 2016
No, that looks like a different issue. If the hostname matching fails the
error is explicit about it.
On Thu, Apr 21, 2016, 9:36 AM Daniel Nephin [email protected]
wrote:
cc @shin- https://github.com/shin- is this a duplicate of the verify
hostname issue?—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/docker/compose/issues/3365#issuecomment-213003593
shin-
on 21 Apr 2016
Fix floated in issue #890 does resolve this issue. Using something like:
$ CURL_CA_BUNDLE= docker-compose ps
Name Command State Ports
----------------------------------------------------------------
Suspect that makes this a duplicate?
mattgiles
on 30 Apr 2016
The only thing unsetting CURL_CA_BUNDLE this way is disable TLS verification entirely (because of the weird way it interacts with the code in requests, see here and here), so I definitely wouldn't consider it a fix.
Is docker-compose installed on Mac OS using Homebrew?
shin-
on 30 Apr 2016
Not a fix, but maybe suggests the underlying issue is the same as in the other thread.
All of the above is on an ubuntu machine that used pip to install docker-compose.
mattgiles
on 30 Apr 2016
Rather than disable the CURL_CA_BUNDLE you can run using:
CURL_CA_BUNDLE=~/.docker/machine/machines/default/ca.pem docker-compose ps
aboutlo
on 7 May 2016
@aboutlo it does not at least on my linux system:
➜ spark-env docker-compose --verbose version
docker-compose version 1.7.1, build 6c29830
docker-py version: 1.8.1
CPython version: 2.7.6
OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
➜ spark-env docker version
Client:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 5604cbe
Built: Tue Apr 26 23:30:23 2016
OS/Arch: linux/amd64
Server:
Version: swarm/1.2.2
API version: 1.22
Go version: go1.5.4
Git commit: 34e3da3
Built: Mon May 9 17:03:22 UTC 2016
OS/Arch: linux/amd64
Master machine is created using:
docker-machine create \
-d virtualbox \
--swarm --swarm-master \
--swarm-discovery="consul://${consul_ip}:8500" \
--engine-opt="cluster-store=consul://${consul_ip}:8500" \
--engine-opt="cluster-advertise=eth1:2376" \
master
So actually this gives same error
➜ spark-env CURL_CA_BUNDLE=~/.docker/machine/machines/master/ca.pem docker-compose ps
...
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connection.py", line 313, in _match_hostname
match_hostname(cert, asserted_hostname)
File "/usr/local/lib/python2.7/dist-packages/backports/ssl_match_hostname/__init__.py", line 151, in match_hostname
% (hostname, dnsnames[0]))
backports.ssl_match_hostname.CertificateError: hostname '192.168.99.107' doesn't match 'localhost'
as just this:
➜ spark-env docker-compose ps
...
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connection.py", line 313, in _match_hostname
match_hostname(cert, asserted_hostname)
File "/usr/local/lib/python2.7/dist-packages/backports/ssl_match_hostname/__init__.py", line 151, in match_hostname
% (hostname, dnsnames[0]))
backports.ssl_match_hostname.CertificateError: hostname '192.168.99.107' doesn't match 'localhost'
dennybaa
on 10 May 2016
@dennybaa it works for me on OSX. Here they are talking about the same issue: https://github.com/docker/compose/issues/890#issuecomment-217983585
aboutlo
on 11 May 2016
I think this is a regression in either docker-compose or one of it's deps. I typically install/upgrade/manager docker-compose via pip pretty much everywhere I use Docker and Docker Compose. After recently upgrading (just now) I'm getting this failure:
#!python
Traceback (most recent call last):
File "/Users/prologic/bin/docker-compose", line 11, in <module>
sys.exit(main())
File "/Users/prologic/lib/python2.7/site-packages/compose/cli/main.py", line 58, in main
command()
File "/Users/prologic/lib/python2.7/site-packages/compose/cli/main.py", line 109, in perform_command
handler(command, command_options)
File "/Users/prologic/lib/python2.7/site-packages/compose/cli/main.py", line 462, in ps
self.project.containers(service_names=options['SERVICE'], stopped=True) +
File "/Users/prologic/lib/python2.7/site-packages/compose/project.py", line 457, in containers
containers = self._labeled_containers(stopped, one_off)
File "/Users/prologic/lib/python2.7/site-packages/compose/project.py", line 448, in _labeled_containers
filters={'label': self.labels(one_off=one_off)})])
File "/Users/prologic/lib/python2.7/site-packages/docker/api/container.py", line 70, in containers
res = self._result(self._get(u, params=params), True)
File "/Users/prologic/lib/python2.7/site-packages/docker/utils/decorators.py", line 47, in inner
return f(self, *args, **kwargs)
File "/Users/prologic/lib/python2.7/site-packages/docker/client.py", line 120, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/Users/prologic/lib/python2.7/site-packages/requests/sessions.py", line 477, in get
return self.request('GET', url, **kwargs)
File "/Users/prologic/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/Users/prologic/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/Users/prologic/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/Users/prologic/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/Users/prologic/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
self._validate_conn(conn)
File "/Users/prologic/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 761, in _validate_conn
conn.connect()
File "/Users/prologic/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 253, in connect
match_hostname(cert, self.assert_hostname or hostname)
File "/Users/prologic/lib/python2.7/site-packages/backports/ssl_match_hostname/__init__.py", line 151, in match_hostname
% (hostname, dnsnames[0]))
backports.ssl_match_hostname.CertificateError: hostname '10.0.0.10' doesn't match 'localhost'
The only work-around in my case is in fact unsetting CURL_CA_BUNDLE. I may be on OS X when running docker-compose; however I'm talking to remote Docker hosts (typically RancherOS 0.4.5+)
prologic
on 4 Jun 2016
This is still happening in 1.8.0-rc1
darkermatter
on 30 Jun 2016
I have docker-compose installed via homebrew as well. I'm encountering the issue on v1.7.0 and v1.7.1. I rolled back to brew switch docker-compose 1.6.2 because that still works as normal.
The suggestion to use CURL_CA_BUNDLE= docker-compose ps kinda works but I get some nasty python SSL errors with a polite suggestion to go read a page that doesn't exist.
22:59:39 django-playground master » CURL_CA_BUNDLE= docker-compose ps 1 ↵
/usr/local/Cellar/docker-compose/1.7.1/libexec/vendor/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
/usr/local/Cellar/docker-compose/1.7.1/libexec/vendor/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
/usr/local/Cellar/docker-compose/1.7.1/libexec/vendor/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
/usr/local/Cellar/docker-compose/1.7.1/libexec/vendor/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
/usr/local/Cellar/docker-compose/1.7.1/libexec/vendor/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
/usr/local/Cellar/docker-compose/1.7.1/libexec/vendor/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
/usr/local/Cellar/docker-compose/1.7.1/libexec/vendor/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
Name Command State Ports
-------------------------------------------------------------------------------------------
django_data_1 /docker-entrypoint.sh true Up 5432/tcp
django_nginx_1 /usr/sbin/nginx Up 0.0.0.0:80->80/tcp
django_postgres_1 /docker-entrypoint.sh postgres Up 0.0.0.0:5432->5432/tcp
django_redis_1 /entrypoint.sh redis-server Up 0.0.0.0:6379->6379/tcp
django_webapp_1 /usr/local/bin/gunicorn su ... Up 8000/tcp
philbert
on 4 Jul 2016
I also tried regenerating the certs as suggested here but to no avail.
philbert
on 4 Jul 2016
Are there any updates on this? Because i ran into the same issue and i don't have any ideas anymore. I described everything on here.
manixx
on 14 Jul 2016
I am guessing that people running into this are following the instructions here https://docs.docker.com/engine/security/https/ to generate certificates.
The problem is this line:
$ echo subjectAltName = IP:10.10.10.20,IP:127.0.0.1 > extfile.cnf
The fix is simple:
$ echo subjectAltName = DNS:$HOST,IP:10.10.10.20,IP:127.0.0.1 > extfile.cnf
After reading hints regarding potential a subjectAltName "issue", I've debugged a bit and discovered that not all subjectAltName entries were actually being used, e.g. there was no "IP Address" entry being evaluated although my cert contained them. Only the DNS entry containing the IP was evaluated and failed - in my case.
I'll discovered that the SubjectAltName was provided by a wrapper around ndg-httpsclient. PIP showed me that I had 0.4.0 installed. After uninstalling ndg-httpsclient completely using pip the problem was solved for me.
My configuration is
docker-compose version 1.7.1, build unknown
docker-py version: 1.8.1
CPython version: 2.7.10
OpenSSL version: OpenSSL 0.9.8zh 14 Jan 2016
boxsterman
on 21 Jul 2016
Seems like sound bug to me, but it also seems not to be a docker-compose one...
$ dig gemini-clearcare-docker
...
;; ANSWER SECTION:
gemini-clearcare-docker. 0 IN A 192.168.7.57
...
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1dbf305ed10d clearcare_worker "celery worker --app " 19 hours ago Exited (0) About an hour ago cc_worker
...
$ docker-compose -f docker-compose-v3.yml up
Traceback (most recent call last):
File "<string>", line 3, in <module>
File "compose/cli/main.py", line 60, in main
File "compose/cli/main.py", line 111, in perform_command
File "compose/cli/main.py", line 814, in up
File "compose/project.py", line 372, in up
File "compose/project.py", line 413, in initialize
File "compose/network.py", line 162, in initialize
File "compose/network.py", line 30, in ensure
File "compose/network.py", line 82, in inspect
File ".tox/py27/lib/python2.7/site-packages/docker/utils/decorators.py", line 35, in wrapper
File ".tox/py27/lib/python2.7/site-packages/docker/api/network.py", line 56, in inspect_network
File ".tox/py27/lib/python2.7/site-packages/docker/utils/decorators.py", line 47, in inner
File ".tox/py27/lib/python2.7/site-packages/docker/client.py", line 138, in _get
File ".tox/py27/lib/python2.7/site-packages/requests/sessions.py", line 477, in get
File ".tox/py27/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
File ".tox/py27/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
File ".tox/py27/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
File ".tox/py27/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
File ".tox/py27/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
File ".tox/py27/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 761, in _validate_conn
File ".tox/py27/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 253, in connect
File ".tox/py27/lib/python2.7/site-packages/backports/ssl_match_hostname/__init__.py", line 147, in match_hostname
backports.ssl_match_hostname.CertificateError: hostname 'gemini-clearcare-docker' doesn't match either of '192.168.7.57', '127.0.0.1'
docker-compose returned -1
danbaragan
on 22 Jul 2016
Actually, I don't think it's a bug, it's a breaking fix. The software is now behaving correctly as specified by the RFCs.
The info is here: http://wiki.cacert.org/FAQ/subjectAltName
What is subjectAltName ?
subjectAltName specifies additional subject identities, but for host names (and everything else defined for subjectAltName) :
subjectAltName must always be used (RFC 2818 4.2.1.7, 1. paragraph). CN is only evaluated if subjectAltName is not present and only for compatibility with old, non-compliant software. So if you set subjectAltName, you have to use it for all host names, email addresses, etc., not just the "additional" ones.
darkermatter
on 22 Jul 2016
darkmatter's hint on adding a DNS:FQDN entry to the subject alternate name worked for me.
I just created a new csr for the server, and signed it. Then restarted docker. After that, docker-compose (1.7.1) worked without complaints.
(backup /root/tls)
cd /root/tls
openssl req -subj "/CN=<FQDN>" -sha256 -new -key server-key.pem -out server.csr
openssl x509 -req -days 9999 -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extfile extfile.cnf
asgerjensen
on 27 Jul 2016
Cheers @boxsterman! Uninstalling ndg-httpsclient-0.4.0 fixed compose for me!
philbert
on 5 Aug 2016
@darkermatter is right! I followed these instructions:
https://docs.docker.com/engine/security/https/
And I got the docker-compose certficiate hostname error. The instructions need to be updated. Where it says:
$ echo subjectAltName = IP:10.10.10.20,IP:127.0.0.1 > extfile.cnf
It should say:
$ echo subjectAltName = DNS:$HOST,IP:10.10.10.20,IP:127.0.0.1 > extfile.cnf
And it would be helpful that it also stated that the DNS:$HOST part is mandatory according to the specifications of subjectAltName.
jonaskello
on 25 Aug 2016
I debugged docker-compose and docker-py and figured out that you should either use environment variables or options in command. You should not mix these . If you even specify --tls in command then you will have to specify all the options as the TLSConfig object, as now TLSConfig object is created completely from the command options and operide the TFSConfig object created from the environment variable.
jitekuma
on 8 Mar 2017
The docs have been updated to say $ echo subjectAltName = DNS:$HOST,IP:10.10.10.20,IP:127.0.0.1 > extfile.cnf and I did that and still had this issue.
I fixed this for myself by appending the ca cert to the server cert on the docker host. I believe I needed this because my docker ca was an intermediate CA for a self signed CA that I use for all my internal stuff.
WyseNynja
on 13 Mar 2017
I created a machine on DigitalOcean using docker-machine and I am getting the same error as mentioned above, while trying to do the following:
docker-compose up
backports.ssl_match_hostname.CertificateError: hostname 'X.X.X.X' doesn't match 'localhost'
rajat1saxena
on 16 Jun 2017
You do realise that the 10.10.10.20 is not a magic number, right? It should be the IP of the Docker host. Perhaps the documentation should read:
$ echo subjectAltName = DNS:$HOST,IP:$HOST_IP,DNS:localhost,IP:127.0.0.1 > extfile.cnf
darkermatter
on 16 Jun 2017
Also, it looks like you're using localhost to reach your Docker engine. Why not just use the socket instead of TLS?
darkermatter
on 16 Jun 2017
@darkermatter I think you've not read my comment properly. I used docker-machine create command to create a new remote docker host. This command, I guess, configures the certs automatically for TLS transport. I mean, that's what the logs told me.
So, after I am done with exec $(docker-machine env docker-host), I should be all set to run docker-compose up on remote docker host.
That's when I encountered that error. I am not trying to deploy on 'localhost'.
rajat1saxena
on 16 Jun 2017
@rajat1saxena Yes, sorry, I just zoomed in on the error. What do the environment variables look like?
darkermatter
on 16 Jun 2017
@rajat1saxena Also, you didn't give any versions of anything
darkermatter
on 16 Jun 2017
I updated to docker-compose version '1.14.0-rc2', things seem to be working fine, now.
rajat1saxena
on 16 Jun 2017
Got the same error as prologic on docker-compose 1.8.0. Upgrading to 1.22 solved it.
kgawrys
on 28 Aug 2018
I just spent a long time fighting this problem. It turned it that if you use the same CN when generating your CA certificate (https://docs.docker.com/engine/security/https/ says this is the way to go via the $HOST variable) it doesn't seem to work. I changed the CN of my CA certificate to something else (not the hostname of the server) and boom, docker-compose works.
nickdnk
on 4 Oct 2020
Related issues
guycalledseven
·
3Comments
darkrasid
·
3Comments
AvdN
·
3Comments
DhairyashilBhosale
·
3Comments
bergtwvd
·
3Comments
Most helpful comment
I am guessing that people running into this are following the instructions here https://docs.docker.com/engine/security/https/ to generate certificates.
The problem is this line:
The fix is simple: