Machine: docker-compose build -> SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
!!! SOLUTION !!!
(thanks for @pmahoney https://github.com/docker/compose/issues/890#issuecomment-159750418)
echo $CURL_CA_BUNDLE
unset CURL_CA_BUNDLE
docker-compose up
!!! END OF SOLUTION !!!
Hi,
As I was advised in https://github.com/docker/compose/issues/890#issuecomment-141495624 (thanks @posita for the help), I want to raise a new issue that I can not make docker-compose works on my mac.
It is worth to state, that I have python and OpenSSL installed via brew, every package is up to date.
I have tried to install docker & co in two ways:
- using brew
- using *.pkg
But everytime I got the SSL issue.
I have the next setup currently:
OpenSSL 0.9.8zg 14 July 2015
docker-compose version: 1.4.0
docker-machine version 0.4.1 (e2c88d6)
Python 2.7.10
And that's what I have when doing docker-compose:
As @posita suggested, the issue has place because of the certificates misconfiguration:
https://github.com/docker/compose/issues/890#issuecomment-141473612
And the certificates are created during docker-machine provisioning and creation, so I have created an issues here.
However, I am not a specialist in this area :( But I want to have docker-compose work on my Mac still.
As I understand, the most full debug information is the result of the next command:
(
set -x
eval $( docker-machine env shiny-new-machine-74d5a19e )
env | grep DOCKER
ls -al "${DOCKER_CERT_PATH}"
openssl x509 -in "${DOCKER_CERT_PATH}/cert.pem" -text
openssl x509 -in "${DOCKER_CERT_PATH}/ca.pem" -text
docker-compose --verbose version
docker-compose --verbose ps
DOCKER_TLS_VERIFY=0 docker-compose --verbose ps
) >"${HOME}/Desktop/docker-compose-890-outerr-$( date -u +%Y-%m-%dT%H:%M:%SZ ).txt" 2>&1
Here you can find it: http://pastebin.com/vWqZgVKi
Any advice is appreciated.
Thanks for the help!
Regards,
PavelPolyakov
All 79 comments
Any thoughts are appreciated, I'm stuck, and do not want to reinstall my OS to make docker-compose work :)
PavelPolyakov
on 19 Sep 2015
Thanks for reporting. Are you happening to be using a boot2docker vm with machine?
ehazlett
on 20 Sep 2015
This may be a red herring, but one thing I noticed is that docker-machine+boot2docker does not rely on boot2docker's default generated certs. Rather, machine seems to generate its own. When it does, the Subject of cert.pem is the same as the Issuer (i.e., the same as the Subject of ca.pem).
Due to a bug with certain (recent) versions of OpenSSL, this is substantially the same behavior that caused similar errors when using docker tools directly with boot2docker (i.e., _without_ docker-machine; see also boot2docker/boot2docker#1029 and SvenDowideit/generate_cert#10). A fix was released with boot2docker 1.8.0 (@eaaaeae). However, as I mentioned, the (potentially) problematic behavior seems to be preserved when using machine, due to machine's own certificate handling.
My recommendation is to patch machine to make sure that the Subject of cert.pem differs from ca.pem and then have @PavelPolyakov test it in his environment. If the patched machine makes the error go away for him, I think you've got your fix. If not, then it's something else that is beyond my understanding of the problem.
I would submit a PR, but I am not (yet) very skilled with Go, nor am I familiar with how machine handles certificate creation/installation with boot2docker. Unfortunately my current schedule does not allow me to dedicate the requisite time to come up to speed. In lieu of that, I hope this summary is useful.
posita
on 20 Sep 2015
@ehazlett
Sorry, I haven't understood the question.
Here is how the machine was created:
docker-machine create --driver virtualbox my-docker-machine
The latest image was downloaded from the internet.
@posita
Thanks for the help, ready to test anything which could lead to success.
PavelPolyakov
on 20 Sep 2015
@ehazlett
Hi guys, could you tell me if I can help somehow? And how we can make it working?
Regards,
PavelPolyakov
on 22 Sep 2015
@rklt
Tried that:
Seems like the result is the same.
Any other thoughts?
PavelPolyakov
on 23 Sep 2015
@PavelPolyakov Sorry, I was wrong…
I am looking for a solution ...
rkit
on 23 Sep 2015
@PavelPolyakov Done. That's what did:
- Removed all machines
- Run
docker-machine regenerate-certs - Add
export DOCKER_HOST=tcp://192.168.99.100:2376
export DOCKER_CERT_PATH=/Users/$USERNAME/.docker/machine/certs
export DOCKER_TLS_VERIFY=1
The error was gone.
rkit
on 23 Sep 2015
@rkit
Thanks for the solution.
Trying to implement it.
However, couple of questions:
- When I have removed the machine, I think I am not able to regenerate-certs.
[~/tmp/microservices-workshop/msworkshop/step4]$ docker-machine rm default
Successfully removed default
[~/tmp/microservices-workshop/msworkshop/step4]$ docker-machine regenerate-certs *[master]
Regenerate TLS machine certs? Warning: this is irreversible. (y/n): y
Regenerating TLS certificates
Error: Expected to get one or more machine names as arguments.
- Shouldn't this IP be dynamic, and is available only after the docker-machine is created?
export DOCKER_HOST=tcp://192.168.99.100:2376
export DOCKER_CERT_PATH=/Users/$USERNAME/.docker/machine/certs
export DOCKER_TLS_VERIFY=1
- Yes, you need to create a machine
- Dynamic, yes. To change env, you need to execute:
eval "$(docker-machine env dev)", but default env from bash profile, and I added this line in the config:
export DOCKER_CERT_PATH=/Users/$USERNAME/.docker/machine/certs
After that there was no error.
rkit
on 23 Sep 2015
Am I interpreting correctly, that requiring regenerate-certs is a work-around that confirms my speculation about the cause of this issue? I.E., that docker-machine has a sister issue to boot2docker/boot2docker#808 (see also docker/docker-py#465, docker/compose#890) which was not resolved with boot2docker/boot2docker#1029 because docker-machine ignores the boot2docker default certificates and creates its own? Is that an accurate summary?
posita
on 23 Sep 2015
@rkit
Sorry, I have the same thing still :(
What I did:
docker-machine rm defaultdocker-machine create --driver virtualbox defaultdocker-machine regenerate-certs defaulteval "$(docker-machine env default)" export DOCKER_CERT_PATH=/Users/$USERNAME/.docker/machine/certs
Have I missed something? Any thoughts how I can overcome it?
Can I provide some additional debug?
PavelPolyakov
on 23 Sep 2015
@PavelPolyakov, after doing the regenerate-certs and eval, can you do:
for i in ca cert ; do c="${DOCKER_CERT_PATH}/${i}.pem" ; ( set -x ; openssl x509 -in "${c}" -text | grep -E '^ +(Issuer|Subject): ' ) ; done
@posita
Thanks, here we are:
PavelPolyakov
on 23 Sep 2015
Okay, so regenerate-certs doesn't actually address the (potential) problem where ca.pem Subject == cert.pem Subject. :disappointed: In other words:
+-zsh:xxx> openssl x509 -in /.../.docker/machine/certs/ca.pem -text
+-zsh:xxx> grep ... -E '^ +Subject: '
Subject: O=[thing] # <<<-- THIS SHOULD *NOT*
+-zsh:xxx> openssl x509 -in /.../.docker/machine/certs/cert.pem -text
+-zsh:xxx> grep ... -E '^ +Subject: '
Subject: O=[thing] # <<<-- EQUAL THIS
If it does, and you have the "wrong" version of OpenSSL (i.e., any version that exhibits this bug), you will receive the [SSL: CERTIFICATE_VERIFY_FAILED] error.
I suspect this is why you're still having issues, and why @rkit's suggestion does not work.
posita
on 23 Sep 2015
Hi guys, @rkit , @ehazlett , any thoughts how I can overcome that?
PavelPolyakov
on 25 Sep 2015
@PavelPolyakov, for giggles, can you try this (replace [machine-name] with your actual machine name)?
% eval $( docker-machine env [machine-name] )
% docker-machine ssh [machine-name]
...
Boot2Docker version 1.8.2, build master : aba6192 - Thu Sep 10 20:58:17 UTC 2015
Docker version 1.8.2, build 0a8c2e3
docker@boot2docker:~$ rm -fv ~docker/.docker/* # get rid of copies of certificates we're about to destroy
removed '/home/docker/.docker/ca.pem'
removed '/home/docker/.docker/cert.pem'
removed '/home/docker/.docker/key.pem'
docker@boot2docker:~$ sudo -s
root@boot2docker:/home/docker# cat /var/lib/boot2docker/profile # so I can see what is being overridden
...
root@boot2docker:/home/docker# rm -fv /var/lib/boot2docker/*.pem /var/lib/boot2docker/tls/* # remove any existing certificates
...
root@boot2docker:/home/docker# /usr/local/etc/init.d/docker restart
... # should regenerate default certificates
root@boot2docker:/home/docker# exit
docker@boot2docker:~$ exit
% rm -v "${DOCKER_CERT_PATH}"/*.pem # remove host-side (stale) certificates
...
% docker-machine scp [machine-name]:/home/docker/.docker/\*.pem "${DOCKER_CERT_PATH}" # copy newly-created certificates from machine to host
...
% docker ps
...
% docker-compose ps
...
% openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -key "${DOCKER_CERT_PATH}/key.pem" -cert "${DOCKER_CERT_PATH}/cert.pem" -CAfile "${DOCKER_CERT_PATH}/ca.pem" -tls1 </dev/null
...
Basically, we're trying to avoid having docker-machine create the certificates, and instead rely on boot2docker's default mechanism. By logging in to the boot2docker instance, removing the certificates, and then restarting docker, we trigger that process. Then we replace the host-side certificates with the ones we just got boot2docker to (re)create.
posita
on 25 Sep 2015
@posita
Can't regenerate certificates
First:
Boot2Docker version 1.8.2, build master : aba6192 - Thu Sep 10 20:58:17 UTC 2015
Docker version 1.8.2, build 0a8c2e3
docker@default:~$ rm -fv ~docker/.docker/*
removed '/home/docker/.docker/ca.pem'
removed '/home/docker/.docker/cert.pem'
removed '/home/docker/.docker/key.pem'
docker@default:~$ sudo -s
root@default:/home/docker# cat /var/lib/boot2docker/profile
EXTRA_ARGS='
--label provider=virtualbox
'
CACERT=/var/lib/boot2docker/ca.pem
DOCKER_HOST='-H tcp://0.0.0.0:2376'
DOCKER_STORAGE=aufs
DOCKER_TLS=auto
SERVERKEY=/var/lib/boot2docker/server-key.pem
SERVERCERT=/var/lib/boot2docker/server.pem
root@default:/home/docker# rm -fv /var/lib/boot2docker/tls/*
removed '/var/lib/boot2docker/tls/ca.pem'
removed '/var/lib/boot2docker/tls/cakey.pem'
removed '/var/lib/boot2docker/tls/cert.pem'
removed '/var/lib/boot2docker/tls/hostnames'
removed '/var/lib/boot2docker/tls/key.pem'
removed '/var/lib/boot2docker/tls/server.pem'
removed '/var/lib/boot2docker/tls/serverkey.pem'
Then:
root@default:~# /usr/local/etc/init.d/docker restart
Need TLS certs for default,127.0.0.1,10.0.2.15,192.168.99.104
-------------------
Generating CA cert
2015/09/25 18:06:48 Preventing overwrite: the following files already exist: "/var/lib/boot2docker/ca.pem". To overwrite files, add `--overwrite`.
rm: can't remove '/var/lib/boot2docker/server.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/server-key.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls/cert.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls/key.pem': No such file or directory
Generate server cert
/usr/local/bin/generate_cert --host=default,127.0.0.1,10.0.2.15,192.168.99.104 --ca=/var/lib/boot2docker/ca.pem --ca-key=/var/lib/boot2docker/tls/cakey.pem --cert=/var/lib/boot2docker/server.pem --key=/var/lib/boot2docker/server-key.pem --org=Boot2Docker
2015/09/25 18:06:48 Generating a server cert
2015/09/25 18:06:48 Failure to generate cert: open /var/lib/boot2docker/tls/cakey.pem: no such file or directory
Generating client cert
2015/09/25 18:06:48 no --host parameters, making a client cert
2015/09/25 18:06:48 Failure to generate cert: open /var/lib/boot2docker/tls/cakey.pem: no such file or directory
cp: can't stat '/var/lib/boot2docker/tls/cert.pem': No such file or directory
cp: can't stat '/var/lib/boot2docker/tls/key.pem': No such file or directory
Ah, okay. I was afraid of that (/var/lib/boot2docker/profile overrides some default locations). Try the same thing, but instead of:
root@default:/home/docker# rm -fv /var/lib/boot2docker/tls/*
Do:
root@default:/home/docker# rm -fv /var/lib/boot2docker/*.pem /var/lib/boot2docker/tls/*
@posita
rm -fv /var/lib/boot2docker/*.pem /var/lib/boot2docker/tls/*
removed '/var/lib/boot2docker/ca.pem'
removed '/var/lib/boot2docker/tls/hostnames'
root@default:~# /usr/local/etc/init.d/docker restart
Need TLS certs for default,127.0.0.1,10.0.2.15,192.168.99.104
-------------------
Generating CA cert
2015/09/25 18:18:14 Generating a new certificate authority.
rm: can't remove '/var/lib/boot2docker/server.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/server-key.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls/cert.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls/key.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls//hostnames': No such file or directory
Generate server cert
/usr/local/bin/generate_cert --host=default,127.0.0.1,10.0.2.15,192.168.99.104 --ca=/var/lib/boot2docker/ca.pem --ca-key=/var/lib/boot2docker/tls/cakey.pem --cert=/var/lib/boot2docker/server.pem --key=/var/lib/boot2docker/server-key.pem --org=Boot2Docker
2015/09/25 18:18:15 Generating a server cert
Generating client cert
2015/09/25 18:18:15 no --host parameters, making a client cert
root@default:~# ls /var/lib/boot2docker/tls/
cakey.pem cert.pem hostnames key.pem
(exit; exit)
but then:
[~/tmp/microservices-workshop/msworkshop/step4]$ docker ps *[master]
An error occurred trying to connect: Get https://192.168.99.104:2376/v1.20/containers/json: x509: certificate signed by unknown authority
[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose ps *[master]
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
the last command:
openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -key "${DOCKER_CERT_PATH}/key.pem"
CONNECTED(00000003)
depth=0 O = default
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = default
verify error:num=21:unable to verify the first certificate
verify return:1
140735289209680:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1472:SSL alert number 42
140735289209680:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
0 s:/O=default
i:/O=PavelPolyakov
-----BEGIN CERTIFICATE-----
MIIDADCCAeqgAwIBAgIQR4h88WLsll8AkYK6MIeEMTALBgkqhkiG9w0BAQswGDEW
MBQGA1UEChMNUGF2ZWxQb2x5YWtvdjAeFw0xNTA5MjMxNjMwMDBaFw0xODA5MDcx
NjMwMDBaMBIxEDAOBgNVBAoTB2RlZmF1bHQwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQC7MYqKigy4aJCPiaCQrOKypICUYbIvLYtHKCYG7d9d274MuYWf
dH7mMKTTu8M3GOrAzUj+tvHDAHIZgV0nDMz5qwEkYI2JD7N7hcHQ91JUPJVVpV3e
v2v4hzFra8KB/9h4grjTwAnMLLJNNcWUwu8JPvy3xRmBjpl8UIVY1+VLYwiLodfS
7bWBdqYy2E0sxoJjgtNVPWgxkOloApOrQhq0t/HcyXv0mA3hVPG+9J1r5Mxqs0nY
/qIMKGpywhTGqsWfDnBe22Bf2tFHagDN0frLRrIrF8HT8Xggwawz9pN4RpRWUrI0
lAsgxznXZR1xnvffbxmdy2P94BX1UHqzVy/dAgMBAAGjUDBOMA4GA1UdDwEB/wQE
AwIAqDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIw
ADAPBgNVHREECDAGhwTAqGNoMAsGCSqGSIb3DQEBCwOCAQEAYoGQ5x3wXGVxZey9
dcxyVnBkYSZiInVIeI+bdsKtxKUv7HoXRW1iLyKSeNq8BGQzpfaEPkDtfblcEctz
U0Oj7jS4MlrdzGK1A75Bl8Bo1VDpu146M8cp/Wa09tjmrJktMrVOVZVQteDCPmZo
RIgaHWH0gRJPOy2SCTvLhmyZTK0bO2YKpgvjf2f7NMw6GBJi1hgcZuw3wDjVutjS
2hOWh2HeVyguu1na+YTpZ0csH+MSrKt79ulBpZ3SlYiG5pDFJ5Bjf74OpL0KW30Z
sBA51T1y3Xfyohb27XT28mJLa5Q1cH1PTnComd4K9AlVy7zY0u/MlDN8kaZb0ggX
pqpv2Q==
-----END CERTIFICATE-----
---
Server certificate
subject=/O=default
issuer=/O=PavelPolyakov
---
Acceptable client certificate CA names
/O=PavelPolyakov
Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256
Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1247 bytes and written 138 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID:
Session-ID-ctx:
Master-Key: C628B5966CBC7A394199AF9D0ECA66BF01744B865D7595D3F3080158A9C9AEDA72B53714D24FCFF11F542F38CBB5CC50
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1443205317
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
---
Did you do this after exiting docker-machine ssh [machine-name] but before doing docker ps?
% rm -v "${DOCKER_CERT_PATH}"/*.pem
...
% docker-machine scp [machine-name]:/home/docker/.docker/\*.pem "${DOCKER_CERT_PATH}"
...
Keep in mind, you still need to replace your host-side certificates with the ones you just got boot2docker to (re)create before executing docker or docker-compose commands on the host. Otherwise, you'll get certificate mismatches like the ones you experienced.
By the way, you'll probably need to start over from the beginning of the steps I outline in my https://github.com/docker/machine/issues/1880#issuecomment-143227151. I've updated that comment to reflect the change in my https://github.com/docker/machine/issues/1880#issuecomment-143312225.
posita
on 25 Sep 2015
In my $DOCKER_CERT_PATH I have new (I think so) certificates located:
[~/.docker/machine/certs]$ ls
ca.pem cert.pem key.pem
However, docker ps says:
[~/.docker/machine/certs]$ docker ps
An error occurred trying to connect: Get https://192.168.99.104:2376/v1.20/containers/json: x509: certificate signed by unknown authority
I'm assuming your screenshot was just to show the details of cert.pem, not that you have added any of these *.pem files to any of your OS X keychains, correct?
There is a mismatch somewhere. What does this give you right now?
openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -CAfile "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem"
If you aren't getting output that looks something like the following, you probably need to start over from scratch with the steps in my https://github.com/docker/machine/issues/1880#issuecomment-143227151.
...
---
Certificate chain
0 s:/O=Boot2Docker
i:/O=Boot2DockerCA
...
---
Server certificate
subject=/O=Boot2Docker
issuer=/O=Boot2DockerCA
---
Acceptable client certificate CA names
/O=Boot2DockerCA
...
Yes, this is just info, I haven't add them to the keychain. But they are located in my DOCKER_CERT_PATH.
However, I still have:
Certificate chain
0 s:/O=default
i:/O=PavelPolyakov
and etc. , will try to startover again, thanks for the help!
PavelPolyakov
on 25 Sep 2015
I am having the same problem. I've followed the steps listed above and have the following signed certs:
+ openssl x509 -in /Users/cschmid/.docker/machine/certs/ca.pem -text
+ grep -E '^ +(Issuer|Subject): '
Issuer: O=Boot2DockerCA
Subject: O=Boot2DockerCA
+ openssl x509 -in /Users/cschmid/.docker/machine/certs/cert.pem -text
+ grep -E '^ +(Issuer|Subject): '
Issuer: O=Boot2DockerCA
Subject: O=Boot2Docker
But when I try to run docker-compose I continue to get the error
Chriss-MacBook-Pro:docker cschmid$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Chriss-MacBook-Pro:docker cschmid$ docker-compose ps
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
I'm running docker-compose 1.4.2, docker 1.8.2 and have openssl v1.0.1j_1 installed.
cischmidt
on 25 Sep 2015
@cischmidt, what does this give you (in the same env)?
openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -CAfile "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null
An error response, but it may be because I have to connect to docker via localhost because of my Cisco VPN client:
Chriss-MacBook-Pro:docker cschmid$ openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -CAfile "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null
CONNECTED(00000003)
50891:error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version:/SourceCache/OpenSSL098/OpenSSL098-52.40.1/src/ssl/s23_clnt.c:593:
My docker environment variables are:
DOCKER_HOST=tcp://127.0.0.1:2376
DOCKER_MACHINE_NAME=default
DOCKER_TLS_VERIFY=1
DOCKER_CERT_PATH=/Users/cschmid/.docker/machine/certs
@cischmidt, you're on OS X, correct? I don't understand your DOCKER_HOST setting. :confused:
posita
on 25 Sep 2015
I am on Mac OS X. I port-forward from my virtualbox VM to localhost:2376 to get around routing problems that the VPN introduces every time I connect to it. Docker related commands work well using this configuration, and I remember using docker-compose a few months ago without problem (today is the first day for some time that I began using docker-compose again).
cischmidt
on 25 Sep 2015
@cischmidt, just to confirm, localhost:2376 points to your docker-machine VM running on your local machine? Is your docker-machine VM boot2docker (e.g., running in VirtualBox)?
@PavelPolyakov and @cischmidt, can you do the following after having run the steps above in my https://github.com/docker/machine/issues/1880#issuecomment-143227151? There's no need to run through those steps again if you're still in the same environment and the machine is still up.
% docker-machine ssh [machine-name]
docker@boot2docker:~$ export DOCKER_CERT_PATH="${HOME}/.docker" DOCKER_TLS_VERIFY=1 DOCKER_HOST=tcp://127.0.0.1:2376
docker@boot2docker:~$ docker ps
...
docker@boot2docker:~$ openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -CAfile "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null
...
docker@boot2docker:~$ exit
@ehazlett, does docker-machine do some kind of SSL proxy or tunnel magic with boot2docker?
posita
on 25 Sep 2015
@cischmidt, just to confirm, localhost:2376 points to your docker-machine VM running on your local machine? Is your docker-machine VM boot2docker (e.g., running in VirtualBox)?
Yes, that is correct. I'm running docker-machine against a boot2docker VM within VirtualBox. It is also the converted VM that the old Mac version of boot2docker created for me.
Chriss-MacBook-Pro:docker cschmid$ docker-machine ssh default
...
Boot2Docker version 1.8.1, build master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
Docker version 1.8.1, build d12ea79
docker@default:~$ export DOCKER_CERT_PATH="${HOME}/.docker" DOCKER_TLS_VERIFY=1 DOCKER_HOST=tcp://127.0.0.1:2376
docker@default:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
docker@default:~$ openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -CAfile "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null
CONNECTED(00000003)
depth=1 O = Boot2DockerCA
verify return:1
depth=0 O = Boot2Docker
verify return:1
139887085803152:error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate:s3_pkt.c:1197:SSL alert number 42
139887085803152:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
0 s:/O=Boot2Docker
i:/O=Boot2DockerCA
-----BEGIN CERTIFICATE-----
MIIC9TCCAd+gAwIBAgIRAKmIA6DlGQTnpVHJLetbUvAwCwYJKoZIhvcNAQELMBgx
FjAUBgNVBAoTDUJvb3QyRG9ja2VyQ0EwHhcNMTUwOTI1MDM1NDEyWhcNMTYwOTI0
MDM1NDEyWjAWMRQwEgYDVQQKEwtCb290MkRvY2tlcjCCASIwDQYJKoZIhvcNAQEB
BQADggEPADCCAQoCggEBAMZBiSJRdlzGCozwZAswmKl14ZvMoP7AnK7E5+mba3Xi
NRNjaIBP7YN7s2N121chvoKO0NmMM+h5AyCH6OU7R2HXOLNmZZu5O7bzQ7/sishm
NIdi625zw/alaHieikx2YSK4e6ZXuj+ws/CcoNpTHatSZnEl6uUCZHMkNJWmGYMQ
tWqyRFkm84N7yTtTzNgIaWC530TKDvGRoTl3y/VPG8DnHeMSHKc4xTKeoVTvjaDM
jDXr83/sFfmodfoavAQKVpXoWb+ZcjA1Ma/u96AMW9g0vbnxFfzTr1BwhGxGVwzY
eqiIKhz7YDdq4INJbPsRQ/q8TR86Tp5qPaRQLQLMdVMCAwEAAaNAMD4wDgYDVR0P
AQH/BAQDAgCgMAwGA1UdEwEB/wQCMAAwHgYDVR0RBBcwFYIHZGVmYXVsdIcEfwAA
AYcECgACDzALBgkqhkiG9w0BAQsDggEBAHRHV0jFr1duyxIE7O4rM++0yO6O+o/P
b6cs+mqp8dvURCOZBC8usNWJGX+t7k3qMiri0qsE2fBJ9WaIRjGdUNjOrMU4ds9d
Bk+BYkTZ1syoE+NoMXmE6fB+aPpqUH2HG5gKS1o8uHyTqXYrkb8mk8AtTomEEwPX
aTyCbR4uQaEDngwsFp/HDJA2LgkWcPyVW+pe2p2n75mHzyYooePp/KTHKMNzmpa3
nt9W4O7XKHn1gGHLTH3LLJBqGlBviNNPT72eE1K44H1ptUO0T/gRZuWvWme3/1dv
rOWT5HqmxSbhj3RG5reWRYp+shO14JH2VM5Ena/xMVZzkudLrl18pGw=
-----END CERTIFICATE-----
---
Server certificate
subject=/O=Boot2Docker
issuer=/O=Boot2DockerCA
---
Acceptable client certificate CA names
/O=Boot2DockerCA
---
SSL handshake has read 1228 bytes and written 146 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : ECDHE-RSA-AES256-SHA
Session-ID:
Session-ID-ctx:
Master-Key: 30F5D5CD8276F7BAADE68E164251359B9D7D8531BA9FF288D2BAF998AF651B161C094D17A2BF9657B03B30278A7FF07F
Key-Arg : None
PSK identity: None
PSK identity hint: None
Start Time: 1443159065
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
@cischmidt, your boot2docker instance has the correct certificates (i.e., the ones generated by the default process in boot2docker, and _not_ the ones generated by docker-machine). Assuming those match your certs on your host (you can examine them with openssl x509 -in "${DOCKER_CERT_PATH}/cert.pem" -text and openssl x509 -in "${DOCKER_CERT_PATH}/ca.pem" -text), then there's something else going on, probably with your VPN routing or something. You're also getting a different error than @PavelPolyakov which leads me to believe yours is a different issue than the one described here.
posita
on 25 Sep 2015
@posita
For some reason, the flow which is described by you, doesn't lead me to the correct certificates:
Boot2Docker version 1.8.2, build master : aba6192 - Thu Sep 10 20:58:17 UTC 2015
Docker version 1.8.2, build 0a8c2e3
docker@default:~$ rm -fv ~docker/.docker/*
removed '/home/docker/.docker/ca.pem'
removed '/home/docker/.docker/cert.pem'
removed '/home/docker/.docker/key.pem'
docker@default:~$ sudo -i
Boot2Docker version 1.8.2, build master : aba6192 - Thu Sep 10 20:58:17 UTC 2015
Docker version 1.8.2, build 0a8c2e3
root@default:~# cat /var/lib/boot2docker/profile
EXTRA_ARGS='
--label provider=virtualbox
'
CACERT=/var/lib/boot2docker/ca.pem
DOCKER_HOST='-H tcp://0.0.0.0:2376'
DOCKER_STORAGE=aufs
DOCKER_TLS=auto
SERVERKEY=/var/lib/boot2docker/server-key.pem
SERVERCERT=/var/lib/boot2docker/server.pem
root@default:~# rm -fv /var/lib/boot2docker/*.pem /var/lib/boot2docker/tls/*
removed '/var/lib/boot2docker/ca.pem'
removed '/var/lib/boot2docker/server-key.pem'
removed '/var/lib/boot2docker/server.pem'
removed '/var/lib/boot2docker/tls/cakey.pem'
removed '/var/lib/boot2docker/tls/cert.pem'
removed '/var/lib/boot2docker/tls/hostnames'
removed '/var/lib/boot2docker/tls/key.pem'
root@default:~# /usr/local/etc/init.d/docker restart
Need TLS certs for default,127.0.0.1,10.0.2.15,192.168.99.104
-------------------
Generating CA cert
2015/09/25 20:27:04 Generating a new certificate authority.
rm: can't remove '/var/lib/boot2docker/server.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/server-key.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls/cert.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls/key.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls//hostnames': No such file or directory
Generate server cert
/usr/local/bin/generate_cert --host=default,127.0.0.1,10.0.2.15,192.168.99.104 --ca=/var/lib/boot2docker/ca.pem --ca-key=/var/lib/boot2docker/tls/cakey.pem --cert=/var/lib/boot2docker/server.pem --key=/var/lib/boot2docker/server-key.pem --org=Boot2Docker
2015/09/25 20:27:04 Generating a server cert
Generating client cert
2015/09/25 20:27:06 no --host parameters, making a client cert
root@default:~# exit
docker@default:~$ exit
[~]$ rm -v "${DOCKER_CERT_PATH}"/*.pem
/Users/PavelPolyakov/.docker/machine/certs/ca.pem
/Users/PavelPolyakov/.docker/machine/certs/cert.pem
/Users/PavelPolyakov/.docker/machine/certs/key.pem
[~]$ docker-machine scp default:/home/docker/.docker/\*.pem "${DOCKER_CERT_PATH}"
ca.pem 100% 1050 1.0KB/s 00:00
cert.pem 100% 1070 1.0KB/s 00:00
key.pem 100% 1679 1.6KB/s 00:00
[~]$ docker ps
An error occurred trying to connect: Get https://192.168.99.104:2376/v1.20/containers/json: x509: certificate signed by unknown authority
certs info:
[~]$ openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -key "${DOCKER_CERT_PATH}/key.pem" -cert "${DOCKER_CERT_PATH}/cert.pem" -CAfile "${DOCKER_CERT_PATH}/ca.pem" -tls1 </dev/null
CONNECTED(00000003)
depth=0 O = default
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = default
verify error:num=21:unable to verify the first certificate
verify return:1
140735289209680:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1472:SSL alert number 42
140735289209680:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:656:
---
Certificate chain
0 s:/O=default
i:/O=PavelPolyakov
-----BEGIN CERTIFICATE-----
MIIDADCCAeqgAwIBAgIQR4h88WLsll8AkYK6MIeEMTALBgkqhkiG9w0BAQswGDEW
MBQGA1UEChMNUGF2ZWxQb2x5YWtvdjAeFw0xNTA5MjMxNjMwMDBaFw0xODA5MDcx
NjMwMDBaMBIxEDAOBgNVBAoTB2RlZmF1bHQwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQC7MYqKigy4aJCPiaCQrOKypICUYbIvLYtHKCYG7d9d274MuYWf
dH7mMKTTu8M3GOrAzUj+tvHDAHIZgV0nDMz5qwEkYI2JD7N7hcHQ91JUPJVVpV3e
v2v4hzFra8KB/9h4grjTwAnMLLJNNcWUwu8JPvy3xRmBjpl8UIVY1+VLYwiLodfS
7bWBdqYy2E0sxoJjgtNVPWgxkOloApOrQhq0t/HcyXv0mA3hVPG+9J1r5Mxqs0nY
/qIMKGpywhTGqsWfDnBe22Bf2tFHagDN0frLRrIrF8HT8Xggwawz9pN4RpRWUrI0
lAsgxznXZR1xnvffbxmdy2P94BX1UHqzVy/dAgMBAAGjUDBOMA4GA1UdDwEB/wQE
AwIAqDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIw
ADAPBgNVHREECDAGhwTAqGNoMAsGCSqGSIb3DQEBCwOCAQEAYoGQ5x3wXGVxZey9
dcxyVnBkYSZiInVIeI+bdsKtxKUv7HoXRW1iLyKSeNq8BGQzpfaEPkDtfblcEctz
U0Oj7jS4MlrdzGK1A75Bl8Bo1VDpu146M8cp/Wa09tjmrJktMrVOVZVQteDCPmZo
RIgaHWH0gRJPOy2SCTvLhmyZTK0bO2YKpgvjf2f7NMw6GBJi1hgcZuw3wDjVutjS
2hOWh2HeVyguu1na+YTpZ0csH+MSrKt79ulBpZ3SlYiG5pDFJ5Bjf74OpL0KW30Z
sBA51T1y3Xfyohb27XT28mJLa5Q1cH1PTnComd4K9AlVy7zY0u/MlDN8kaZb0ggX
pqpv2Q==
-----END CERTIFICATE-----
---
Server certificate
subject=/O=default
issuer=/O=PavelPolyakov
---
Acceptable client certificate CA names
/O=PavelPolyakov
Client Certificate Types: RSA sign, ECDSA sign
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1239 bytes and written 1903 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1
Cipher : ECDHE-RSA-AES256-SHA
Session-ID:
Session-ID-ctx:
Master-Key: 33C6C897C3CAFCE41E0BC24728616B2C26DD94D50F25D5D5A730BBE84A72E72355A9C05E41AD60524462FB27FE34FF00
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1443212895
Timeout : 7200 (sec)
Verify return code: 21 (unable to verify the first certificate)
---
Path is correct:
[~]$ echo $DOCKER_CERT_PATH
/Users/PavelPolyakov/.docker/machine/certs
@PavelPolyakov, what do you get when you do the steps from my https://github.com/docker/machine/issues/1880#issuecomment-143338960 (i.e., examining the certificates on the boot2docker side)?
posita
on 25 Sep 2015
@posita
Sorry, I have missed that, thought you were referring to another comment.
Here is hat I get:
docker@default:~$ export DOCKER_CERT_PATH="${HOME}/.docker" DOCKER_TLS_VERIFY=1
DOCKER_HOST=tcp://127.0.0.1:2376
docker@default:~$ docker ps
An error occurred trying to connect: Get https://127.0.0.1:2376/v1.20/containers/json: x509: certificate is valid for 192.168.99.104, not 127.0.0.1
docker@default:~$ openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}"
-CAfile "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key
> .pem" </dev/null
Error opening client certificate private key file /home/docker/.docker/key
.pem
140684102092432:error:02001002:system library:fopen:No such file or directory:bss_file.c:391:fopen('/home/docker/.docker/key
.pem','r')
140684102092432:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:393:
unable to load client certificate private key file
Thanks @posita, is there a way to get additional information on the point where the 'SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)' error occurs in the process when I call docker-compose? I'm happy to open a new ticket to explore what's happening on my machine, but I don't know what additional information I can provide to differentiate what's happening for me vs others.
cischmidt
on 25 Sep 2015
@PavelPolyakov, check your commands and read your error output more carefully. It looks like a copy/paste error.
posita
on 25 Sep 2015
@posita
You are correct again, sorry for me being not enough careful, here is the log:
docker@default:~$ openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}"
-CAfile "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/
null
CONNECTED(00000003)
depth=0 O = default
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = default
verify error:num=21:unable to verify the first certificate
verify return:1
139747887232656:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1472:SSL alert number 42
139747887232656:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
0 s:/O=default
i:/O=PavelPolyakov
-----BEGIN CERTIFICATE-----
MIIDADCCAeqgAwIBAgIQR4h88WLsll8AkYK6MIeEMTALBgkqhkiG9w0BAQswGDEW
MBQGA1UEChMNUGF2ZWxQb2x5YWtvdjAeFw0xNTA5MjMxNjMwMDBaFw0xODA5MDcx
NjMwMDBaMBIxEDAOBgNVBAoTB2RlZmF1bHQwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQC7MYqKigy4aJCPiaCQrOKypICUYbIvLYtHKCYG7d9d274MuYWf
dH7mMKTTu8M3GOrAzUj+tvHDAHIZgV0nDMz5qwEkYI2JD7N7hcHQ91JUPJVVpV3e
v2v4hzFra8KB/9h4grjTwAnMLLJNNcWUwu8JPvy3xRmBjpl8UIVY1+VLYwiLodfS
7bWBdqYy2E0sxoJjgtNVPWgxkOloApOrQhq0t/HcyXv0mA3hVPG+9J1r5Mxqs0nY
/qIMKGpywhTGqsWfDnBe22Bf2tFHagDN0frLRrIrF8HT8Xggwawz9pN4RpRWUrI0
lAsgxznXZR1xnvffbxmdy2P94BX1UHqzVy/dAgMBAAGjUDBOMA4GA1UdDwEB/wQE
AwIAqDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIw
ADAPBgNVHREECDAGhwTAqGNoMAsGCSqGSIb3DQEBCwOCAQEAYoGQ5x3wXGVxZey9
dcxyVnBkYSZiInVIeI+bdsKtxKUv7HoXRW1iLyKSeNq8BGQzpfaEPkDtfblcEctz
U0Oj7jS4MlrdzGK1A75Bl8Bo1VDpu146M8cp/Wa09tjmrJktMrVOVZVQteDCPmZo
RIgaHWH0gRJPOy2SCTvLhmyZTK0bO2YKpgvjf2f7NMw6GBJi1hgcZuw3wDjVutjS
2hOWh2HeVyguu1na+YTpZ0csH+MSrKt79ulBpZ3SlYiG5pDFJ5Bjf74OpL0KW30Z
sBA51T1y3Xfyohb27XT28mJLa5Q1cH1PTnComd4K9AlVy7zY0u/MlDN8kaZb0ggX
pqpv2Q==
-----END CERTIFICATE-----
---
Server certificate
subject=/O=default
issuer=/O=PavelPolyakov
---
Acceptable client certificate CA names
/O=PavelPolyakov
Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256
Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1247 bytes and written 138 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID:
Session-ID-ctx:
Master-Key: 3C14268CC4FE8700076CAFC6E11AD067C1B691F18B7CD339898E785EBF922DBE787EF82557D13CA82AEEEF29363D4F4F
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1443214121
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
---
@cischmidt, for debugging Docker-related SSL stuff, try:
( set -x ; openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -tls1 -CAfile "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null )
This is expected to error out (you haven't provided a -cert argument), but you should see something like this toward the end of the output:
SSL-Session:
Protocol : TLSv1
Cipher : ECDHE-RSA-AES256-SHA
Session-ID:
Session-ID-ctx:
Master-Key: ...
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: ...
Timeout : 7200 (sec)
Verify return code: 0 (ok)
That means your client has verified that the server certificate is good. If you want to check to see if your certs actually work, then simply doing docker ps should do the trick. Alternately, you can do:
( set -x ; openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -tls1 -CAfile "${DOCKER_CERT_PATH}/ca.pem" -cert "${DOCKER_CERT_PATH}/cert.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null )
And you'll see the hex dump from the connection, but I find docker ps is easier on the eyes. :tired_face:
posita
on 25 Sep 2015
@PavelPolyakov, look at the subject and issuer of those certificates. They are _not_ certs generated by boot2docker. I suspect somehow docker-machine is overwriting them. Did you restart your machine instance at any point?
@ehazlett, at what point does docker-machine intervene and (re)create certificates on boot2docker?
posita
on 25 Sep 2015
No I haven't, I see that the subjects are not the ones we are looking for, but...
If I understood you right, then, when we:
- removed the certs being in the docker machine
- restarted docker there
... - we should receive the correct certs generated by boot2docker
But even right after those two commands:
root@default:~# rm -fv /var/lib/boot2docker/*.pem /var/lib/boot2docker/tls/*
removed '/var/lib/boot2docker/ca.pem'
removed '/var/lib/boot2docker/server-key.pem'
removed '/var/lib/boot2docker/server.pem'
removed '/var/lib/boot2docker/tls/cakey.pem'
removed '/var/lib/boot2docker/tls/cert.pem'
removed '/var/lib/boot2docker/tls/hostnames'
removed '/var/lib/boot2docker/tls/key.pem'
root@default:~# /usr/local/etc/init.d/docker restart
Need TLS certs for default,127.0.0.1,10.0.2.15,192.168.99.104
-------------------
Generating CA cert
2015/09/25 20:58:36 Generating a new certificate authority.
rm: can't remove '/var/lib/boot2docker/server.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/server-key.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls/cert.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls/key.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls//hostnames': No such file or directory
Generate server cert
/usr/local/bin/generate_cert --host=default,127.0.0.1,10.0.2.15,192.168.99.104 --ca=/var/lib/boot2docker/ca.pem --ca-key=/var/lib/boot2docker/tls/cakey.pem --cert=/var/lib/boot2docker/server.pem --key=/var/lib/boot2docker/server-key.pem --org=Boot2Docker
2015/09/25 20:58:37 Generating a server cert
Generating client cert
2015/09/25 20:58:39 no --host parameters, making a client cert
The certificates are not correct:
docker@default:~$ openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}"
-CAfile "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/
null
...
Certificate chain
0 s:/O=default
i:/O=PavelPolyakov
...
However, the actual certificates are fresh and new, just generated:
docker@default:~/.docker$ ls -al
total 12
drwx------ 2 docker staff 100 Sep 25 20:58 ./
drwxr-sr-x 5 docker staff 180 Sep 25 18:03 ../
-rw-r--r-- 1 docker staff 1050 Sep 25 20:58 ca.pem
-rw-r--r-- 1 docker staff 1070 Sep 25 20:58 cert.pem
-rw------- 1 docker staff 1675 Sep 25 20:58 key.pem
docker@default:~/.docker$ date
Fri Sep 25 21:02:23 UTC 2015
docker@default:~/.docker$
@PavelPolyakov, you got it. I was _hoping_ we could (by hand) replace what I suspect are the docker-machine-created certificates with the boot2docker-created ones. If that _worked_ (i.e., solved your certificate error), it would have validated my theory about the docker-machine subject/issuer problem.
Unfortunately, it appears that docker-machine won't even let us get that far. :confounded: In other words, my theory could still be good, but we can't verify it this way.
There's one more thing we can try, which is instead of doing /usr/local/etc/init.d/docker restart, we can try:
root@default:~# rm -fv ~docker/.docker/*.pem /var/lib/boot2docker/*.pem /var/lib/boot2docker/tls/*
...
root@default:~# /usr/local/etc/init.d/docker stop
...
root@default:~# ps aux | grep -i docker # make sure all the docker processes are dead; kill them by hand if necessary
...
root@default:~# /usr/local/etc/init.d/docker start
...
root@default:~# export DOCKER_CERT_PATH=/home/docker/.docker DOCKER_TLS_VERIFY=1 DOCKER_HOST=tcp://127.0.0.1:2376
root@default:~# docker ps
...
root@default:~# openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -CAfile "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null
...
Hopefully you have the correct certificates at that point (you had them once before, I don't know why you can't reproduce them all of a sudden). Then don't exit from your root shell. Check the timestamps of your certs:
root@default:~# ls -al ~docker/.docker/*.pem /var/lib/boot2docker/*.pem /var/lib/boot2docker/tls/*.pem
...
From _another_ shell on the host, do:
% eval $( docker-machine env [machine-name] )
% docker ps
...
If it fails, check the timestamps of your certificates again in the boot2docker root shell that you left open. If they don't match, then docker-machine is rewriting them at some point. If they _do_ match, something else is going on with your setup that is not being accounted for, and I give up. :disappointed:
posita
on 25 Sep 2015
The thing is, that each time we generate the certificates, they are good and they are from boot2docker (as I have shown on the screenshot).
However this particular command:
openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -CAfile "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null
shows the certificates with my name in the subject.
Besides, when I do:
root@default:~# docker ps
An error occurred trying to connect: Get https://127.0.0.1:2376/v1.20/containers/json: x509: certificate is valid for 192.168.99.104, not 127.0.0.1
Does it makes sense - this difference in IPs?
Thanks a lot for all of your time you are spending trying to solve my issue! I really appreciate!
Regards,
PavelPolyakov
on 25 Sep 2015
@PavelPolyakov, no problem. You're helping me validate a theory (albeit slowly). :wink:
However this particular command:
openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -CAfile "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/nullshows the certificates with my name in the subject.
Meaning when you run it from the host side (on OS X), or both on the host and from within the VM? Yes, this is what is driving me nuts. I can't figure out why this is happening if all the steps are being executed as spec'ed. In the future, try this instead (it will at least provide some additional information):
( set -x ; openssl x509 -in "${DOCKER_CERT_PATH}/cert.pem" -text ; openssl x509 -in "${DOCKER_CERT_PATH}/ca.pem" -text ; openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -CAfile "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null )
Besides, when I do:
root@default:~# docker ps An error occurred trying to connect: Get https://127.0.0.1:2376/v1.20/containers/json: x509: certificate is valid for 192.168.99.104, not 127.0.0.1Does it makes sense - this difference in IPs?
Ah, yes. Try this (substitute [www.xxx.yyy.zzz] with whatever your VM's IP address is, i.e., 192.168.99.104 as of your last comment, but ifconfig should be available from within boot2docker if needed):
root@default:~# DOCKER_HOST=[www.xxx.yyy.zzz]:2376 docker ps
Any of the dances helps so far :)
Another interestnig things (or not), that I have two locations with certificates:
- /Users/PavelPolyakov/.docker/machine/certs/
- /Users/PavelPolyakov/.docker/machine/machines/default
The other interesting thing, is, that even after the clean installation of the docker machine (without any manipulations, which are described above), when I do the next (on docker machine):
docker@default:~$ export DOCKER_CERT_PATH="${HOME}/.docker" DOCKER_TLS_VERIFY=1 DOCKER_HOST=tcp://192.168.99.1
06:2376
docker@default:~$ docker ps
The server probably has client authentication (--tlsverify) enabled. Please check your TLS client certification settings: Get https://192.168.99.106:2376/v1.20/containers/json: remote error: bad certificate
I still have issues with certificates.
And third, after the certificates are removed and regenerated (while restart).
I have three certificates here /home/docker/.docker.
- ca.pem - Issuer: O=Boot2DockerCA
- cert.pem - Issuer: O=Boot2DockerCA
- key.pem
root@default:/home/docker/.docker# openssl x509 -noout -in key.pem -text
unable to load certificate
140604952614544:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE
Looks something is wrong with it, the same command works for the first two certificates.
PavelPolyakov
on 26 Sep 2015
@PavelPolyakov, the certs in /home/docker/.docker on the VM instance are generated by boot2docker. They're copied into that directory as part of boot2dockers default certificate (re)generation process. If docker-machine then creates _new_ certificates for Docker, my guess is that it's not removing the ones in /home/docker/.docker, which are likely stale, which is why you get the error when calling docker ps on the VM instance.
Also, openssl x509 -noout -in key.pem -text won't work. There's no X509 data in key.pem. It's a private key. openssl x509 -noout -in cert.pem -text or openssl x509 -noout -in key.pem -text should work.
I don't know enough about docker-machine to know why it maintains two separate certificate paths (i.e,. ~/.docker/machine/certs and ~/.docker/machine/machines/[machine-name]). But even so, based on your prior comments, irrespective of what _client_ certificates exist on the host, the VM instance's _server_ certificates are being rewritten by docker-machine, even after one tries to manually install them (in our case, via boot2docker's own certificate generation process). (If we can't get past that, it really doesn't matter what certificates exist on the client side and where.)
My whole goal with this experiment has been to try to get a _server_ certificate on the VM instance where its subject is not the same as it's issuer's subject to see if that was the cause of the error that prompted you to file this issue. Unfortunately, that has proved very difficult.
An easier path at this point might be if someone knowledgeable in Go looks at the docker-machine sources, finds where it does its own certificate generation magic, tweaks it to ensure that the subject in ca.pem is different from the subject in its signed certificates (e.g., ca.pem, server.pem, etc.), and makes a build that you can test.
@ehazlett, recommendations?
posita
on 26 Sep 2015
@PavelPolyakov, I should be sleeping :tired_face:, but try this build1. Save it as docker-machine_darwin-amd64 and make sure it has executable permissions. E.G.:
% curl >~/Desktop/docker-machine_darwin-amd64 --location 'https://www.dropbox.com/s/2leyxf2sy6k0i9o/docker-machine_darwin-amd64'
...
% openssl dgst -r -sha256 ~/Desktop/docker-machine_darwin-amd64
cf3f82323b5f5f3556b0286c2ea3edb51ffe1bccd5fbeecddd59f591486f2089 */.../Desktop/docker-machine_darwin-amd64
% chmod +x ~/Desktop/docker-machine_darwin-amd64
% ~/Desktop/docker-machine_darwin-amd64 create --driver virtualbox fubar
...
% eval $( ~/Desktop/docker-machine_darwin-amd64 env fubar )
% docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1 SHA-256 checksum: cf3f82323b5f5f3556b0286c2ea3edb51ffe1bccd5fbeecddd59f591486f2089.
posita
on 26 Sep 2015
@posita
Please, go sleep :)
Installed the machine:
[~]$ curl >~/Desktop/docker-machine_darwin-amd64 --location 'https://www.dropbox.com/s/2leyxf2sy6k0i9o/docker-machine_darwin-amd64'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 486 0 486 0 0 269 0 --:--:-- 0:00:01 --:--:-- 269
100 12.7M 100 12.7M 0 0 1153k 0 0:00:11 0:00:11 --:--:-- 2555k
[~]$ openssl dgst -r -sha256 ~/Desktop/docker-machine_darwin-amd64
cf3f82323b5f5f3556b0286c2ea3edb51ffe1bccd5fbeecddd59f591486f2089 */Users/PavelPolyakov/Desktop/docker-machine_darwin-amd64
[~]$ chmod +x ~/Desktop/docker-machine_darwin-amd64
[~]$ ~/Desktop/docker-machine_darwin-amd64 create --driver virtualbox fubar
Running pre-create checks...
Creating machine...
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
Waiting for machine to be running, this may take a few minutes...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Provisioning created instance...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
To see how to connect Docker to this machine, run: /Users/PavelPolyakov/Desktop/docker-machine_darwin-amd64 env fubar
[~]$ eval $( ~/Desktop/docker-machine_darwin-amd64 env fubar )
[~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Being on the machine I have the next:
docker@fubar:~$ export DOCKER_CERT_PATH="${HOME}/.docker" DOCKER_TLS_VERIFY=1 DOCKER_HOST=tcp://127.0.0.1
docker@fubar:~$ ( set -x ; openssl x509 -in "${DOCKER_CERT_PATH}/cert.pem" -text ; openssl x509 -in "${DOCKER_
CERT_PATH}/ca.pem" -text ; openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -CAfile "${DOCKER_C
ERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null )
+ openssl x509 -in /home/docker/.docker/cert.pem -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
5a:59:69:9a:b9:cb:c7:fa:b2:23:00:23:22:3a:fc:c2
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=Boot2DockerCA
Validity
Not Before: Sep 26 09:57:59 2015 GMT
Not After : Sep 25 09:57:59 2016 GMT
Subject: O=Boot2Docker
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:ad:e8:2f:36:48:54:2b:a3:19:96:e3:8a:d5:b4:
53:dc:e5:40:43:45:5a:a2:80:89:d5:49:5a:3d:cd:
42:f5:8d:f4:61:19:41:ec:22:af:24:59:da:07:be:
46:f7:79:1d:b9:b7:c4:09:02:5e:7b:f7:6b:eb:bd:
0c:0e:db:d9:24:da:11:7a:bf:96:2c:ac:3c:05:d1:
35:48:6e:86:ad:d0:25:34:3e:1d:30:34:d3:51:df:
be:b9:b6:c4:89:19:96:63:e4:10:ae:59:90:e7:a9:
3c:f3:9e:9a:a0:79:3a:9d:d9:46:b1:d5:a2:58:b2:
ae:84:4b:78:70:f6:54:a0:9d:67:7d:61:91:fb:1b:
9b:a7:f7:6a:ca:b6:f9:f2:23:05:d4:1c:e5:8f:1d:
ff:00:88:2f:b5:ee:a4:68:5c:37:11:7c:6b:c6:0d:
24:70:e2:4e:1b:62:4f:e9:5d:80:d9:02:0c:d7:29:
f1:cb:ae:91:8d:81:4a:0b:6c:f5:c2:be:52:ab:13:
da:01:d2:a2:cc:05:17:76:97:b5:29:c2:c9:6e:34:
c6:47:05:cc:10:29:3e:a5:af:ec:f9:e6:56:4e:03:
10:96:90:42:2f:17:5b:4a:97:11:aa:73:2e:21:f4:
cf:55:3c:93:2a:70:5a:ce:77:55:19:8e:fd:6e:51:
1c:3d
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature
X509v3 Extended Key Usage:
TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
Signature Algorithm: sha256WithRSAEncryption
4a:53:4d:0a:ff:bd:fc:b7:c3:c6:50:55:48:85:9c:1a:4d:4f:
07:9c:86:bb:d4:58:a4:34:3b:7a:fc:18:16:c6:9e:21:2c:41:
8b:f9:8e:ab:3a:de:09:a0:b7:a5:0e:46:25:4e:5e:c4:47:13:
13:6f:0c:94:1a:9b:bb:86:91:e2:65:f8:c0:08:e4:44:6f:ac:
c2:42:21:54:94:7e:1c:7a:e5:70:e1:f2:00:ac:57:ae:b4:ed:
d9:d2:c4:ce:0d:07:7d:5d:83:0e:17:ef:98:48:ed:29:0d:6b:
d6:9e:c5:14:dd:03:f3:91:eb:4d:b9:07:fe:4a:7e:72:80:f4:
06:b6:36:0e:d9:c7:31:a3:f8:c3:c7:d2:1e:93:ea:ae:93:f1:
75:ac:0d:13:2f:57:b1:3a:58:26:7f:bd:fb:b6:71:b3:60:c9:
7d:98:68:46:81:6b:8f:a8:45:67:06:d1:ec:8d:90:be:36:bb:
0b:94:56:70:b1:45:c0:82:05:6c:b1:9f:cb:18:6d:a0:e4:07:
53:c4:14:19:02:2e:04:5e:27:b4:51:3b:ae:ef:2c:56:31:40:
b1:b1:3c:7f:2e:73:96:0a:13:53:51:ae:5b:a3:5a:91:7c:a9:
ee:3f:1e:c8:03:e8:30:59:07:83:4c:69:51:79:13:c2:d0:31:
d0:04:9e:83
-----BEGIN CERTIFICATE-----
MIIC6TCCAdOgAwIBAgIQWllpmrnLx/qyIwAjIjr8wjALBgkqhkiG9w0BAQswGDEW
MBQGA1UEChMNQm9vdDJEb2NrZXJDQTAeFw0xNTA5MjYwOTU3NTlaFw0xNjA5MjUw
OTU3NTlaMBYxFDASBgNVBAoTC0Jvb3QyRG9ja2VyMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAregvNkhUK6MZluOK1bRT3OVAQ0VaooCJ1UlaPc1C9Y30
YRlB7CKvJFnaB75G93kdubfECQJee/dr670MDtvZJNoRer+WLKw8BdE1SG6GrdAl
ND4dMDTTUd++ubbEiRmWY+QQrlmQ56k8856aoHk6ndlGsdWiWLKuhEt4cPZUoJ1n
fWGR+xubp/dqyrb58iMF1Bzljx3/AIgvte6kaFw3EXxrxg0kcOJOG2JP6V2A2QIM
1ynxy66RjYFKC2z1wr5SqxPaAdKizAUXdpe1KcLJbjTGRwXMECk+pa/s+eZWTgMQ
lpBCLxdbSpcRqnMuIfTPVTyTKnBazndVGY79blEcPQIDAQABozUwMzAOBgNVHQ8B
Af8EBAMCAIAwEwYDVR0lBAwwCgYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADALBgkq
hkiG9w0BAQsDggEBAEpTTQr/vfy3w8ZQVUiFnBpNTwechrvUWKQ0O3r8GBbGniEs
QYv5jqs63gmgt6UORiVOXsRHExNvDJQam7uGkeJl+MAI5ERvrMJCIVSUfhx65XDh
8gCsV6607dnSxM4NB31dgw4X75hI7SkNa9aexRTdA/OR6025B/5KfnKA9Aa2Ng7Z
xzGj+MPH0h6T6q6T8XWsDRMvV7E6WCZ/vfu2cbNgyX2YaEaBa4+oRWcG0eyNkL42
uwuUVnCxRcCCBWyxn8sYbaDkB1PEFBkCLgReJ7RRO67vLFYxQLGxPH8uc5YKE1NR
rlujWpF8qe4/HsgD6DBZB4NMaVF5E8LQMdAEnoM=
-----END CERTIFICATE-----
+ openssl x509 -in /home/docker/.docker/ca.pem -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
9b:4f:8c:0f:dc:b5:1d:ad:ec:54:1b:f8:73:ee:67:7a
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=PavelPolyakov
Validity
Not Before: Sep 26 07:47:00 2015 GMT
Not After : Sep 10 07:47:00 2018 GMT
Subject: O=PavelPolyakov
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:e3:3b:a8:23:92:6d:63:59:27:b7:39:e1:7d:4e:
93:57:e2:55:67:51:81:ed:a5:f8:5b:62:e9:e9:e8:
5f:31:63:10:8a:23:c8:1d:97:54:34:02:62:9a:2a:
b9:a1:b8:bc:94:8e:4c:55:f3:8e:00:71:d0:63:1f:
0d:7a:ab:30:a4:62:b7:f6:b2:f8:73:ad:6b:5d:3b:
60:35:a5:14:1d:a1:58:c9:e0:01:e6:03:e5:59:59:
f3:39:c9:4f:21:c8:86:ef:09:e8:ac:25:4b:a6:0f:
f1:d4:85:29:fe:d2:df:cf:95:94:f7:0f:28:00:1f:
c7:f8:29:37:5b:84:e1:0e:c4:88:61:c5:07:60:c8:
1a:a6:97:dd:5c:00:a3:af:35:21:45:8d:02:8e:9d:
6a:c8:81:72:9d:0b:52:4a:99:66:e4:04:96:f3:f7:
29:8f:b5:ed:e3:1a:2f:0c:36:24:42:69:ca:99:f6:
63:4c:d4:19:70:c0:3e:0c:9d:e2:25:be:79:19:ee:
3d:ed:27:49:f9:8f:98:8f:c1:67:74:ea:71:ea:dd:
21:e1:21:f3:a7:a8:35:81:5c:c4:96:2c:be:04:4f:
40:a3:23:e5:d1:1c:7e:f8:8b:c0:0f:81:90:4d:49:
e9:54:4c:13:2f:59:03:c1:12:85:fc:24:68:90:8f:
21:45
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Key Agreement, Certificate Sign
X509v3 Basic Constraints: critical
CA:TRUE
Signature Algorithm: sha256WithRSAEncryption
b7:31:60:c6:92:93:59:09:02:dd:2f:a7:de:58:40:a3:c7:32:
dc:ba:94:99:b6:9d:df:b8:8b:f1:a0:0c:d8:dd:e8:d3:83:91:
f8:e8:38:f0:02:8a:9f:e7:73:ec:c4:7b:9b:99:10:63:2d:60:
17:cd:50:dd:00:ce:da:fa:33:2d:fd:f2:40:d7:44:2b:89:a3:
b4:d3:a6:ca:49:3a:7f:1d:50:67:9b:94:e1:1c:29:33:53:04:
f9:cc:1b:76:d3:60:33:a4:c2:e4:d0:aa:c5:e4:e9:d0:64:f4:
39:58:00:fa:ca:1e:83:c8:9c:07:78:f6:a1:50:0b:1c:3e:e1:
29:8a:09:ca:4b:bb:94:5b:5e:1a:f1:13:52:15:43:fe:67:08:
cb:a5:0f:5c:50:5e:aa:10:1f:3f:ca:41:48:79:b2:f3:d4:f7:
95:5b:ee:22:f9:22:2c:9f:c1:5a:2e:82:9b:8d:a2:32:f7:b2:
df:5c:dd:4f:b8:4f:7e:f1:c7:de:33:73:db:8c:36:c0:40:b7:
71:1e:5b:7f:66:e9:b1:67:a4:bb:9f:95:86:53:05:de:dd:89:
4f:9e:8d:3b:4e:f0:25:ee:8f:f3:2a:c2:7a:0d:7a:e1:3b:9c:
72:aa:36:22:a4:c7:27:b7:62:7f:b2:d3:a0:bf:a9:e9:db:12:
ab:35:78:3a
-----BEGIN CERTIFICATE-----
MIIC2jCCAcSgAwIBAgIRAJtPjA/ctR2t7FQb+HPuZ3owCwYJKoZIhvcNAQELMBgx
FjAUBgNVBAoTDVBhdmVsUG9seWFrb3YwHhcNMTUwOTI2MDc0NzAwWhcNMTgwOTEw
MDc0NzAwWjAYMRYwFAYDVQQKEw1QYXZlbFBvbHlha292MIIBIjANBgkqhkiG9w0B
AQEFAAOCAQ8AMIIBCgKCAQEA4zuoI5JtY1kntznhfU6TV+JVZ1GB7aX4W2Lp6ehf
MWMQiiPIHZdUNAJimiq5obi8lI5MVfOOAHHQYx8NeqswpGK39rL4c61rXTtgNaUU
HaFYyeAB5gPlWVnzOclPIciG7wnorCVLpg/x1IUp/tLfz5WU9w8oAB/H+Ck3W4Th
DsSIYcUHYMgappfdXACjrzUhRY0Cjp1qyIFynQtSSplm5ASW8/cpj7Xt4xovDDYk
QmnKmfZjTNQZcMA+DJ3iJb55Ge497SdJ+Y+Yj8FndOpx6t0h4SHzp6g1gVzEliy+
BE9AoyPl0Rx++IvAD4GQTUnpVEwTL1kDwRKF/CRokI8hRQIDAQABoyMwITAOBgNV
HQ8BAf8EBAMCAKwwDwYDVR0TAQH/BAUwAwEB/zALBgkqhkiG9w0BAQsDggEBALcx
YMaSk1kJAt0vp95YQKPHMty6lJm2nd+4i/GgDNjd6NODkfjoOPACip/nc+zEe5uZ
EGMtYBfNUN0Aztr6My398kDXRCuJo7TTpspJOn8dUGeblOEcKTNTBPnMG3bTYDOk
wuTQqsXk6dBk9DlYAPrKHoPInAd49qFQCxw+4SmKCcpLu5RbXhrxE1IVQ/5nCMul
D1xQXqoQHz/KQUh5svPU95Vb7iL5IiyfwVougpuNojL3st9c3U+4T37xx94zc9uM
NsBAt3EeW39m6bFnpLuflYZTBd7diU+ejTtO8CXuj/MqwnoNeuE7nHKqNiKkxye3
Yn+y06C/qenbEqs1eDo=
-----END CERTIFICATE-----
+ openssl s_client -showcerts -connect 127.0.0.1 -CAfile /home/docker/.docker/ca.pem -key /home/docker/.docker/key.pem
no port defined
usage: s_client args
-host host - use -connect instead
-port port - use -connect instead
-connect host:port - who to connect to (default is localhost:4433)
-verify_host host - check peer certificate matches "host"
-verify_email email - check peer certificate matches "email"
-verify_ip ipaddr - check peer certificate matches "ipaddr"
-verify arg - turn on peer certificate verification
-verify_return_error - return verification errors
-cert arg - certificate file to use, PEM format assumed
-certform arg - certificate format (PEM or DER) PEM default
-key arg - Private key file to use, in cert file if
not specified but cert file is.
-keyform arg - key format (PEM or DER) PEM default
-pass arg - private key file pass phrase source
-CApath arg - PEM format directory of CA's
-CAfile arg - PEM format file of CA's
-no_alt_chains - only ever use the first certificate chain found
-reconnect - Drop and re-make the connection with the same Session-ID
-pause - sleep(1) after each read(2) and write(2) system call
-prexit - print session information even on connection failure
-showcerts - show all certificates in the chain
-debug - extra output
-msg - Show protocol messages
-nbio_test - more ssl protocol testing
-state - print the 'ssl' states
-nbio - Run with non-blocking IO
-crlf - convert LF from terminal into CRLF
-quiet - no s_client output
-ign_eof - ignore input eof (default when -quiet)
-no_ign_eof - don't ignore input eof
-psk_identity arg - PSK identity
-psk arg - PSK in hex (without 0x)
-srpuser user - SRP authentification for 'user'
-srppass arg - password for 'user'
-srp_lateuser - SRP username into second ClientHello message
-srp_moregroups - Tolerate other than the known g N values.
-srp_strength int - minimal length in bits for N (default 1024).
-ssl2 - just use SSLv2
-ssl3 - just use SSLv3
-tls1_2 - just use TLSv1.2
-tls1_1 - just use TLSv1.1
-tls1 - just use TLSv1
-dtls1 - just use DTLSv1
-fallback_scsv - send TLS_FALLBACK_SCSV
-mtu - set the link layer MTU
-no_tls1_2/-no_tls1_1/-no_tls1/-no_ssl3/-no_ssl2 - turn off that protocol
-bugs - Switch on all SSL implementation bug workarounds
-serverpref - Use server's cipher preferences (only SSLv2)
-cipher - preferred cipher to use, use the 'openssl ciphers'
command to see what is available
-starttls prot - use the STARTTLS command before starting TLS
for those protocols that support it, where
'prot' defines which one to assume. Currently,
only "smtp", "pop3", "imap", "ftp" and "xmpp"
are supported.
-engine id - Initialise and use the specified engine
-rand file:file:...
-sess_out arg - file to write SSL session to
-sess_in arg - file to read SSL session from
-servername host - Set TLS extension servername in ClientHello
-tlsextdebug - hex dump of all TLS extensions received
-status - request certificate status from server
-no_ticket - disable use of RFC4507bis session tickets
-serverinfo types - send empty ClientHello extensions (comma-separated numbers)
-nextprotoneg arg - enable NPN extension, considering named protocols supported (comma-separated list)
-alpn arg - enable ALPN extension, considering named protocols supported (comma-separated list)
-legacy_renegotiation - enable use of legacy renegotiation (dangerous)
-use_srtp profiles - Offer SRTP key management with a colon-separated profile list
-keymatexport label - Export keying material using label
-keymatexportlen len - Export len bytes of keying material (default 20)
Maybe there is something I am missing? However I am trying to execute everything as you describe...
PavelPolyakov
on 26 Sep 2015
Maybe there is something I am missing? However I am trying to execute everything as you describe...
Sorry, I didn't mean to be unclear. There was no need to go through my list of steps again. Your docker ps appears to have worked without the SSL error, correct? Try docker-compose ps. If I'm not mistaken, my patched version of docker-machine just solved your original problem. Can you verify?
Also, can you verify your docker-machine version again? (The one you were using before the patched version.)
Finally, did you ever try to reproduce your error by starting with a new machine generated with docker-machine 0.4.0 or 0.4.1?
posita
on 26 Sep 2015
@PavelPolyakov, I'm sorry for having missed this before, but as of your https://github.com/docker/machine/issues/1880#issuecomment-143314311, if my theory is correct, you should have been able to proceed without experiencing any SSL: CERTIFICATE_VERIFY_FAILED errors. Your subject/issuer for your docker-machine-generated certificate went from:
0 s:/O=PavelPolyakov
i:/O=PavelPolyakov
To:
0 s:/O=default
i:/O=PavelPolyakov
I don't know when or why that happened, but had you done the following on your host at the time, everything should have worked:
% eval $( docker-machine env default )
% docker ps
...
% docker-compose ps
...
This may have been why @rkit's https://github.com/docker/machine/issues/1880#issuecomment-142557209 worked for him.
I'm concerned that perhaps you had artifacts from a docker machine environment or instance that was frustrating your efforts? When I try to create a new instance using docker-machine 0.4.1, my certificates match the s:/O=[machine name] and i:/O=[user name] pattern, not the s:/O=[user name] and i:/O=[user name] pattern that was probably giving you problems.
Again, I'm sorry for not catching this before, but this probably should have fixed this for you. Can you confirm?
posita
on 26 Sep 2015
Well,
No :( I have tried it before.
If I do:
[~/tmp/microservices-workshop/msworkshop/step4]$ eval $( ~/Desktop/docker-machine_darwin-amd64 env fubar )
[~/tmp/microservices-workshop/msworkshop/step4]$ docker ps *[master]
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose build *[master]
Building frontend...
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
[~/tmp/microservices-workshop/msworkshop/step4]$ ~/Desktop/docker-machine_darwin-amd64 --version *[master]
docker-machine_darwin-amd64 version 0.5.0-dev (c4cd238)
I have docker ps working, but I had it working before as well (when using default flow of docker toolbox from the website).
PavelPolyakov
on 26 Sep 2015
Argh. :confounded: Okay, I'm running out of ideas then. Try saving the following as compose-debug.py:
import logging
import re
import sys
from compose.cli.main import main
if __name__ == '__main__':
logging.basicConfig(format='%(levelname)-8s: %(message)s')
logging.getLogger().setLevel(logging.DEBUG)
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(main())
Then do:
python compose-debug.py --verbose build
Here we are:
[~/tmp/microservices-workshop/msworkshop/step4]$ python compose-debug.py --verbose build *[master]
Traceback (most recent call last):
File "compose-debug.py", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/compose/cli/main.py", line 39, in main
command.sys_dispatch()
File "/usr/local/lib/python2.7/site-packages/compose/cli/docopt_command.py", line 21, in sys_dispatch
self.dispatch(sys.argv[1:], None)
File "/usr/local/lib/python2.7/site-packages/compose/cli/command.py", line 27, in dispatch
super(Command, self).dispatch(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/compose/cli/docopt_command.py", line 24, in dispatch
self.perform_command(*self.parse(argv, global_options))
File "/usr/local/lib/python2.7/site-packages/compose/cli/command.py", line 57, in perform_command
verbose=options.get('--verbose'))
File "/usr/local/lib/python2.7/site-packages/compose/cli/command.py", line 79, in get_project
self.get_client(verbose=verbose))
File "/usr/local/lib/python2.7/site-packages/compose/cli/command.py", line 64, in get_client
version_info = six.iteritems(client.version())
File "/usr/local/lib/python2.7/site-packages/docker/client.py", line 837, in version
return self._result(self._get(url), json=True)
File "/usr/local/lib/python2.7/site-packages/docker/clientbase.py", line 86, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 477, in get
return self.request('GET', url, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 344, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 314, in _raise_timeout
if 'timed out' in str(err) or 'did not complete (read)' in str(err): # Python 2.6
TypeError: __str__ returned non-string (type Error)
[~/tmp/microservices-workshop/msworkshop/step4]$ python --version *[master]
Python 2.7.10
@PavelPolyakov, my apologies if it wasn't obvious, but did you make sure your docker-machine instance was running before you tried the above command?
Also, when posting debug output with stack traces, it is helpful if you can wrap them in ``` tags to preserve formatting.
posita
on 26 Sep 2015
yes, the machine is up, I can do (if this is the evidence).
docker-machine ssh default
Here is the other variant (when I skip --verbose the output is a little different):
[~/tmp/microservices-workshop/msworkshop/step4]$ python compose-debug.py build *[master]
INFO : Building frontend...
Building frontend...
Traceback (most recent call last):
File "compose-debug.py", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/compose/cli/main.py", line 39, in main
command.sys_dispatch()
File "/usr/local/lib/python2.7/site-packages/compose/cli/docopt_command.py", line 21, in sys_dispatch
self.dispatch(sys.argv[1:], None)
File "/usr/local/lib/python2.7/site-packages/compose/cli/command.py", line 27, in dispatch
super(Command, self).dispatch(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/compose/cli/docopt_command.py", line 24, in dispatch
self.perform_command(*self.parse(argv, global_options))
File "/usr/local/lib/python2.7/site-packages/compose/cli/command.py", line 59, in perform_command
handler(project, command_options)
File "/usr/local/lib/python2.7/site-packages/compose/cli/main.py", line 135, in build
project.build(service_names=options['SERVICE'], no_cache=no_cache)
File "/usr/local/lib/python2.7/site-packages/compose/project.py", line 233, in build
service.build(no_cache)
File "/usr/local/lib/python2.7/site-packages/compose/service.py", line 710, in build
dockerfile=self.options.get('dockerfile', None),
File "/usr/local/lib/python2.7/site-packages/docker/client.py", line 158, in build
timeout=timeout,
File "/usr/local/lib/python2.7/site-packages/docker/clientbase.py", line 83, in _post
return self.post(url, **self._set_request_timeout(kwargs))
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 508, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 344, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 314, in _raise_timeout
if 'timed out' in str(err) or 'did not complete (read)' in str(err): # Python 2.6
TypeError: __str__ returned non-string (type Error)
Of course I use the formatting, last time it was a typo ``` - and I started the same line, sorry.
@posita
I know you are tired of trying to solve this, so do I :) I don't know why my current setup is cursed. Maybe indeed it's something about my particular OS installation and the way other software is installed here (brew). It's a pity - spending so much of your time trying to solve the issue, which only appears on my machine.
My further long term plan - try to install new OSX, when it's out, or install even install the OSX from the scratch and hope that default flow would work.
PavelPolyakov
on 26 Sep 2015
The stack traces look weird to me (they don't look like SSL errors). What do you get for docker-compose --verbose build?
posita
on 26 Sep 2015
[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose build *[master]
Building frontend...
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose --verbose build *[master]
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
What do you get with docker-compose --verbose ps and python compose-debug.py --verbose ps? This is what I get:
% docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
% docker-compose --verbose ps
Compose version 1.4.1
Docker base_url: https://www.xxx.yyy.zzz:2376
Docker version: KernelVersion=4.0.9-boot2docker, Os=linux, BuildTime=Thu Sep 10 19:10:10 UTC 2015, ApiVersion=1.20, Version=1.8.2, GitCommit=0a8c2e3, Arch=amd64, GoVersion=go1.4.2
docker containers <- (all=True, filters={u'label': [u'com.docker.compose.project=...', u'com.docker.compose.oneoff=False']})
docker containers -> (list with 0 items)
docker containers <- (all=True)
docker containers -> (list with 0 items)
docker containers <- (all=False, filters={u'label': [u'com.docker.compose.project=...', u'com.docker.compose.oneoff=True']})
docker containers -> (list with 0 items)
docker containers <- (all=True)
docker containers -> (list with 0 items)
Name Command State Ports
------------------------------
% python compose-debug.py --verbose ps
DEBUG : Trying /.../.docker/config.json
DEBUG : File doesn't exist
DEBUG : Trying /.../.dockercfg
DEBUG : Attempting to parse as JSON
DEBUG : ...
INFO : Compose version 1.4.1
Compose version 1.4.1
INFO : Docker base_url: https://www.xxx.yyy.zzz:2376
Docker base_url: https://www.xxx.yyy.zzz:2376
INFO : Docker version: KernelVersion=4.0.9-boot2docker, Os=linux, BuildTime=Thu Sep 10 19:10:10 UTC 2015, ApiVersion=1.20, Version=1.8.2, GitCommit=0a8c2e3, Arch=amd64, GoVersion=go1.4.2
Docker version: KernelVersion=4.0.9-boot2docker, Os=linux, BuildTime=Thu Sep 10 19:10:10 UTC 2015, ApiVersion=1.20, Version=1.8.2, GitCommit=0a8c2e3, Arch=amd64, GoVersion=go1.4.2
INFO : docker containers <- (all=True, filters={u'label': [u'com.docker.compose.project=...', u'com.docker.compose.oneoff=False']})
docker containers <- (all=True, filters={u'label': [u'com.docker.compose.project=...', u'com.docker.compose.oneoff=False']})
INFO : docker containers -> (list with 0 items)
docker containers -> (list with 0 items)
INFO : docker containers <- (all=True)
docker containers <- (all=True)
INFO : docker containers -> (list with 0 items)
docker containers -> (list with 0 items)
INFO : docker containers <- (all=False, filters={u'label': [u'com.docker.compose.project=...', u'com.docker.compose.oneoff=True']})
docker containers <- (all=False, filters={u'label': [u'com.docker.compose.project=...', u'com.docker.compose.oneoff=True']})
INFO : docker containers -> (list with 0 items)
docker containers -> (list with 0 items)
INFO : docker containers <- (all=True)
docker containers <- (all=True)
INFO : docker containers -> (list with 0 items)
docker containers -> (list with 0 items)
Name Command State Ports
------------------------------
@posita
That's what I have:
[~/tmp/microservices-workshop/msworkshop/step4]$ docker ps *[master]
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[~/tmp/microservices-workshop/msworkshop/step4]$ docker --verbose ps *[master]
flag provided but not defined: --verbose
See 'docker --help'.
[~/tmp/microservices-workshop/msworkshop/step4]$ python compose-debug.py --verbose ps *[master]
Traceback (most recent call last):
File "compose-debug.py", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/compose/cli/main.py", line 39, in main
command.sys_dispatch()
File "/usr/local/lib/python2.7/site-packages/compose/cli/docopt_command.py", line 21, in sys_dispatch
self.dispatch(sys.argv[1:], None)
File "/usr/local/lib/python2.7/site-packages/compose/cli/command.py", line 27, in dispatch
super(Command, self).dispatch(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/compose/cli/docopt_command.py", line 24, in dispatch
self.perform_command(*self.parse(argv, global_options))
File "/usr/local/lib/python2.7/site-packages/compose/cli/command.py", line 57, in perform_command
verbose=options.get('--verbose'))
File "/usr/local/lib/python2.7/site-packages/compose/cli/command.py", line 79, in get_project
self.get_client(verbose=verbose))
File "/usr/local/lib/python2.7/site-packages/compose/cli/command.py", line 64, in get_client
version_info = six.iteritems(client.version())
File "/usr/local/lib/python2.7/site-packages/docker/client.py", line 837, in version
return self._result(self._get(url), json=True)
File "/usr/local/lib/python2.7/site-packages/docker/clientbase.py", line 86, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 477, in get
return self.request('GET', url, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 344, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 314, in _raise_timeout
if 'timed out' in str(err) or 'did not complete (read)' in str(err): # Python 2.6
TypeError: __str__ returned non-string (type Error)
[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose --verbose ps *[master]
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
[~/tmp/microservices-workshop/msworkshop/step4]$ python -V *[master]
Python 2.7.10
Do you think it's an issue of my python?
Which version do you have?
I also found this issue:
https://github.com/kennethreitz/requests/issues/2524
Regarding the https and ulr library which python uses, I don't know if docker-compose uses the same lib though.
PavelPolyakov
on 27 Sep 2015
@PavelPolyakov, try docker-compose --verbose ps, not docker --verbose ps.
I doubt it's a Python version thing. I'm running 2.7.10 as well. It could be a docker-py thing? Try python -c 'import docker ; print(docker.version)'.
What version of OpenSSL do you have?
which openssl
openssl version
which python
python -c 'import ssl ; print(ssl.__file__) ; print(ssl.OPENSSL_VERSION)'
Here we are:
[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose --verbose ps *[master]
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
[~/tmp/microservices-workshop/msworkshop/step4]$ python -c 'import docker ; print(docker.version)' *[master]
1.3.1
[~/tmp/microservices-workshop/msworkshop/step4]$ which openssl *[master]
/usr/local/bin/openssl
[~/tmp/microservices-workshop/msworkshop/step4]$ openssl version *[master]
OpenSSL 1.0.2d 9 Jul 2015
[~/tmp/microservices-workshop/msworkshop/step4]$ which python *[master]
/usr/local/bin/python
[~/tmp/microservices-workshop/msworkshop/step4]$ python -c 'import ssl ; print(ssl.__file__) ; print(ssl.OPENSSL_VERSION)'
/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.pyc
OpenSSL 1.0.2d 9 Jul 2015
How is docker-py getting installed on your machine? How did you upgrade (if you upgraded) docker-compose to the latest version?
posita
on 27 Sep 2015
Currently all the installation was done from here: https://www.docker.com/toolbox .
I haven't updated the docker-compose.
I have tried different installation as well - from brew (same results, SSL error), but the one we are experimenting with is from toolbox pkg.
PavelPolyakov
on 27 Sep 2015
Okay, your versions aren't that different from mine, except for docker-py (I don't know about requests):
% docker version --format='{{.Client.Version}}'
1.8.2
% docker-machine --version
docker-machine version 0.4.1 (e2c88d6)
% docker-compose --version
docker-compose version: 1.4.1
% openssl version
OpenSSL 1.0.2d 9 Jul 2015
% python -c 'import ssl ; print(ssl.OPENSSL_VERSION)' # just to make sure it's picking up the same version
OpenSSL 1.0.2d 9 Jul 2015
% python -c 'import docker ; print(docker.version)'
1.4.0
% python -c 'import requests.packages.urllib3 ; print(requests.__version__) ; print(requests.packages.urllib3.__version__)'
2.6.2
1.10.3
Have you tried installing/running docker-compose from a virtual environment (that really should have been our first step :disappointed:)?
% virtualenv .venv
...
% . ./.venv/bin/activate # don't forget the dot; you're "sourcing" the file
% ./.venv/bin/pip install docker-compose
...
% rehash # necessary on some shells like zsh
% which docker-compose
.../.venv/bin/docker-compose
% docker-compose --version
docker-compose version: 1.4.2
% python -c 'import docker ; print(docker.version)'
1.4.0
% python -c 'import requests.packages.urllib3 ; print(requests.__version__) ; print(requests.packages.urllib3.__version__)'
2.7.0
1.10.4
% docker-compose --verbose ps
...?
% python compose-debug.py --verbose ps
...?
% docker-compose build
...?
The same:
[~/tmp/microservices-workshop/msworkshop/step4]$ virtualenv .venv *[master]
New python executable in .venv/bin/python
Installing setuptools, pip, wheel...done.
[~/tmp/microservices-workshop/msworkshop/step4]$ . ./.venv/bin/activate *[master]
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ ./.venv/bin/pip install docker-compose *[master]
Collecting docker-compose
Downloading docker-compose-1.4.2.tar.gz (82kB)
100% |████████████████████████████████| 86kB 2.2MB/s
Collecting docopt<0.7,>=0.6.1 (from docker-compose)
Downloading docopt-0.6.2.tar.gz
Collecting PyYAML<4,>=3.10 (from docker-compose)
Downloading PyYAML-3.11.tar.gz (248kB)
100% |████████████████████████████████| 249kB 1.4MB/s
Collecting requests<2.7,>=2.6.1 (from docker-compose)
Downloading requests-2.6.2-py2.py3-none-any.whl (470kB)
100% |████████████████████████████████| 471kB 839kB/s
Collecting texttable<0.9,>=0.8.1 (from docker-compose)
Downloading texttable-0.8.3.tar.gz
Collecting websocket-client<1.0,>=0.32.0 (from docker-compose)
Downloading websocket_client-0.32.0.tar.gz (192kB)
100% |████████████████████████████████| 192kB 1.0MB/s
Collecting docker-py<1.4,>=1.3.1 (from docker-compose)
Downloading docker-py-1.3.1.tar.gz (49kB)
100% |████████████████████████████████| 53kB 3.7MB/s
Collecting dockerpty<0.4,>=0.3.4 (from docker-compose)
Downloading dockerpty-0.3.4.tar.gz
Collecting six<2,>=1.3.0 (from docker-compose)
Downloading six-1.9.0-py2.py3-none-any.whl
Building wheels for collected packages: docker-compose, docopt, PyYAML, texttable, websocket-client, docker-py, dockerpty
Running setup.py bdist_wheel for docker-compose
Stored in directory: /Users/PavelPolyakov/Library/Caches/pip/wheels/85/15/68/da301b79c711f2eb9a2065f44f16dcef736862e9c334bf87c4
Running setup.py bdist_wheel for docopt
Stored in directory: /Users/PavelPolyakov/Library/Caches/pip/wheels/0d/5c/a7/cb986749520c1950217b5d8405def5c18541322dbc411a80d1
Running setup.py bdist_wheel for PyYAML
Stored in directory: /Users/PavelPolyakov/Library/Caches/pip/wheels/fa/db/f6/dee55793d344f1706dc4a5a693298f0115241d1085cc212364
Running setup.py bdist_wheel for texttable
Stored in directory: /Users/PavelPolyakov/Library/Caches/pip/wheels/d6/44/0b/20a20bc6ab19b4a7f4a43fa67010cf5008140d8abab61d58d5
Running setup.py bdist_wheel for websocket-client
Stored in directory: /Users/PavelPolyakov/Library/Caches/pip/wheels/48/45/ba/e955834950f99f1ca7a5778808d7bef1d4962edb1a4b14600a
Running setup.py bdist_wheel for docker-py
Stored in directory: /Users/PavelPolyakov/Library/Caches/pip/wheels/18/b4/cb/62ce0b04a3055cf5d599f3cfd01213d8488e9b367fb9cfb238
Running setup.py bdist_wheel for dockerpty
Stored in directory: /Users/PavelPolyakov/Library/Caches/pip/wheels/52/29/66/0c53de7d30b0e2a838ba252f6db929e9cc3d528892e7d759d5
Successfully built docker-compose docopt PyYAML texttable websocket-client docker-py dockerpty
Installing collected packages: docopt, PyYAML, requests, texttable, six, websocket-client, docker-py, dockerpty, docker-compose
Successfully installed PyYAML-3.11 docker-compose-1.4.2 docker-py-1.3.1 dockerpty-0.3.4 docopt-0.6.2 requests-2.6.2 six-1.9.0 texttable-0.8.3 websocket-client-0.32.0
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ rehash *[master]
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ which docker-compose *[master]
/Users/PavelPolyakov/tmp/microservices-workshop/msworkshop/step4/.venv/bin/docker-compose
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose --version *[master]
docker-compose version: 1.4.2
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ python -c 'import docker ; print(docker.version)'
1.3.1
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ python -c 'import requests.packages.urllib3 ; print(requests.__version__) ; print(requests.packages.urllib3.__version__)'
2.6.2
1.10.3
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose --verbose ps *[master]
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ python compose-debug.py --verbose ps *[master]
ERROR : SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose build *[master]
Building frontend...
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
For some reason, you have all the versions higher then mine (if I understand correct):
- docker
- request
- urlib3
why? and could it have such dramatic effect?
PavelPolyakov
on 27 Sep 2015
For some reason, you have all the versions higher then mine (if I understand correct): ... why?
I'm not sure. Try deactivate-ing your virtualenv, removing it, and recreating it, but with ./.venv/bin/pip install --no-cache-dir docker-compose instead (note the addition of --no-cache-dir).
... and could it have such dramatic effect?
Apparently, yes. You're at least getting the same cryptic error for python compose-debug.py --verbose ps as docker-compose --verbose ps (instead of the stacktrace you were getting before). So chalk that up to requests, I guess.
FYI, this is what I get from my virtualenv (and a fresh ~/.docker/machine directory):
% deactivate
% rm -frv ./.venv
...
% for i in $( docker-machine ls --quiet ) ; do docker-machine stop "${i}" ; done
...
% docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
% mv -v ~/.docker/machine ~/.docker/machine.bak
/.../.docker/machine -> /.../.docker/machine.bak
% docker-machine create --driver virtualbox testes
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
To see how to connect Docker to this machine, run: docker-machine env testes
docker-machine create --driver virtualbox testes 8.12s user 6.22s system 13% cpu 1:47.48 total
% docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
testes virtualbox Running tcp://192.168.99.100:2376
% eval $( docker-machine env testes )
% docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
% virtualenv .venv
...
% . ./.venv/bin/activate
% ./.venv/bin/pip install --no-cache-dir docker-compose
...
% rehash
% cat Dockerfile
FROM debian:stable
% cat docker-compose.yml
deb1:
build: .
deb2:
image: debian:stable
links:
- deb1
% docker-compose ps
Name Command State Ports
------------------------------
% docker-compose --verbose build
Compose version 1.4.2
Docker base_url: https://192.168.99.100:2376
Docker version: KernelVersion=4.0.9-boot2docker, Os=linux, BuildTime=Thu Sep 10 19:10:10 UTC 2015, ApiVersion=1.20, Version=1.8.2, GitCommit=0a8c2e3, Arch=amd64, GoVersion=go1.4.2
Building deb1...
docker build <- (pull=False, stream=True, nocache=False, tag=u'test_deb1', rm=True, path='/...', dockerfile=None)
docker build -> <generator object _stream_helper at 0x10f29b5f0>
Step 0 : FROM debian:stable
stable: Pulling from library/debian
401015d2a1e5: Pull complete
315baabd82d5: Pull complete
library/debian:stable: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:2ee35f51e54da93075fe46631a28d84ef4e23eb4ca51e7a8ef9f9ba625e7f6be
Status: Downloaded newer image for debian:stable
---> 315baabd82d5
Successfully built 315baabd82d5
docker close <- ()
docker close -> None
deb2 uses an image, skipping
Nope :(
... steps above were done as well ...
[~/tmp/docker-compose-fixing]$ for i in $( docker-machine ls --quiet ) ; do docker-machine stop "${i}" ; done
[~/tmp/docker-compose-fixing]$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
testes virtualbox Stopped
[~/tmp/docker-compose-fixing]$ mv -v ~/.docker/machine ~/.docker/machine.bak
/Users/PavelPolyakov/.docker/machine -> /Users/PavelPolyakov/.docker/machine.bak/machine
[~/tmp/docker-compose-fixing]$ docker-machine create --driver virtualbox testes
Creating CA: /Users/PavelPolyakov/.docker/machine/certs/ca.pem
Creating client certificate: /Users/PavelPolyakov/.docker/machine/certs/cert.pem
Image cache does not exist, creating it at /Users/PavelPolyakov/.docker/machine/cache...
No default boot2docker iso found locally, downloading the latest release...
Downloading https://github.com/boot2docker/boot2docker/releases/download/v1.8.2/boot2docker.iso to /Users/PavelPolyakov/.docker/machine/cache/boot2docker.iso...
Creating VirtualBox VM...
Creating SSH key...
Error creating machine: exit status 1
You will want to check the provider to make sure the machine and associated resources were properly removed.
[~/tmp/docker-compose-fixing]$ docker-machine create --driver virtualbox testes
Creating CA: /Users/PavelPolyakov/.docker/machine/certs/ca.pem
Creating client certificate: /Users/PavelPolyakov/.docker/machine/certs/cert.pem
Image cache does not exist, creating it at /Users/PavelPolyakov/.docker/machine/cache...
No default boot2docker iso found locally, downloading the latest release...
Downloading https://github.com/boot2docker/boot2docker/releases/download/v1.8.2/boot2docker.iso to /Users/PavelPolyakov/.docker/machine/cache/boot2docker.iso...
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
To see how to connect Docker to this machine, run: docker-machine env testes
[~/tmp/docker-compose-fixing]$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
testes virtualbox Running tcp://192.168.99.109:2376
[~/tmp/docker-compose-fixing]$ eval $( docker-machine env testes )
[~/tmp/docker-compose-fixing]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[~/tmp/docker-compose-fixing]$ virtualenv .venv
New python executable in .venv/bin/python
Installing setuptools, pip, wheel...done.
[~/tmp/docker-compose-fixing]$ . ./.venv/bin/activate
(.venv)[~/tmp/docker-compose-fixing]$ ./.venv/bin/pip install --no-cache-dir docker-compose
Collecting docker-compose
Downloading docker-compose-1.4.2.tar.gz (82kB)
100% |████████████████████████████████| 86kB 16.0MB/s
Collecting docopt<0.7,>=0.6.1 (from docker-compose)
Downloading docopt-0.6.2.tar.gz
Collecting PyYAML<4,>=3.10 (from docker-compose)
Downloading PyYAML-3.11.tar.gz (248kB)
100% |████████████████████████████████| 249kB 2.7MB/s
Collecting requests<2.7,>=2.6.1 (from docker-compose)
Downloading requests-2.6.2-py2.py3-none-any.whl (470kB)
100% |████████████████████████████████| 471kB 15.5MB/s
Collecting texttable<0.9,>=0.8.1 (from docker-compose)
Downloading texttable-0.8.3.tar.gz
Collecting websocket-client<1.0,>=0.32.0 (from docker-compose)
Downloading websocket_client-0.32.0.tar.gz (192kB)
100% |████████████████████████████████| 192kB 20.1MB/s
Collecting docker-py<1.4,>=1.3.1 (from docker-compose)
Downloading docker-py-1.3.1.tar.gz (49kB)
100% |████████████████████████████████| 53kB 10.5MB/s
Collecting dockerpty<0.4,>=0.3.4 (from docker-compose)
Downloading dockerpty-0.3.4.tar.gz
Collecting six<2,>=1.3.0 (from docker-compose)
Downloading six-1.9.0-py2.py3-none-any.whl
Installing collected packages: docopt, PyYAML, requests, texttable, six, websocket-client, docker-py, dockerpty, docker-compose
Running setup.py install for docopt
Running setup.py install for PyYAML
Running setup.py install for texttable
Running setup.py install for websocket-client
Running setup.py install for docker-py
Running setup.py install for dockerpty
Running setup.py install for docker-compose
Successfully installed PyYAML-3.11 docker-compose-1.4.2 docker-py-1.3.1 dockerpty-0.3.4 docopt-0.6.2 requests-2.6.2 six-1.9.0 texttable-0.8.3 websocket-client-0.32.0
(.venv)[~/tmp/docker-compose-fixing]$ rehash
(.venv)[~/tmp/docker-compose-fixing]$ cat Dockerfile
FROM debian:stable
(.venv)[~/tmp/docker-compose-fixing]$ cat docker-compose.yml
deb1:
build: .
deb2:
image: debian:stable
links:
- deb1
(.venv)[~/tmp/docker-compose-fixing]$ docker-compose ps
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
However, library versions are the same, basically:
(.venv)[~/tmp/docker-compose-fixing]$ python -c 'import requests.packages.urllib3 ; print(requests.__version__) ; print(requests.packages.urllib3.__version__)'
2.6.2
1.10.3
(.venv)[~/tmp/docker-compose-fixing]$ python -c 'import docker ; print(docker.version)'
1.3.1
However, library versions are the same, basically:
(.venv)[~/tmp/docker-compose-fixing]$ python -c 'import requests.packages.urllib3 ; print(requests.__version__) ; print(requests.packages.urllib3.__version__)' 2.6.2 1.10.3 (.venv)[~/tmp/docker-compose-fixing]$ python -c 'import docker ; print(docker.version)' 1.3.1
Oops...you're right. That's my fault (I was getting my terminals confused). I was (mistakenly) partially copying version numbers from my terminal where I had done something like pip install --upgrade ...). :weary: In my working environment, I have the same as you:
% which python
.../.venv/bin/python
% python -c 'import docker ; print(docker.version)'
1.3.1
% python -c 'import requests.packages.urllib3 ; print(requests.__version__) ; print(requests.packages.urllib3.__version__)'
2.6.2
1.10.3
What do you get for this in your current environment?
( set -x ; openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -tls1 -CAfile "${DOCKER_CERT_PATH}/ca.pem" -cert "${DOCKER_CERT_PATH}/cert.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null ; echo "exit code: ${?}" ) | awk '$0 ~ /TLS session ticket:/ { ticket = 1; } !ticket || $1 !~ /^[0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f]$/ { print; }'
Also, can you try this (corrected)?
% deactivate
% virtualenv -p /usr/bin/python .venv2
...
% ./.venv2/bin/pip install --no-cache-dir docker-compose
% ./.venv2/bin/python -c 'import ssl ; print(ssl.__file__) ; print (ssl.OPENSSL_VERSION)'
...
% ./.venv2/bin/python -c 'import _ssl ; print(_ssl.__file__) ; print (_ssl.OPENSSL_VERSION)' # note the underscore-ssl
...
% ./.venv2/bin/docker-compose ps
...
1st:
(.venv)[~/tmp/docker-compose-fixing]$ ( set -x ; openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -tls1 -CAfile "${DOCKER_CERT_PATH}/ca.pem" -cert "${DOCKER_CERT_PATH}/cert.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null ; echo "exit code: ${?}" ) | awk '$0 ~ /TLS session ticket:/ { ticket = 1; } !ticket || $1 !~ /^[0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f]$/ { print; }'
+-zsh:111> openssl s_client -showcerts -connect 192.168.99.109:2376 -tls1 -CAfile /Users/PavelPolyakov/.docker/machine/machines/testes/ca.pem -cert /Users/PavelPolyakov/.docker/machine/machines/testes/cert.pem -key /Users/PavelPolyakov/.docker/machine/machines/testes/key.pem
depth=1 O = PavelPolyakov
verify return:1
depth=0 O = testes
verify return:1
DONE
CONNECTED(00000003)
---
Certificate chain
0 s:/O=testes
i:/O=PavelPolyakov
-----BEGIN CERTIFICATE-----
MIIC/zCCAemgAwIBAgIQHxlNz0mEt4afcgoccLhiGzALBgkqhkiG9w0BAQswGDEW
MBQGA1UEChMNUGF2ZWxQb2x5YWtvdjAeFw0xNTA5MjcxODE2MDBaFw0xODA5MTEx
ODE2MDBaMBExDzANBgNVBAoTBnRlc3RlczCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBAOhSHinVHHH9BjwuGyNwMPlvyLQvJHngyUqbRx5L+NaOL6lTGinr
vXG+xosolY8TuTdTd9LZu25ccQMQ3AbwVV0XNklkqxUc1cQma9pZ/9VsmfOlHl+n
HkgFrJIhiV+GpsjbXy/oNiwkuNGIu0/DByPbf7NWVoTKMOUKyJBFQnqWE9VWLXA/
4BVnl/8VOFnCdGCqx8EkJEk0RC4h8No76M0Z/t32w2wj1Cj8BFzO5slGwwcFuOTt
TQNfOyicxg/KUPeeLEr0st9WenBOE90lzozecqeBLLEwUcpFFFdNrC79Gy4W9YEd
f60kNsAxFxlvKLg5svxNJPwBmiA2mam54GkCAwEAAaNQME4wDgYDVR0PAQH/BAQD
AgCoMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATAMBgNVHRMBAf8EAjAA
MA8GA1UdEQQIMAaHBMCoY20wCwYJKoZIhvcNAQELA4IBAQCfOH1AxRjw+ZDS04S8
ne+uUwk7KW5Ws2lZELAzedB9Izz/QOUyN89GGSttPrlUP0YOmMbwXOqFyMVuqx7e
WEGcGH1dduCbvP7SELBPYrocvAEQPWLqyA5W7uRiJA9KipRe2r5bVer1bvADt6Ir
UuNpoqYWM5fG+X0LT3PbNCEaxC8Itzb4KnVVCHrE+Me931uvOahLWrcisM6eQEYS
asDytbK9KvKM86TVEkyWFEAzTc4VWCLeK3JTy6Tfi+86Yqvk0TTaoh1X6hjKnfWY
5xT7SOL6DsNBZHDAUG0W/2FDR8EYfOmKoDW4PH5F7Qyamd0fcgMBrwZ/uMqhXHAd
vIsF
-----END CERTIFICATE-----
---
Server certificate
subject=/O=testes
issuer=/O=PavelPolyakov
---
Acceptable client certificate CA names
/O=PavelPolyakov
Client Certificate Types: RSA sign, ECDSA sign
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2165 bytes and written 1378 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1
Cipher : ECDHE-RSA-AES256-SHA
Session-ID: 9329F3198FC79ACBDF2CAB18200C02B19F9AFD84AF4462C7044878A6A16BF257
Session-ID-ctx:
Master-Key: 52FFF668D11FCE75BADCA40CAFB7F016B888E975D87E2010EAC9ADE5CC0EEB5B562074D822D7306991E149BC2DCA83E6
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket:
Start Time: 1443381037
Timeout : 7200 (sec)
Verify return code: 0 (ok)
---
+-zsh:111> echo 'exit code: 0'
exit code: 0
and 2nd:
(.venv)[~/tmp/docker-compose-fixing]$ deactivate
[~/tmp/docker-compose-fixing]$ virtualenv -p /usr/bin/python .venv2
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in .venv2/bin/python
Installing setuptools, pip, wheel...done.
[~/tmp/docker-compose-fixing]$ ./.venv2/bin/pip install --no-cache-dir docker-compose
Collecting docker-compose
Downloading docker-compose-1.4.2.tar.gz (82kB)
100% |████████████████████████████████| 86kB 1.9MB/s
Collecting docopt<0.7,>=0.6.1 (from docker-compose)
Downloading docopt-0.6.2.tar.gz
Collecting PyYAML<4,>=3.10 (from docker-compose)
Downloading PyYAML-3.11.tar.gz (248kB)
100% |████████████████████████████████| 249kB 2.6MB/s
Collecting requests<2.7,>=2.6.1 (from docker-compose)
Downloading requests-2.6.2-py2.py3-none-any.whl (470kB)
100% |████████████████████████████████| 471kB 11.1MB/s
Collecting texttable<0.9,>=0.8.1 (from docker-compose)
Downloading texttable-0.8.3.tar.gz
Collecting websocket-client<1.0,>=0.32.0 (from docker-compose)
Downloading websocket_client-0.32.0.tar.gz (192kB)
100% |████████████████████████████████| 192kB 12.5MB/s
Collecting docker-py<1.4,>=1.3.1 (from docker-compose)
Downloading docker-py-1.3.1.tar.gz (49kB)
100% |████████████████████████████████| 53kB 12.5MB/s
Collecting dockerpty<0.4,>=0.3.4 (from docker-compose)
Downloading dockerpty-0.3.4.tar.gz
Collecting six<2,>=1.3.0 (from docker-compose)
Downloading six-1.9.0-py2.py3-none-any.whl
Installing collected packages: docopt, PyYAML, requests, texttable, six, websocket-client, docker-py, dockerpty, docker-compose
Running setup.py install for docopt
Running setup.py install for PyYAML
Running setup.py install for texttable
Running setup.py install for websocket-client
Running setup.py install for docker-py
Running setup.py install for dockerpty
Running setup.py install for docker-compose
Successfully installed PyYAML-3.11 docker-compose-1.4.2 docker-py-1.3.1 dockerpty-0.3.4 docopt-0.6.2 requests-2.6.2 six-1.9.0 texttable-0.8.3 websocket-client-0.32.0
[~/tmp/docker-compose-fixing]$ ./.venv/bin/python -c 'import ssl ; print(ssl.__file__) ; print (ssl.OPENSSL_VERSION)'
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.pyc
OpenSSL 0.9.8zg 14 July 2015
[~/tmp/docker-compose-fixing]$ ./.venv2/bin/python -c 'import _ssl ; print(_ssl.__file__) ; print (_ssl.OPENSSL_VERSION)'
/Users/PavelPolyakov/tmp/docker-compose-fixing/.venv2/lib/python2.7/lib-dynload/_ssl.so
OpenSSL 0.9.8zg 14 July 2015
[~/tmp/docker-compose-fixing]$ ./.venv2/bin/docker-compose ps
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
Okay, I've officially run out of ideas. I cannot for the life of me fathom why openssl s_client can work, _and_ docker ps can work, but you're still getting SSL certificate verification errors with docker-compose across two different versions of Python and two different versions of OpenSSL. Can you do a straight Docker build (i.e., docker build .)?
posita
on 27 Sep 2015
Yes, docker build works ok:
[~/tmp/docker-compose-fixing]$ docker build -t hhh .
Sending build context to Docker daemon 27.28 MB
Step 0 : FROM debian:stable
---> 315baabd82d5
Successfully built 315baabd82d5
[~/tmp/docker-compose-fixing]$ docker run hhh --verbose
Error response from daemon: Cannot start container 4843e6bbbbbdd363356cb5536f82ce8cfada0960ed865242ebfe825ba7eade50: [8] System error: exec: "--verbose": executable file not found in $PATH
But docker-compose build not :)
[~/tmp/docker-compose-fixing]$ docker-compose build deb1
Building deb1...
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
Nevermind :) Thanks for the help and effort. Anyhow it is good to know that someone had docker-compose working on OSX, probably, as some point I would have it working as well!
PavelPolyakov
on 27 Sep 2015
I suppose one silver lining is that I no longer think your issue is with docker-machine. :disappointed_relieved:
This _may_ be an issue with docker-compose, but it's not docker/compose#890. Since it seems so difficult to reproduce outside of your environment, I suspect that you'll need to do some serious digging on your end to properly diagnose this (e.g., using pdb, maybe even dtrace or wireshark to fully understand what's going on).
I wish I could be of more help. :disappointed:
posita
on 27 Sep 2015
Closing this, please re-open as a new issue if you can provide a reproducible test case. Thanks
nathanleclaire
on 5 Nov 2015
@PavelPolyakov
One very dirty solution that did the trick for me :
Declare in the /etc/hosts, your docker ip address as localhost like
192.168.99.100 localhost
and then you put in the DOCKER_HOST localhost instead of your IP address.
export DOCKER_HOST=tcp://localhost:2376
And finally, on your mac, you will have either docker and docker-compose working together.
But I admit, it's really dirty :d but if it could help for a time, it worth may be a try.
herveDarritchon
on 24 Nov 2015
For those who are new to this issue, but have arrived here from outside (e.g,. web search), see @pmahoney's https://github.com/docker/compose/issues/890#issuecomment-159750418 regarding the CURL_CA_BUNDLE environment variable being a possible cause. (@PavelPolyakov, if you can verify this was the cause of your issue, please consider updating your OP so others don't have to read the entire commit history.)
posita
on 26 Nov 2015
@posita
Done, thanks for the help, good to have this issue solved.
PavelPolyakov
on 26 Nov 2015
I ran into the same error when I was using docker-compose commands. I finally uninstalled docker-machine and use boot2docker to re-build a mv and problem solved.
My OS is El Capitan and it seems mv created by docker-machine had some certificate conflicts with my system.
hongyanh
on 13 Dec 2015
Because docker-compose is python based and uses the requests python package, you must also unset the REQUESTS_CA_BUNDLE environment variable to make this work:
unset CURL_CA_BUNDLE
unset REQUESTS_CA_BUNDLE
docker-compose up
lcarva
on 8 May 2020
Related issues
iongion
·
4Comments
masaeedu
·
4Comments
jrz
·
5Comments
oobles
·
3Comments
AlexZeitler
·
4Comments
Most helpful comment
@PavelPolyakov Done. That's what did:
docker-machine regenerate-certsThe error was gone.