Machine: x509: certificate is valid for 192.168.99.103, not 192.168.99.100

[stephenlawrence]

After having my machine running on the .103 IP I rebooted my Mac. Upon reboot my docker-machine switched to the .100 address. When I tried to perform any docker commands on my machine I get things like this:

docker exec -it mycontainer bash:
FATA[0000] An error occurred trying to connect: Post https://192.168.99.100:2376/v1.16/containers/mycontainer/exec: x509: certificate is valid for 192.168.99.103, not 192.168.99.100

[ehazlett]

Can you give some more detail on the issue?

• What version of Machine?
• What driver (I'm assuming vbox based upon the IP)?
• Command line used to create machine

Thanks

[stephenlawrence]

docker-machine version 0.1.0
virtualbox
docker-machine create -d virtualbox --virtualbox-disk-size '40000' --virtualbox-memory '4096' devbox

I also added an --insecure-registry to the daemon so I could talk to our private registry (if that matters).

[ehazlett]

Would you mind testing from master (you can use https://docker-machine-builds.evanhazlett.com/latest/ if you don't want to build locally). I have tested this with VMs changing IPs and everything works as expected:

ehazlett machine> docker-machine ls
NAME        ACTIVE   DRIVER       STATE     URL
test-vbox   *        virtualbox   Running   tcp://192.168.99.100:2376
ehazlett machine> docker-machine stop test-vbox
ehazlett machine> docker-machine create -d virtualbox test-vbox-2
...
ehazlett machine> docker-machine ls
NAME          ACTIVE   DRIVER       STATE     URL
test-vbox              virtualbox   Stopped   
test-vbox-2   *        virtualbox   Running   tcp://192.168.99.100:2376
ehazlett machine> docker-machine start test-vbox
...      
ehazlett machine> docker-machine ls
NAME          ACTIVE   DRIVER       STATE     URL
test-vbox              virtualbox   Running   tcp://192.168.99.101:2376
test-vbox-2   *        virtualbox   Running   tcp://192.168.99.100:2376            
ehazlett machine> docker $(docker-machine config test-vbox) ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

You can see that test-vbox switched IPs.

[stephenlawrence]

Ok, here is what I did:

Stop the current machine
Start using the new machine bin
"machine ls" with new machine was ok and shows:
"devbox * virtualbox Running tcp://192.168.99.100:2376
devbox2 virtualbox Stopped"

"docker info" produces:
FATA[0000] An error occurred trying to connect: Get https://192.168.99.100:2376/v1.16/info: x509: certificate is valid for 192.168.99.103, not 192.168.99.100

[ehazlett]

thanks. did you try with the master build above? I ran the same procedure you did and I didn't have any cert issues.

[stephenlawrence]

I downloaded that "latest" binary.

[ehazlett]

Hmm ok. @sthulb can you try to see if you can reproduce this?

• Create a virtualbox machine
• Stop first machine
• Create second virtualbox machine
• Start first machine
• Check that the first machine IP changed
• Check if you can access the docker daemon on both

@stephenlawrence is that process correct?

Thanks

[stephenlawrence]

@ehazlett Technically this is not how it started. It started when I rebooted my Mac and the DM came up with a different IP on the reboot. I don't know if that is different than just doing what you describe. But now, any attempts at accessing that previously-working machine result in the x509 message.

[ehazlett]

@stephenlawrence i cannot re-create that as my IP always changes and the VM stays. I simply emulated the VM getting a different IP which should create the same thing I believe. Would you mind trying the above with the new binary and a fresh set of VMs? I'm wondering if the certs were created using an old build not using the correct process or something.

[cpuguy83]

@stephenlawrence Are you sure you don't have something in your bashrc that is messing with your settings, like DOCKER_CERT_PATH?

[md5]

Since I didn't see anyone else reference the relevant code, I'll just point out that machine is creating a certificate with the IP address in it here: https://github.com/docker/machine/blob/973b267fec3ec0a900e26557475b387891c0138a/host.go#L123

If the IP address changes, that cert will no longer work.

In the b2d init scripts, there is code that regenerates the server certificates if the relevant IP addresses change: https://github.com/boot2docker/boot2docker/blob/5db7efbb4e0557f6efefdb56cb0263f80ed55834/rootfs/rootfs/usr/local/etc/init.d/docker#L46

I'm not entirely sure why that bit of b2d code isn't triggered in the machine case since I thought you guys were using the b2d ISO.

[ehazlett]

Yeah thanks, I know this code is there however, when the IP changed (as
shown above) there were no issues when testing so I'm trying to at least
get this re-producable to test.

On Sat, Feb 14, 2015 at 12:58 AM, Mike Dillon [email protected]
wrote:

Since I didn't see anyone else reference the relevant code, I'll just
point out that machine is creating a certificate with the IP address in
it here:
https://github.com/docker/machine/blob/973b267fec3ec0a900e26557475b387891c0138a/host.go#L123

If the IP address changes, that cert will no longer work.

In the b2d init scripts, there is code that regenerates the server
certificates if the relevant IP addresses change:
https://github.com/boot2docker/boot2docker/blob/5db7efbb4e0557f6efefdb56cb0263f80ed55834/rootfs/rootfs/usr/local/etc/init.d/docker#L46

I'm not entirely sure why that bit of b2d code isn't triggered in the
machine case since I thought you guys were using the b2d ISO.

—
Reply to this email directly or view it on GitHub
https://github.com/docker/machine/issues/531#issuecomment-74363452.

[ehazlett]

@md5 @stephenlawrence would you mind checking to see if you have anything that would be using the b2d certs by chance? after debugging, it looks like the b2d certs are being used with machine.

[md5]

@ehazlett I'm not sure what you're asking me to check.

[ehazlett]

@md5 sorry i should have been more clear. check the environment variables DOCKER_CERT_PATH and DOCKER_HOST to see if they are pointed at the correct machine. It sounds like the the DOCKER_CERT_PATH may be configured to the b2d certs when accessing a machine or vice versa.

[md5]

@ehazlett if this were a case of an incorrect DOCKER_CERT_PATH, it would be a different error. The problem in that situation would either be that ca.pem is not a trust root for the cert presented by the Docker server on the machine-controlled instance or that the cert.pem client certificate is not accepted by the server.

The error that we're seeing here is clearly that the client (i.e. docker exec) is not accepting the server's certificate. It's attempting to hit the server at 192.168.99.100, but the server is presenting a certificate that is only valid for 192.168.99.103. This can only mean that a server that is responding at 192.168.99.100 on port 2376 is configured to use a certificate that was created for 192.168.99.103.

[ehazlett]

@md5 that makes sense. how do i re-create? as you can see above i've changed the IP of my virtualbox instances and don't see this issue.

[md5]

That's a good question. I haven't been able to reproduce it myself either.

[ehazlett]

Closing as we cannot seem to reproduce. My suspicion is that the environment variables are getting mixed for the boot2docker VM and the machine instance. I would recommend to check your .bashrc etc for anything that may be setting them.

[stephenlawrence]

I have this issue happening again. I had a docker-machine running on 192.168.99.101, and after a re-boot of my Mac, that machine is now running on 192.168.99.100.

This seems as if its a docker issue since I can do "docker-compose run" into a container OK, but trying any docker commands throws this error:

$docker ps
FATA[0000] An error occurred trying to connect: Get https://192.168.99.100:2376/v1.17/containers/json: x509: certificate is valid for 192.168.99.101, not 192.168.99.100

[stephenlawrence]

Current env:
DOCKER_CERT_PATH=/Users/myusername/.docker/machines/.client
DOCKER_TLS_VERIFY=yes
DOCKER_HOST=tcp://192.168.99.100:2376

$ ls -l ~/.docker/machines/.client/
total 48
-rw-r--r-- 1 myusername staff 1054 Feb 11 10:21 ca.pem
-rw-r--r-- 1 myusername staff 1054 Jan 30 08:53 ca.pem.bak
-rw-r--r-- 1 myusername staff 1094 Feb 11 10:21 cert.pem
-rw-r--r-- 1 myusername staff 1094 Jan 30 08:53 cert.pem.bak
-rw------- 1 myusername staff 1675 Feb 11 10:21 key.pem
-rw------- 1 myusername staff 1679 Jan 30 08:53 key.pem.bak

[ehazlett]

@stephenlawrence ugh ok. would you be using b2d as well by chance? the only time i've seen this was something between b2d and machine. i can't personally reproduce but it had to do with the certs and env vars.

maybe we need that "regenerate-certs" command sooner rather than later :)

[stephenlawrence]

Well, I had been using b2d alone before I switch to DM. DM uses b2d as well no?

[razic]

@stephenlawrence just use machine to manage b2d's. delete your original b2d installation and VM

[parshap]

I've had this happen to me as well. I used machine to create the virtual machine, not b2d directly. A command to regenerate certs sounds helpful.

[stephenlawrence]

Removing b2d doesn't necessarily resolve my issue, but I have removed it.

I would add that this is making docker-machine unusable for me at the moment. I just created a new machine hoping to get rid of that issue, but I have the same issue on a new machine.

[ehazlett]

@stephenlawrence can you try my PR above and see if that solves the issue? it should as it will regenerate the certs for the instance.

[jmorganca]

702 fixed this for me :smiley: Regenerating the certificate after machine start works after I reboot my Mac.

[ehazlett]

@jeffdm thx for the feedback!

[bfirsh]

I think this is still an issue. Two possible solutions I can think of:

• Assume that IP addresses are going to change. Do we need to check the IP address in the TLS handshake? Can't we validate the specific certificate that the machine has?
• Not rely on DHCP. Perhaps we can give VMs static IPs?

[cwilkes]

I'm having a similar problem ... the Cisco VPN client I'm using sets up ipfw rules which pretty much block the ability to go to 192.168.x addresses that are bound to vboxnetN.

So I setup a port mapping rule in virtualbox so that 127.0.0.1:2376 hits the docker server. The trouble is is that the docker server's certificate is for 192.168.99.101 not 127.0.0.1:

$ export DOCKER_TLS_VERIFY=""
$ docker images
FATA[0000] Get http://127.0.0.1:2376/v1.17/images/json: malformed HTTP response "\x15\x03\x01\x00\x02\x02". Are you trying to connect to a TLS-enabled daemon without TLS?
$ export DOCKER_TLS_VERIFY=yes
$ docker images
FATA[0000] An error occurred trying to connect: Get https://127.0.0.1:2376/v1.17/images/json: x509: certificate is valid for 192.168.99.101, not 127.0.0.1
$ docker --tlsverify=false images
(this works)

While this isn't a show stopper as I can put "--tlsverify=false" in all my hand run commands but other tools like fig wouldn't do that. Also for some reason DOCKER_OPTS doesn't work for me.

Perhaps an option when creating a virtualbox instance to have it also make the certificate valid for another address? I can see that also being a problem if the server has an internal and external IP address that are both used (for example in ec2, but then I would use the DNS name).

[sthulb]

@cwilkes there is/was an issue to allow that to be configured.

[cwilkes]

Looks like https://github.com/docker/machine/pull/702 might help me out, and I see it was pushed into master today https://github.com/docker/machine/compare/v0.1.0...master . Hopefully there will be a 0.1.1 release soon :)

[ehazlett]

Ok after discussing this with @sthulb and @nathanleclaire I think we should do a check similar to what b2d does. We can inspect the IP at start of the machine and compare that against the IP SAN in the cert. If they don't match, we can regenerate. Thoughts @sthulb @nathanleclaire ?

[sthulb]

Agreed.

[nathanleclaire]

Yes please @ehazlett

[killerwolf]

+1 having the same issue

[duro]

I am running into this as well. I spun up a machine using the AWS driver and then assigned an Elastic IP to the box. Tried stoping and starting the machine via the CLI, but every time I run a docker ps after activating the env I get:

certificate is valid for 52.11.XXX.XXX, not 52.10.XXX.XXX

I did try to manually change the IP in the `~/.docker/machine' config. That did not work.

[ehazlett]

@duro @killerwolf @cwilkes @jeffdm @stephenlawrence @md5

please checkout #770. this adds auto-regeneration in the event of a certificate error. note that it only works with the config and env commands so if you have an issue with docker (i.e. docker ps etc) you will need to run $(docker-machine env <name>) or docker $(docker-machine config <name>) ps to have them fixed. That PR also adds the ability to regenerate certs manually with the regenerate-certs command (I used the functionality from #702).

Hopefully this will resolve the TLS errors once and for all :)

[rossbachp]

I detect the AWS IP change after normal reboot of the aws instance.
docker-machine ssh amazonec2-03 only work after manually edit the IPAddress at /.docker/machine/machines/amazonec2-03/confg.json

docker daemon certs don't work, I can check the #770 tomorrow.

[ehazlett]

PTAL @nathanleclaire @sthulb

[jareware]

Having the same issue, will check out #770 soon...

[rossbachp]

You can use the current head:)

[ehazlett]

Or you can use the RCs -- they have the fix.

[gschmutz]

had the same problem but with the latest milestone version 0.2.0 it is solved! Thanks!

[ehazlett]

@gschmutz thanks for the feedback!

[gravis]

I'm having this since my upgrade to 1.7.0.
Restarting b2d will lead to this error:

$ eval "$(boot2docker shellinit)"
Writing /Users/gravis/.boot2docker/certs/boot2docker-vm/ca.pem
Writing /Users/gravis/.boot2docker/certs/boot2docker-vm/cert.pem
Writing /Users/gravis/.boot2docker/certs/boot2docker-vm/key.pem
Your environment variables are already set correctly.
$ docker ps -a
An error occurred trying to connect: Get https://192.168.59.103:2376/v1.19/containers/json?all=1: x509: certificate is valid for 127.0.0.1, 10.0.2.15, not 192.168.59.103

thanks

[cpuguy83]

@gravis How come you are using boot2docker shellinit with machine?

[gravis]

Sorry, I posted in the wrong repo :)
I thought it was b2d, I messed up with my tabs.

[allen-servedio]

For anyone else that comes across this, here is a possible solution: https://github.com/boot2docker/boot2docker/issues/824#issuecomment-113904164
AND
https://github.com/boot2docker/boot2docker/pull/411

So far, this has fixed the problem for me. I can see that the /var/lib/boot2docker/tls/hostnames file does not have the VM's IP (ie. it does not have the IP you see when you type: boot2docker ip), so this allows it to get added (it appears at the end of the hostnames list after adding the delay they suggested).

[pasitolonen]

I tested boot2docker some time ago, forgot it for a long time and today I downloaded the latest version of boot2docker and installed it. After installation, I encountered this same X.509 problem mentioned here and I solved it by

1) removing .boot2docker directory (located in my home directory) and its contents and
2) removing all boot2docker virtual machine files as well

Hope this helps.

ps. Since I only tested boot2docker, I didn't care if I lost something when deleting that stuff...

[jvivian]

I fixed the problem the same way pasitolonen fixed it. Deleting .boot2docker dir and running boot2docker init. This has the side-effect of deleting my images (which I didn't mind).

[rajaraodv]

This happens to me whenever the wifi changes. For example when I use it from home and then again in office.

Error: An error occurred trying to connect: Get https://192.168.59.104:2376/v1.19/version: x509: certificate is valid for 127.0.0.1, 10.0.2.15, not 192.168.59.104

$ docker version
Client version: 1.7.0
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 0baf609
OS/Arch (client): darwin/amd64

[rajaraodv]

Apparently this seems to solve the problem: https://github.com/boot2docker/boot2docker/issues/938#issuecomment-118211836

alias docker="docker --tlsverify=false"

[naoko]

I experienced this after I reallocated RAM. as rajaraodv metnioned alias docker="docker --tlsverify=false" solved

[indygreg]

Disabling security (--tlsverify=false) should never be a recommended workaround.

A workaround that works for me is boot2docker ssh 'sudo /etc/init.d/docker restart', which will cause Docker to generate a new x509 certificate valid for boot2docker in addition to other IP addresses. Please try that before turning off security features.

[squarism]

@indygreg solution worked for me. Many thanks.

[rynam0]

@indygreg saved me as well. thanks!

[hupiv]

thanks it works for me too :-)

[nathanleclaire]

Hi all, if you are still encountering the cert issue with boot2docker please make sure to first try boot2docker upgrade to update your VM to the latest (1.7.1 fixes this issue), and if anything else comes up report the issues at github.com/boot2docker/boot2docker-cli . This repo is for Docker Machine issues. Thanks!

[dayuoba]

@indygreg awesome thx

[xiaojiet2142]

@indygreg thanks it works for me too :-)

[ivarrian]

@indygreg Worked for me too! Thank you.

[ajsharp]

Worked for me, thanks!

[reustle]

Thanks @indygreg! I'm adding a link here to the comment so googler can easily see what he recommended.

https://github.com/docker/machine/issues/531#issuecomment-120554419

[johnmccabe]

Hit this problem with Docker Toolbox 1.9.1 on Windows after a reboot resulting in the VM being allocated a different IP, and this was the top hit on google.

So for anyone else hitting this, the current fix is even easier. (where default is your docker machine)

docker-machine regenerate-certs default

[sleepylemur]

Thanks @johnmccabe, worked for me with Docker Toolbox 1.10.0 on OS X.

[onnimonni]

This happens a lot to me because of switching wlans and using VPN sometimes. Is there a way that this could be Automated? Or could The certificates be valid for a range of ip addresses instead?

[VonC]

@johnmccabe Note, you can simply make sure your VM always starts with the same IP: http://stackoverflow.com/a/35367277/6309

[onnimonni]

I'm using OS X and tired of using these hacks and having them broken eventually.

Vagrant has option to set ip address in configuration. I can't understand why this kind of parameter or flag is not already existing in docker-machine (Just comparing to vagrant because both use virtualbox)

[cpuguy83]

@onnimonni: this has been resolved for a long long time (almost a year). Machine includes a command to regenerate certs in case the ip changed.

[lancehunt]

I just ran into this issue for the first time and checked everything in this thread with any of the following options successfully fixing the issue: (pick one)

• docker-machine regenerate-certs default2
• docker-machine kill default2 then docker-machine create default2 ...
• docker-machine upgrade default2 (if not already upgraded)

To repro this issue on OSX:

1. Install Docker Toolbox (v1.10)
2. Create a default machine which gets 192.168.99.100
3. Create a default2 machine which gets 192.168.99.101
4. Shutdown both (sometimes requires a reboot too)
5. docker-compose start default2 which gets 192.168.99.100 (and x509 cert conflict)

[MikeTLive]

so it comes down to multiple machines started in different order may cause a different IP which breaks the certificate which requires searching github for this issue to find the resolution of creating a new certificate to allow you to use them again ... is there a way to PREVENT this happening? is it so unusual for folks to flip between multiple docker machines?

[ivancarrancho]

$alias docker="docker --tlsverify=false" Work for me!

[nathanleclaire]

@ivancarrancho Why not docker-machine regenerate-certs -f and keep TLS enabled?

[ivancarrancho]

@nathanleclaire yes, that's better than "--tlsveify" many many thanks

[l15k4]

@nathanleclaire because it takes ~ 4 minutes ... Imagine you have a cluster of 6+ nodes... I will have to write a parallel certificate regenerator thingy to not spend whole day by regenerating some certificates.... Moreover it restarts all docker containers on the target machine ...

This requirement to regenerate certificates after IP address change is a huge pain on aws cloud... Public IPs change all the time. The only solution known to me is creating new instances from an ec2 instance, but for some reason it doesn't work https://github.com/docker/machine/pull/1453#issuecomment-215371216

Btw any idea if docker-machine create --amazonec2-use-private-address is supposed to create an ec2 instance from another ec2 instance?

This is the only way I'm aware of how to avoid constant certificate regeneration...

[nathanleclaire]

Btw any idea if docker-machine create --amazonec2-use-private-address is supposed to create an ec2 instance from another ec2 instance?

Yes, unless there is some other way you have to access the private IP from the creation node (e.g. proxy).

We are considering a variety of solutions, e.g. Elastic IP, to potentially resolve this issue.

[l15k4]

yeah,but as I mentioned,it is hanging on ssh access even with this option
enabled
On May 2, 2016 20:26, "Nathan LeClaire" [email protected] wrote:

Btw any idea if docker-machine create --amazonec2-use-private-address is
supposed to create an ec2 instance from another ec2 instance?

Yes, unless there is some other way you have to access the private IP from
the creation node (e.g. proxy).

We are considering a variety of solutions, e.g. Elastic IP, to potentially
resolve this issue.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/docker/machine/issues/531#issuecomment-216319103

[nathanleclaire]

Maybe the security group is not configured properly? Currently, if using --amazonec2-use-private-ip-address, it's assumed that user is willing to ensure that these sorts of network access details are configured properly beforehand.

[ShashankaNataraj]

I dont understand why I have to run that ugly $eval statement everytime docker needs to be started via a terminal.

I dont understand why this issue even exists.

Docker is seeming more and more like a horribly broken product that a lot of people got behind because it was the "in" thing.

[twang2218]

I'm using the latest version of docker and docker-machine, and I still have the same issue, I created several virtualbox docker host by

    docker-machine create -d virtualbox xxxx \
        --engine-opt="cluster-store=${KVSTORE}" \
        --engine-opt="cluster-advertise=eth1:2376" \
        ${NAME}
...

and almost every time I reboot the VMs, or reboot my Mac, I will face the error like certificate is valid for 192.168.99.100, not 192.168.99.101.

My versions are:

$ docker --version
Docker version 1.12.0-rc4, build e4a0dbc, experimental
$ docker-machine -v
docker-machine version 0.8.0-rc2, build 4ca1b85
$ vboxmanage -v
5.1.0r108711

Is there any solution to prevent those error to happen? or create a virtualbox host with a fixed IP?

[pyhedgehog]

Workaround: docker-machine provision.

[qixiuying]

Fixed the issue by docker-machine regenerate-certs xxx

$ docker-machine ls
NAME        ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
xxx   -        virtualbox   Running   tcp://192.168.99.100:2376           Unknown   Unable to query docker version: Get https://192.168.99.100:2376/v1.15/version: x509: certificate is valid for 192.168.99.101, not 192.168.99.100
~$ docker-machine regenerate-certs xxx
Regenerate TLS machine certs?  Warning: this is irreversible. (y/n): y
Regenerating TLS certificates
Waiting for SSH to be available...
Detecting the provisioner...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
~$ docker-machine ls
NAME        ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
xxx   *        virtualbox   Running   tcp://192.168.99.100:2376           v1.12.6

[themsay]

Following @VonC link I came across docker-machine-ipconfig https://github.com/fivestars/docker-machine-ipconfig

$ cd ~/.docker/
$ git clone https://github.com/fivestars/docker-machine-ipconfig.git
# Add to ~/.profile
$ echo 'alias docker-machine-ipconfig=~/.docker/docker-machine-ipconfig/docker-machine-ipconfig' >> ~/.profile 
$ source ~/.profile

E.g: Assign machine-name a static IP:

$ docker-machine-ipconfig static machine-name
# or specify implicit IP
$ docker-machine-ipconfig static machine-name 192.168.99.110

This eliminate need of docker-machine regenerate-certs -f machine-name on every restart. An easy way to set machine static IP running on virtualbox.

[pyhedgehog]

Does it supports Windows? I mean both "for" and "on".

[z0r1k]

it looks like it is still an issue. any way to fix it?

[mvma]

@johnmccabe,
thanks.
works fine. i just restarted my toolbox and everything is back in the game.

[alaagalal]

how can i have static account of container , it enables me to change the account every time i restart the docker-machine ?

[tannerchung]

I see the workaround with docker-machine-ipconfig but just for documentation sake I'm reporting that this happens with Windows 10 Docker Machine and Hyper-V as well

[rdp]

Might want to tweak this message "Started machines may have new IP addresses. You may need to re-run the docker-machine env command." to mention something like "and also do docker-machine regenerate-certs ..." FWIW...

[jotaoncode]

@rdp nice catch, I had this problem was half an hour looking what had happened, and after trying to do some things with kubernetes, installing and uninstalling things... running
docker-machine env
the IP matched in my envs matched the one that was producing the error with the certificate...
then:
eval $(docker-machine env)
make the configuraiton in place for me...

[PatMyron]

minikube stop && minikube start fixed this for me

[Kihagi]

minikube stop && minikube start fixed this for me

@PatMyron Surprisingly, this worked for me too!