machine 🚀 - unexpected EOF on remove machine

When there has been a failure during the creation process, removing with the force option should usually do the trick : docker-machine rm -f machine. Those error messages could probably be improved though.

bamarni on 11 May 2016

🎉4

+1 to @bamarni : -f is always intended to be the escape hatch. Not sure what's up with that EOF though.

nathanleclaire on 2 Jun 2016

I hit this today after hitting #3394.

replaid on 28 Jul 2016

👍1

I've gotten this a bit too... will have to look into it

nathanleclaire on 28 Jul 2016

We're facing exactly the same issue with:

docker-machine version 0.8.2, build e18a919

Machines automatically created be gitlab-ci-multi-runner on digitalocean stay in a defective state:

NAME                                            ACTIVE   DRIVER         STATE     URL                         SWARM   DOCKER    ERRORS
runner-05271b50-ci-runner-1480442822-b39db0ca   -        digitalocean   Error                                         Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480442822-ef9148dd   -        digitalocean   Error                                         Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480442832-a80d1878   -        digitalocean   Error                                         Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480442832-de96936a   -        digitalocean   Error                                         Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480442834-80ffc8d4   -        digitalocean   Error                                         Unknown   dropletID is invalid because cannot be less than 1

When i try to delete them i get the following:

About to remove runner-05271b50-ci-runner-1480442822-b39db0ca
Are you sure? (y/n): y
Error removing host "runner-05271b50-ci-runner-1480442822-b39db0ca": unexpected EOF

The strange thing is, that sometimes the machines exist on DO, sometimes not?!

pnull on 30 Nov 2016

curious what inspect shows for one of those, e.g., runner-05271b50-ci-runner-1480442822-b39db0ca.... sounds like the config.json may have gotten corrupted.

nathanleclaire on 2 Dec 2016

My guess on the EOF is, plugin is panic-ing, so anyone hitting this issue, _please_ make sure to post same output with docker-machine --debug so I can see the logs

nathanleclaire on 2 Dec 2016

@nathanleclaire Sure thing... here you go:

root@gitlab-ci-autoscale-runner:~# docker-machine --debug ls
Docker Machine Version:  0.8.2, build e18a919
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:41957
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:36081
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:41716
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:41469
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:41849
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:45806
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:46082
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:35165
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:45839
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:35635
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:33504
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:42940
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:33666
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:34058
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver digitalocean
Plugin server listening at address 127.0.0.1:45106
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
timeout set to 10s
(runner-05271b50-ci-runner-1480633251-11ab463f) Calling .GetURL
(runner-05271b50-ci-runner-1480624459-e0af5dec) Calling .GetURL
(runner-05271b50-ci-runner-1480625027-c6861d0b) Calling .GetURL
(runner-05271b50-ci-runner-1480625029-9d254439) Calling .GetURL
(runner-05271b50-ci-runner-1480625031-214ca979) Calling .GetURL
(runner-05271b50-ci-runner-1480625033-c77e25c0) Calling .GetURL
(runner-05271b50-ci-runner-1480625035-7f797c24) Calling .GetURL
(runner-05271b50-ci-runner-1480625037-e3dbf764) Calling .GetURL
(runner-05271b50-ci-runner-1480625039-92116735) Calling .GetURL
(runner-05271b50-ci-runner-1480625041-35b53752) Calling .GetURL
(runner-05271b50-ci-runner-1480625601-52d66efc) Calling .GetURL
(runner-05271b50-ci-runner-1480625604-eb9d79d7) Calling .GetURL
(runner-05271b50-ci-runner-1480626154-cb8c7f2a) Calling .GetURL
(runner-05271b50-ci-runner-1480626704-f21a76bd) Calling .GetURL
(runner-05271b50-ci-runner-1480626706-a5de8a9d) Calling .GetURL
(runner-05271b50-ci-runner-1480624459-e0af5dec) Calling .GetState
(runner-05271b50-ci-runner-1480625027-c6861d0b) Calling .GetState
(runner-05271b50-ci-runner-1480626704-f21a76bd) Calling .GetState
(runner-05271b50-ci-runner-1480625601-52d66efc) Calling .GetState
(runner-05271b50-ci-runner-1480625604-eb9d79d7) Calling .GetState
(runner-05271b50-ci-runner-1480625039-92116735) Calling .GetState
(runner-05271b50-ci-runner-1480625037-e3dbf764) Calling .GetState
(runner-05271b50-ci-runner-1480625033-c77e25c0) Calling .GetState
(runner-05271b50-ci-runner-1480625031-214ca979) Calling .GetState
(runner-05271b50-ci-runner-1480625029-9d254439) Calling .GetState
(runner-05271b50-ci-runner-1480626706-a5de8a9d) Calling .GetState
(runner-05271b50-ci-runner-1480626154-cb8c7f2a) Calling .GetState
(runner-05271b50-ci-runner-1480626706-a5de8a9d) Calling .DriverName
(runner-05271b50-ci-runner-1480625601-52d66efc) Calling .DriverName
(runner-05271b50-ci-runner-1480625604-eb9d79d7) Calling .DriverName
(runner-05271b50-ci-runner-1480625029-9d254439) Calling .DriverName
(runner-05271b50-ci-runner-1480625033-c77e25c0) Calling .DriverName
(runner-05271b50-ci-runner-1480625039-92116735) Calling .DriverName
(runner-05271b50-ci-runner-1480625027-c6861d0b) Calling .DriverName
(runner-05271b50-ci-runner-1480626704-f21a76bd) Calling .DriverName
(runner-05271b50-ci-runner-1480625035-7f797c24) Calling .GetState
(runner-05271b50-ci-runner-1480625041-35b53752) Calling .GetState
(runner-05271b50-ci-runner-1480626154-cb8c7f2a) Calling .DriverName
(runner-05271b50-ci-runner-1480624459-e0af5dec) Calling .DriverName
(runner-05271b50-ci-runner-1480625041-35b53752) Calling .DriverName
(runner-05271b50-ci-runner-1480625035-7f797c24) Calling .DriverName
(runner-05271b50-ci-runner-1480625031-214ca979) Calling .DriverName
(runner-05271b50-ci-runner-1480625037-e3dbf764) Calling .DriverName
Reading CA certificate from /root/.docker/machine/certs/ca.pem
Reading client certificate from /root/.docker/machine/certs/cert.pem
Reading client key from /root/.docker/machine/certs/key.pem
(runner-05271b50-ci-runner-1480633251-11ab463f) Calling .DriverName
NAME                                            ACTIVE   DRIVER         STATE     URL                        SWARM   DOCKER    ERRORS
runner-05271b50-ci-runner-1480624459-e0af5dec   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625027-c6861d0b   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625029-9d254439   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625031-214ca979   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625033-c77e25c0   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625035-7f797c24   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625037-e3dbf764   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625039-92116735   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625041-35b53752   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625601-52d66efc   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480625604-eb9d79d7   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480626154-cb8c7f2a   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480626704-f21a76bd   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480626706-a5de8a9d   -        digitalocean   Error                                        Unknown   dropletID is invalid because cannot be less than 1
runner-05271b50-ci-runner-1480633251-11ab463f   -        digitalocean   Running   tcp://138.68.86.154:2376           v1.11.2
Making call to close driver server
(runner-05271b50-ci-runner-1480624459-e0af5dec) Calling .Close
(runner-05271b50-ci-runner-1480624459-e0af5dec) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625027-c6861d0b) Calling .Close
(runner-05271b50-ci-runner-1480625027-c6861d0b) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625029-9d254439) Calling .Close
(runner-05271b50-ci-runner-1480625029-9d254439) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625031-214ca979) Calling .Close
(runner-05271b50-ci-runner-1480625031-214ca979) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625033-c77e25c0) Calling .Close
(runner-05271b50-ci-runner-1480625033-c77e25c0) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625035-7f797c24) Calling .Close
(runner-05271b50-ci-runner-1480625035-7f797c24) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625037-e3dbf764) Calling .Close
(runner-05271b50-ci-runner-1480625037-e3dbf764) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625039-92116735) Calling .Close
(runner-05271b50-ci-runner-1480625039-92116735) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625041-35b53752) Calling .Close
(runner-05271b50-ci-runner-1480625041-35b53752) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625601-52d66efc) Calling .Close
(runner-05271b50-ci-runner-1480625601-52d66efc) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480625604-eb9d79d7) Calling .Close
(runner-05271b50-ci-runner-1480625604-eb9d79d7) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480626154-cb8c7f2a) Calling .Close
(runner-05271b50-ci-runner-1480626154-cb8c7f2a) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480626704-f21a76bd) Calling .Close
(runner-05271b50-ci-runner-1480626704-f21a76bd) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480626706-a5de8a9d) Calling .Close
(runner-05271b50-ci-runner-1480626706-a5de8a9d) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(runner-05271b50-ci-runner-1480633251-11ab463f) Calling .Close
(runner-05271b50-ci-runner-1480633251-11ab463f) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary

As you can see, this is happening quite often... All these have been "created" within two days.

pnull on 2 Dec 2016

Here's a example config.json for one of those corrupted machines:

{
    "ConfigVersion": 3,
    "Driver": {
        "IPAddress": "",
        "MachineName": "runner-05271b50-ci-runner-1480625033-c77e25c0",
        "SSHUser": "core",
        "SSHPort": 22,
        "SSHKeyPath": "",
        "StorePath": "/root/.docker/machine",
        "SwarmMaster": false,
        "SwarmHost": "tcp://0.0.0.0:3376",
        "SwarmDiscovery": "",
        "AccessToken": "REMOVED;)",
        "DropletID": 0,
        "DropletName": "",
        "Image": "coreos-stable",
        "Region": "fra1",
        "SSHKeyID": 0,
        "SSHKeyFingerprint": "",
        "Size": "4gb",
        "IPv6": false,
        "Backups": false,
        "PrivateNetworking": true,
        "UserDataFile": ""
    },
    "DriverName": "digitalocean",
    "HostOptions": {
        "Driver": "",
        "Memory": 0,
        "Disk": 0,
        "EngineOptions": {
            "ArbitraryFlags": [],
            "Dns": null,
            "GraphDir": "",
            "Env": [],
            "Ipv6": false,
            "InsecureRegistry": [],
            "Labels": [],
            "LogLevel": "",
            "StorageDriver": "",
            "SelinuxEnabled": false,
            "TlsVerify": true,
            "RegistryMirror": [
                "REMOVED;)"
            ],
            "InstallURL": "https://get.docker.com"
        },
        "SwarmOptions": {
            "IsSwarm": false,
            "Address": "",
            "Discovery": "",
            "Agent": false,
            "Master": false,
            "Host": "tcp://0.0.0.0:3376",
            "Image": "swarm:latest",
            "Strategy": "spread",
            "Heartbeat": 0,
            "Overcommit": 0,
            "ArbitraryFlags": [],
            "ArbitraryJoinFlags": [],
            "Env": null,
            "IsExperimental": false
        },
        "AuthOptions": {
            "CertDir": "/root/.docker/machine/certs",
            "CaCertPath": "/root/.docker/machine/certs/ca.pem",
            "CaPrivateKeyPath": "/root/.docker/machine/certs/ca-key.pem",
            "CaCertRemotePath": "",
            "ServerCertPath": "/root/.docker/machine/machines/runner-05271b50-ci-runner-1480625033-c77e25c0/server.pem",
            "ServerKeyPath": "/root/.docker/machine/machines/runner-05271b50-ci-runner-1480625033-c77e25c0/server-key.pem",
            "ClientKeyPath": "/root/.docker/machine/certs/key.pem",
            "ServerCertRemotePath": "",
            "ServerKeyRemotePath": "",
            "ClientCertPath": "/root/.docker/machine/certs/cert.pem",
            "ServerCertSANs": [],
            "StorePath": "/root/.docker/machine/machines/runner-05271b50-ci-runner-1480625033-c77e25c0"
        }
    },
    "Name": "runner-05271b50-ci-runner-1480625033-c77e25c0"
}

pnull on 2 Dec 2016

Yeah, this is wrong:

        "DropletID": 0,

Not sure how it got that way (maybe driver panic) but the config.json is corrupted :( However if you fill in the DropletID to the correct value rm should (hopefully) work properly. Or, you could rm -f and clean up the machines yourself manually if they exist.

nathanleclaire on 2 Dec 2016

😕1

@nathanleclaire The real problem is not that i'm not able to resolve this once. The problem is, that this problem occurs multiple times per day!

We're running gitlab-ci-multi-runner (https://gitlab.com/gitlab-org/gitlab-ci-multi-runner) with docker+machine runner type. Machines are started and stopped multiple times per day and fill up the machine list.

Do you have any idea how i can debug this thing to find out WHY the problem occurs?

pnull on 2 Dec 2016

👍1

Do you have any idea how i can debug this thing to find out WHY the problem occurs?

Thanks -- If GitLab keeps --debug logs of the actions it's doing with machines and/or you can dupe where the error starts to occur it would be a start. It will be needed to pinpoint where the panic/issue is likely to be occurring. At some point (possibly even during create) it seems that there is an error causing the machine serialization to not work properly. The --debug output listed above just shows after the serialization failure has already occurred.. Where/when this error _starts_ to show up? Can you create successfully?

nathanleclaire on 2 Dec 2016

Hi, we're seeing the same issue when using vsphere.

Create

docker-machine --debug create \
--driver vmwarevsphere \
--vmwarevsphere-username=removed \
--vmwarevsphere-password=removed \
--vmwarevsphere-vcenter=removed \
--vmwarevsphere-vcenter-port=443 \
--vmwarevsphere-network=removed\
--vmwarevsphere-datastore=removed \
--vmwarevsphere-datacenter=removed \
--vmwarevsphere-hostsystem='removed'  \
runner-test


Docker Machine Version:  0.12.2, build 9371605
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver vmwarevsphere
Plugin server listening at address 127.0.0.1:35607
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(flag-lookup) Calling .GetMachineName
(flag-lookup) Calling .DriverName
(flag-lookup) Calling .GetCreateFlags
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver vmwarevsphere
Plugin server listening at address 127.0.0.1:33651
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(runner-test) Calling .GetMachineName
(runner-test) Calling .DriverName
(runner-test) Calling .GetCreateFlags
(runner-test) Calling .SetConfigFromFlags
Running pre-create checks...
(runner-test) Calling .PreCreateCheck
(runner-test) DBG | Connecting to vSphere for pre-create checks...
(runner-test) Calling .GetConfigRaw
Creating machine...
(runner-test) Calling .Create
(runner-test) Downloading /root/.docker/machine/cache/boot2docker.iso from http://removed/mirrors/boot2docker/v17.06.0-ce/boot2docker.iso...
(runner-test) 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
(runner-test) Generating SSH Keypair...
(runner-test) Creating VM...
(runner-test) Uploading Boot2docker ISO ...
(runner-test) Reconfiguring VM...
(runner-test) Waiting for VMware Tools to come online...
(runner-test) Provisioning certs and ssh keys...
(runner-test) DBG | Creating Tar key bundle...
(runner-test) Calling .GetConfigRaw
(runner-test) Calling .DriverName
(runner-test) Calling .DriverName
Waiting for machine to be running, this may take a few minutes...
(runner-test) Calling .GetState
Detecting operating system of created instance...
Waiting for SSH to be available...
Getting to WaitForSSH function...
(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: <nil>:
Detecting the provisioner...
(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
cat /etc/os-release
SSH cmd err, output: <nil>: NAME=Boot2Docker
VERSION=17.06.0-ce
ID=boot2docker
ID_LIKE=tcl
VERSION_ID=17.06.0-ce
PRETTY_NAME="Boot2Docker 17.06.0-ce (TCL 7.2); HEAD : 0672754 - Thu Jun 29 00:06:31 UTC 2017"
ANSI_COLOR="1;34"
HOME_URL="http://boot2docker.io"
SUPPORT_URL="https://github.com/boot2docker/boot2docker"
BUG_REPORT_URL="https://github.com/boot2docker/boot2docker/issues"

found compatible host: boot2docker
Provisioning with boot2docker...
(runner-test) Calling .GetMachineName
(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo /usr/bin/sethostname runner-test && echo "runner-test" | sudo tee /var/lib/boot2docker/etc/hostname
SSH cmd err, output: <nil>: Setting hostname to runner-test Done.
runner-test

(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
netstat -tln
SSH cmd err, output: <nil>: Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 :::22                   :::*                    LISTEN
tcp        0      0 :::2376                 :::*                    LISTEN

(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo mkdir -p /var/lib/boot2docker
SSH cmd err, output: <nil>:
(runner-test) Calling .GetMachineName
(runner-test) Calling .GetIP
Copying certs to the local machine directory...
generating server cert: /root/.docker/machine/machines/runner-test/server.pem ca-key=/root/.docker/machine/certs/ca.pem private-key=/root/.docker/machine/certs/ca-key.pem org=root.runner-test san=[172.16.6.10 localhost]
(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo /etc/init.d/docker stop
SSH cmd err, output: <nil>:
(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
if [ ! -z "$(ip link show docker0)" ]; then sudo ip link delete docker0; fi
SSH cmd err, output: <nil>:
Copying certs to the remote machine...
(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
printf '%s' '-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----
' | sudo tee /var/lib/boot2docker/ca.pem
SSH cmd err, output: <nil>: -----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----

(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
printf '%s' '-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----
' | sudo tee /var/lib/boot2docker/server.pem
SSH cmd err, output: <nil>: -----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----

(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
printf '%s' '-----BEGIN RSA PRIVATE KEY-----

-----END RSA PRIVATE KEY-----
' | sudo tee /var/lib/boot2docker/server-key.pem
SSH cmd err, output: <nil>: -----BEGIN RSA PRIVATE KEY-----

-----END RSA PRIVATE KEY-----

(runner-test) Calling .GetURL
(runner-test) Calling .DriverName
Setting Docker configuration on the remote daemon...
(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo mkdir -p /var/lib/boot2docker && printf %s "
EXTRA_ARGS='
--label provider=vmwarevsphere

'
CACERT=/var/lib/boot2docker/ca.pem
DOCKER_HOST='-H tcp://0.0.0.0:2376'
DOCKER_STORAGE=aufs
DOCKER_TLS=auto
SERVERKEY=/var/lib/boot2docker/server-key.pem
SERVERCERT=/var/lib/boot2docker/server.pem


" | sudo tee /var/lib/boot2docker/profile
SSH cmd err, output: <nil>:
EXTRA_ARGS='
--label provider=vmwarevsphere

'
CACERT=/var/lib/boot2docker/ca.pem
DOCKER_HOST='-H tcp://0.0.0.0:2376'
DOCKER_STORAGE=aufs
DOCKER_TLS=auto
SERVERKEY=/var/lib/boot2docker/server-key.pem
SERVERCERT=/var/lib/boot2docker/server.pem



(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo /etc/init.d/docker start
SSH cmd err, output: <nil>: Device "eth1" does not exist.
Need TLS certs for runner-test,127.0.0.1,172.16.6.10
-------------------
Generate server cert
/usr/local/bin/generate_cert --host=runner-test,127.0.0.1,172.16.6.10 --ca=/var/lib/boot2docker/ca.pem --ca-key=/var/lib/boot2docker/tls/cakey.pem --cert=/var/lib/boot2docker/server.pem --key=/var/lib/boot2docker/server-key.pem --org=Boot2Docker
2017/08/11 15:47:59 Preventing overwrite: the following files already exist: "/var/lib/boot2docker/server.pem" "/var/lib/boot2docker/server-key.pem". To overwrite files, add `--overwrite`.

(runner-test) Calling .GetSSHHostname
(runner-test) Calling .GetSSHPort
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHKeyPath
(runner-test) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /root/.docker/machine/machines/runner-test/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /root/.docker/machine/machines/runner-test/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
netstat -tln
SSH cmd err, output: <nil>: Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 :::22                   :::*                    LISTEN
tcp        0      0 :::2376                 :::*                    LISTEN

(runner-test) Calling .GetIP
Checking connection to Docker...
(runner-test) Calling .GetURL
Reading CA certificate from /root/.docker/machine/certs/ca.pem
Reading client certificate from /root/.docker/machine/certs/cert.pem
Reading client key from /root/.docker/machine/certs/key.pem
Docker is up and running!
Reticulating splines...
(runner-test) Calling .GetConfigRaw
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env runner-test
Making call to close driver server
(runner-test) Calling .Close
Successfully made call to close driver server
Making call to close connection to plugin binary
Making call to close driver server
(flag-lookup) Calling .Close
Successfully made call to close driver server
Making call to close connection to plugin binary

Remove

docker-machine --debug rm runner-test
Docker Machine Version:  0.12.2, build 9371605
About to remove runner-test
WARNING: This action will delete both local reference and remote instance.
Are you sure? (y/n): y
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver vmwarevsphere
Plugin server listening at address 127.0.0.1:34485
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(runner-test) Calling .Remove
(runner-test) DBG | panic: runtime error: invalid memory address or nil pointer dereference
(runner-test) DBG | [signal SIGSEGV: segmentation violation code=0x1 addr=0xa0 pc=0xedd8e6]
(runner-test) DBG |
(runner-test) DBG | goroutine 10 [running]:
(runner-test) DBG | github.com/docker/machine/vendor/github.com/vmware/govmomi/task.(*taskCallback).fn(0xc4202ea400, 0x0, 0x0, 0x0, 0x0)
(runner-test) DBG |     /go/src/github.com/docker/machine/vendor/github.com/vmware/govmomi/task/wait.go:73 +0x296
(runner-test) DBG | github.com/docker/machine/vendor/github.com/vmware/govmomi/task.(*taskCallback).(github.com/docker/machine/vendor/github.com/vmware/govmomi/task.fn)-fm(0x0, 0x0, 0x0, 0x0)
(runner-test) DBG |     /go/src/github.com/docker/machine/vendor/github.com/vmware/govmomi/task/wait.go:120 +0x48
(runner-test) DBG | github.com/docker/machine/vendor/github.com/vmware/govmomi/property.waitLoop(0x1d167c0, 0xc42032afc0, 0xc42011b8c0, 0xc42040b7c8, 0x0, 0x0)
(runner-test) DBG |     /go/src/github.com/docker/machine/vendor/github.com/vmware/govmomi/property/wait.go:140 +0x1b2
(runner-test) DBG | github.com/docker/machine/vendor/github.com/vmware/govmomi/property.Wait(0x1d167c0, 0xc42032afc0, 0xc42040b870, 0xc4202c43f8, 0x4, 0xc4202c44a0, 0xb, 0xc4202c4580, 0x1, 0x1, ...)
(runner-test) DBG |     /go/src/github.com/docker/machine/vendor/github.com/vmware/govmomi/property/wait.go:67 +0x2d7
(runner-test) DBG | github.com/docker/machine/vendor/github.com/vmware/govmomi/task.Wait(0x1d167c0, 0xc42032afc0, 0xc4202c43f8, 0x4, 0xc4202c44a0, 0xb, 0xc42040b870, 0x0, 0x0, 0x0, ...)
(runner-test) DBG |     /go/src/github.com/docker/machine/vendor/github.com/vmware/govmomi/task/wait.go:120 +0x17e
(runner-test) DBG | github.com/docker/machine/vendor/github.com/vmware/govmomi/object.(*Task).WaitForResult(0xc42021a570, 0x1d167c0, 0xc42032afc0, 0x0, 0x0, 0xc42011b260, 0x2e, 0x0)
(runner-test) DBG |     /go/src/github.com/docker/machine/vendor/github.com/vmware/govmomi/object/task.go:51 +0x137
(runner-test) DBG | github.com/docker/machine/vendor/github.com/vmware/govmomi/object.(*Task).Wait(0xc42021a570, 0x1d167c0, 0xc42032afc0, 0xc4202f3130, 0xb)
(runner-test) DBG |     /go/src/github.com/docker/machine/vendor/github.com/vmware/govmomi/object/task.go:45 +0x51
(runner-test) DBG | github.com/docker/machine/drivers/vmwarevsphere.(*Driver).Remove(0xc4202150e0, 0x0, 0x0)
(runner-test) DBG |     /go/src/github.com/docker/machine/drivers/vmwarevsphere/vsphere.go:839 +0x6f2
(runner-test) DBG | github.com/docker/machine/libmachine/drivers/rpc.(*RPCServerDriver).Remove(0xc420121080, 0x1d73f60, 0x1d73f60, 0x0, 0x0)
(runner-test) DBG |     /go/src/github.com/docker/machine/libmachine/drivers/rpc/server_driver.go:205 +0x36
(runner-test) DBG | reflect.Value.call(0xc42012a840, 0xc420122270, 0x13, 0x13dad6e, 0x4, 0xc42003df20, 0x3, 0x3, 0x0, 0x0, ...)
(runner-test) DBG |     /usr/local/go/src/reflect/value.go:434 +0x91f
(runner-test) DBG | reflect.Value.Call(0xc42012a840, 0xc420122270, 0x13, 0xc420028f20, 0x3, 0x3, 0x0, 0x0, 0x0)
(runner-test) DBG |     /usr/local/go/src/reflect/value.go:302 +0xa4
(runner-test) DBG | net/rpc.(*service).call(0xc42011f4c0, 0xc42011e100, 0xc42011d650, 0xc420125a00, 0xc42022c2c0, 0x106e840, 0x1d73f60, 0x16, 0x106e840, 0x1d73f60, ...)
(runner-test) DBG |     /usr/local/go/src/net/rpc/server.go:387 +0x144
(runner-test) DBG | created by net/rpc.(*Server).ServeCodec
(runner-test) DBG |     /usr/local/go/src/net/rpc/server.go:481 +0x404
Error removing host "runner-test": unexpected EOF

andytime on 11 Aug 2017

+1

maxpain on 23 Feb 2018

I am getting the same "dropletID is invalid because cannot be less than 1" issue with docker-machine version 0.13.0, build 9ba6da9.

Happy to post further logs if that would be helpful.

nathanaherne on 13 Mar 2018

Same here. docker-machine version 0.14.0, build 89b8332 in combination with digitalocean driver.

cyclux on 30 Apr 2018

+1

rnwolf on 16 Jan 2019

Hey guys, I have created a workaround around this issue: link
It requires jq on the management droplet and a Digitalocean read-access API token.
I have it in my crontab running every 30 minutes. Hope it helps at least a bit

PingusPepan on 12 Apr 2019

+1
docker-machine version 0.16.1, build cce350d7

zeppelinen on 8 May 2019

+1
Running gitlabi-runner with docker+machine runner type.

docker-machine version 0.16.0, build 702c267f
gitlab-runner version 11.10.1

agalisteo on 3 Jul 2019

+1 docker-machine version 0.16.1, build cce350d7 with digital ocean

awalias on 4 Feb 2020

+1

$ docker-machine --version
docker-machine version 0.16.0, build 702c267f

$ gitlab-runner --version
Version:      12.7.1
Git revision: 003fe500
Git branch:   12-7-stable
GO version:   go1.13.5
Built:        2020-01-23T09:08:54+0000
OS/Arch:      linux/amd64

klutchell on 7 Feb 2020

Got this as well:

$ docker-machine --version
docker-machine version 0.16.0, build 702c267f

And this is my first time using docker-machine - not exactly a great experience so far...

Latest version MacOS. Latest version docker. Latest version docker-machine.

This happened because the process halted half-way through. I can't remember the exact conditions, but it was something like an image not being available, or the wrong digital ocean image (like ubuntu-10-x61) causing it to fail, and then I had to --force.

J7mbo on 17 Mar 2020

Machine: unexpected EOF on remove machine

Most helpful comment

All 23 comments

Create

Remove

Related issues