Machine: CoreOS instance stops during amazon EC2 provisioning
I am trying to use the amazoneec2 driver with coreos AMIs. When I do the instance stops (according to the AWS console) and docker-machine is unable to make an ssh connection as a result. If I manually create an instance with ami-ee527484 through the AWS console the instance stays up and I can ssh in.
docker-machine --debug create --amazonec2-security-group "docker-machine" --amazonec2-instance-type "t2.micro" --swarm-master --driver "amazonec2" --amazonec2-ssh-user "core" --swarm-discovery "etcd://10.0.2.8:2379,10.0.3.117:2379/swarm" --amazonec2-region "us-east-1" --amazonec2-zone "b" --amazonec2-secret-key "XXXXX" --amazonec2-ami "ami-ee527484" --swarm --amazonec2-access-key "XXXX" --amazonec2-vpc-id "vpc-cf90edab" swarm-master
Docker Machine Version: 0.5.3, build 4d39a66
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver amazonec2
Plugin server listening at address 127.0.0.1:51675
() Calling .GetVersion
Using API Version 1
() Calling .SetConfigRaw
() Calling .GetMachineName
(flag-lookup) Calling .GetCreateFlags
Making call to close connection to plugin binary
Making call to close driver server
(flag-lookup) Calling .Close
Successfully made call to close driver server
Found binary path at /usr/local/bin/docker-machine
Launching plugin server for driver amazonec2
Plugin server listening at address 127.0.0.1:34798
() Calling .GetVersion
Using API Version 1
() Calling .SetConfigRaw
() Calling .GetMachineName
(swarm-master) Calling .GetMachineName
(swarm-master) Calling .DriverName
(swarm-master) Calling .GetCreateFlags
(swarm-master) Calling .SetConfigFromFlags
Running pre-create checks...
(swarm-master) Calling .PreCreateCheck
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeKeyPairs"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeSubnets"
(swarm-master) DBG | ],
(swarm-master) DBG | "Filter.1.Name": [
(swarm-master) DBG | "availabilityZone"
(swarm-master) DBG | ],
(swarm-master) DBG | "Filter.1.Value": [
(swarm-master) DBG | "us-east-1b"
(swarm-master) DBG | ],
(swarm-master) DBG | "Filter.2.Name": [
(swarm-master) DBG | "vpc-id"
(swarm-master) DBG | ],
(swarm-master) DBG | "Filter.2.Value": [
(swarm-master) DBG | "vpc-cf90edab"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) Calling .GetConfigRaw
Creating machine...
(swarm-master) Calling .Create
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeKeyPairs"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) Launching instance...
(swarm-master) DBG | creating key pair: swarm-master
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "ImportKeyPair"
(swarm-master) DBG | ],
(swarm-master) DBG | "KeyName": [
(swarm-master) DBG | "swarm-master"
(swarm-master) DBG | ],
(swarm-master) DBG | "PublicKeyMaterial": [
(swarm-master) DBG | "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFETlZhSTFUNzZUZk5HWFNHMkZBWnlYcm05OHRsUkI3OWJDZ3MvL3VHcWN4TnBxUTRZSGhrcEE3NkZ6aU9abFV5cmRXQkhLTUJ2T002SjUvSGt4ZHhwYWQ3M3hwNmFrUkhwUTRwNGVBeXFaUTF5WUhMdmkrbXVaTVljSVZrZ2dFNFVIYmsxSVRwRnJPYi93TkxNQWsxR01mK3l5MnFXM0w1R094MDlWeGJMaGRxQU0vMGJBeTd1V0hDUm1Zc1gzTm9VVzZieTdXVzB0ZE5mcHZBUE5iT1ZFYVZUSUp0cmRaREZuMkYyTU1FY3U0NlU4M2xrTUwrUDM0endjU0FhS3p4VlVLcVZkRjdobTJyckJrSTR5bDkyYUw3NzZKN3NPODBVWGFmTHBUWHhoOHNRdC84MWIrUWx4UzMyTis4OVh5ZTVkSzF2R0J3TTJPVGcxYzJvVWpoODcK"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | configuring security group in vpc-cf90edab
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeSecurityGroups"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | found existing security group (docker-machine) in vpc-cf90edab
(swarm-master) DBG | configuring security group authorization for 0.0.0.0/0
(swarm-master) DBG | launching instance in subnet subnet-d78578fd
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "RunInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "BlockDeviceMapping.0.DeviceName": [
(swarm-master) DBG | "/dev/sda1"
(swarm-master) DBG | ],
(swarm-master) DBG | "BlockDeviceMapping.0.Ebs.DeleteOnTermination": [
(swarm-master) DBG | "1"
(swarm-master) DBG | ],
(swarm-master) DBG | "BlockDeviceMapping.0.Ebs.VolumeSize": [
(swarm-master) DBG | "16"
(swarm-master) DBG | ],
(swarm-master) DBG | "BlockDeviceMapping.0.Ebs.VolumeType": [
(swarm-master) DBG | "gp2"
(swarm-master) DBG | ],
(swarm-master) DBG | "BlockDeviceMapping.0.VirtualName": [
(swarm-master) DBG | ""
(swarm-master) DBG | ],
(swarm-master) DBG | "ImageId": [
(swarm-master) DBG | "ami-ee527484"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceType": [
(swarm-master) DBG | "t2.micro"
(swarm-master) DBG | ],
(swarm-master) DBG | "KeyName": [
(swarm-master) DBG | "swarm-master"
(swarm-master) DBG | ],
(swarm-master) DBG | "MaxCount": [
(swarm-master) DBG | "1"
(swarm-master) DBG | ],
(swarm-master) DBG | "MinCount": [
(swarm-master) DBG | "1"
(swarm-master) DBG | ],
(swarm-master) DBG | "Monitoring.Enabled": [
(swarm-master) DBG | "0"
(swarm-master) DBG | ],
(swarm-master) DBG | "NetworkInterface.0.AssociatePublicIpAddress": [
(swarm-master) DBG | "1"
(swarm-master) DBG | ],
(swarm-master) DBG | "NetworkInterface.0.DeviceIndex": [
(swarm-master) DBG | "0"
(swarm-master) DBG | ],
(swarm-master) DBG | "NetworkInterface.0.SecurityGroupId.0": [
(swarm-master) DBG | "sg-db0b62a2"
(swarm-master) DBG | ],
(swarm-master) DBG | "NetworkInterface.0.SubnetId": [
(swarm-master) DBG | "subnet-d78578fd"
(swarm-master) DBG | ],
(swarm-master) DBG | "Placement.AvailabilityZone": [
(swarm-master) DBG | "us-east-1b"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | waiting for ip address to become available
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | Got the IP Address, it's "54.173.253.143"
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) DBG | created instance ID i-acc7d82d, IP address 54.173.253.143, Private IP address 10.0.2.155
(swarm-master) DBG | Settings tags for instance
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "CreateTags"
(swarm-master) DBG | ],
(swarm-master) DBG | "ResourceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Tag.1.Key": [
(swarm-master) DBG | "Name"
(swarm-master) DBG | ],
(swarm-master) DBG | "Tag.1.Value": [
(swarm-master) DBG | "swarm-master"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) Calling .GetConfigRaw
(swarm-master) Calling .DriverName
Waiting for machine to be running, this may take a few minutes...
(swarm-master) Calling .GetState
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
Machine is running, waiting for SSH to be available...
Getting to WaitForSSH function...
(swarm-master) Calling .GetSSHHostname
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) Calling .GetSSHPort
(swarm-master) Calling .GetSSHKeyPath
(swarm-master) Calling .GetSSHKeyPath
(swarm-master) Calling .GetSSHUsername
Using SSH client type: external
About to run SSH command:
exit 0
SSH cmd err, output: exit status 255:
Error getting ssh command 'exit 0' : Something went wrong running an SSH command!
command : exit 0
err : exit status 255
output :
Getting to WaitForSSH function...
(swarm-master) Calling .GetSSHHostname
(swarm-master) DBG | Making AWS API call with values:
(swarm-master) DBG | {
(swarm-master) DBG | "Action": [
(swarm-master) DBG | "DescribeInstances"
(swarm-master) DBG | ],
(swarm-master) DBG | "InstanceId.1": [
(swarm-master) DBG | "i-acc7d82d"
(swarm-master) DBG | ],
(swarm-master) DBG | "Version": [
(swarm-master) DBG | "2014-06-15"
(swarm-master) DBG | ]
(swarm-master) DBG | }
(swarm-master) Calling .GetSSHPort
(swarm-master) Calling .GetSSHKeyPath
(swarm-master) Calling .GetSSHKeyPath
(swarm-master) Calling .GetSSHUsername
Using SSH client type: external
About to run SSH command:
exit 0
SSH cmd err, output: exit status 255:
Error getting ssh command 'exit 0' : Something went wrong running an SSH command!
command : exit 0
err : exit status 255
output :
[... lots of ssh retries]
notifying bugsnag: [Error waiting for SSH: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded]
Error creating machine: Error waiting for SSH: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded
When inspecting the instance that shutdown AWS states the following:
State transition reason: Client.InstanceInitiatedShutdown: Instance initiated shutdown
driveec2
zbyte64
All 8 comments
Still an issue with docker-machine 0.6.0
zbyte64
on 5 Feb 2016
I have the same problem on EC2 with a Debian Jessie AMI on 0.6.0.
fabiomargarido
on 23 Feb 2016
Same issue :/ with ubuntu-wily-15.10-amd64-server-20151116.1 and lastest docker-machine:
docker-machine version 0.6.0, build e27fb87
SSH is not accessible.
ebuildy
on 5 Mar 2016
I'm also seeing this when trying to use AWS CoreOS AMI ami-7a627510. I've tried about 5 times now with same result.
The log shows this when running --debug with docker-machine
<stateReason>
<code>Client.InstanceInitiatedShutdown</code>
<message>Client.InstanceInitiatedShutdown: Instance initiated shutdown</message>
</stateReason>
docker-machine version 0.6.0, build e27fb87
Kuberchaun
on 19 Apr 2016
CoreOS HVM AMIs have /dev/xvda as root device, which doesn't match with the default in the EC2 machine provider (/dev/sda, which happens to be the right name for CoreOS PV AMIs).
Adding --amazonec2-device-name /dev/xvda when a HVM AMI is used did the trick for me.
bjaglin
on 27 Apr 2016
🎉3
I still consider this as a workaround though, the manual actions that led me to that conclusion could/should be automated: the default root device name should IMHO be inferred from the AMI instead of having a static default. WDYT?
bjaglin
on 27 Apr 2016
duplicate of https://github.com/docker/machine/issues/3293 (last message there is the same feature request as my last post)
bjaglin
on 28 Apr 2016
This ticket can be closed @zbyte64 ?
vikas027
on 21 Dec 2018
Was this page helpful?
0 / 5 - 0 ratings
Related issues
m-amr
路
5Comments
AlexZeitler
路
4Comments
iongion
路
4Comments
moander
路
5Comments
pschultz
路
3Comments
Most helpful comment
CoreOS HVM AMIs have
/dev/xvdaas root device, which doesn't match with the default in the EC2 machine provider (/dev/sda, which happens to be the right name for CoreOS PV AMIs).Adding
--amazonec2-device-name /dev/xvdawhen a HVM AMI is used did the trick for me.