Vagrant: Vagrant 1.8.1 cannot SSH to guest OS on Virtualbox 5.0.20

[quater]

Today I have updated Virtualbox from 5.0.18 to 5.0.20. This subsequently caused Vagrant 1.8.1 to fail when trying to SSH to the guest OS.

This problem appears to been introduced by Virtualbox but since Vagrant is showing problems, I am filling this bug report with Vagrant.

NOT Working

Vagrant 1.8.1 in conjunction with Virtualbox 5.0.20
(virtualbox-5.0_5.0.20-106931-Ubuntu-xenial_amd64.deb)

Working

Vagrant 1.8.1 in conjunction with Virtualbox 5.0.18
(virtualbox-5.0_5.0.18-106667-Ubuntu-xenial_amd64.deb)

Virtualbox Host

Ubuntu 16.04 Desktop
4.4.0-22-generic x86_64 GNU/Linux

Virtualbox Guest

Ubuntu 16.04 Server
4.4.0-21-generic x86_64 GNU/Linux

How To Reproduce

1. Have a Ubuntu Xenial Desktop machine with Vagrant 1.8.1 and Virtualbox 5.0.20 installed

2. Create a Ubuntu Xenial Server guest OS by using Vagrant

   Observe this will work correctly. The guest OS is created and it's possible to connect via SSH.

3. Use vagrant to suspend the guest OS

4. Use vagrant to up the guest OS

   Observe this will work correctly too.

5. Use vagrant to SSH to the guest OS

   Observe this works correctly.

6. Use vagrant to suspend the guest OS
7. Reboot the Ubuntu Xenial Desktop machine (i.e. the Virtualbox host machine itself)

8. Use vagrant to up the guest OS

   Observe that Vagrant cannot SSH to the Guest OS.

   Waiting for machine to boot. This may take a few minutes...
   SSH address: 127.0.0.1:2222
   SSH username: vagrant
   SSH auth method: private key
   Warning: Remote connection disconnect. Retrying...
   Warning: Remote connection disconnect. Retrying...
   .
   .
   .
   Timed out while waiting for the machine to boot. This means that
   Vagrant was unable to communicate with the guest machine within
   the configured ("config.vm.boot_timeout" value) time period.
   

   Vagrant Debug Log

https://gist.github.com/quater/9175b8aada54e48af9df7d856c5fe112

Additional Comments

Although I have only tested this with Ubuntu Xenial, I believe this problem is equally observed with other operating systems too.

Workaround

Downgrade Virtualbox from 5.0.20 to 5.0.18.

[sethvargo]

Hi @quater

Since this issue is fixable by downgrading Virtualbox, I'm inclined to believe this is not a Vagrant bug, but a Virtualbox bug instead 😦. Were you able to open an issue on their tracker?

[quater]

Hi @sethvargo

I guess you are right. Looking at it a bit closer I think it is more likely a Virtualbox or even Ubuntu 16.04 related problem. I am using my own Vagrant box (build by using Packer). When I tested with the Ubuntu 16.04 box you provide as _official_ box (i.e. vagrant init ubuntu/xenial64; vagrant up --provider virtualbox), I could successfully SSH into the box, although it took about 30 seconds to actually do it.

Unfortunately your _official_ Ubuntu 16.04 has a couple of other Vagrant related bugs and thus renders it useless to me (i.e. not possible to start more than one instance concurrently due to guest OS name and MAC address conflicts etc.).

This is what I determined in the meantime...

After vagrant suspend and vagrant up, and connecting to the terminal in the foreground, I found that the SSHD daemon is running _seemingly_ correctly.

● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2016-05-12 09:15:04 UTC; 8min ago
 Main PID: 605 (sshd)
    Tasks: 1 (limit: 512)
   CGroup: /system.slice/ssh.service
           └─605 /usr/sbin/sshd -D

May 12 09:15:17 ubuntu sudo[745]: pam_unix(sudo:session): session closed for user root
May 12 09:15:17 ubuntu sudo[754]:  vagrant : TTY=unknown ; PWD=/home/vagrant ; USER=root ; COMMAND=/bin/bash -l
May 12 09:15:17 ubuntu sudo[754]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 12 09:15:17 ubuntu sudo[754]: pam_unix(sudo:session): session closed for user root
May 12 09:15:17 ubuntu sshd[677]: pam_unix(sshd:session): session closed for user vagrant
May 12 09:15:49 ubuntu sshd[760]: Accepted publickey for vagrant from 10.0.2.2 port 50670 ssh2: RSA SHA256:CpFak9M8MbQZmY3bIFBTnt3iepMdMl+YmTCZuVvss14
May 12 09:15:49 ubuntu sshd[760]: pam_unix(sshd:session): session opened for user vagrant by (uid=0)
May 12 09:17:23 ubuntu sshd[771]: Received disconnect from 10.0.2.2 port 50670:11: disconnected by user
May 12 09:17:23 ubuntu sshd[771]: Disconnected from 10.0.2.2 port 50670
May 12 09:17:23 ubuntu sshd[760]: pam_unix(sshd:session): session closed for user vagrant

Furthermore, it's listening correctly.

netstat -tulpn

Proto Recv-Q Send-Q Local Address Foreign Address State
tcp     0           0           0.0.0.0:22        0.0.0.0:*         LISTEN

Though, restarting the SSHD daemon failed with error:

Failed to start sshd.service: The name org.freedesktop.PolicyKit1 was not provided by any .service files. See system logs and 'systemctl status sshd.service'

If I restart the guest OS with sudo reboot, I can then subsequently SSH to the guest OS.

My feeling is that somehow the SSHD daemon breaks while the guest OS state is saved and then started again.

Currently I cannot spend more time on this and for the time being I can work with the workaround (i.e. downgrade Virtualbox). In the light of this, I won't get around following this up with Virtualbox development in the near future. You can close this issue but if other people start to encounter this problem, they have a starting point here.

[Chili-Man]

I'm not sure if this is specific to ubuntu 16.04 guest, but I'm using vagrant 1.8.1 with Virtualbox 5.0.20 for bring up ubuntu 14.04 and its working fine for me and my coworkers

[quater]

I have filed a bug report with Oracle Virtualbox today as the additional testing outlined below ruled Vagrant out as culprit. The same problem is observed with a vanilla Ubuntu 16.04 Server on Virtualbox when NAT is used.

See below for the test details.

First Test

1. Created an OVA export from the Vagrant VM
2. Imported the OVA to create a separate VM
3. Started VM

4. Successfully established SSH connection

   $ ssh -i vagrant-private_key -p 2222 vagrant@localhost

5. Saved the state of the VM by using Virtualbox

   $ VBoxManage controlvm VagrantTesting-Imported savestate

6. Started the VM from the saved state by using Virtualbox

   $ VBoxManage startvm VagrantTesting-Imported

7. Tried to SSH to the VM but failed with error: ssh: connect to host localhost port 2222: Connection refused

   $ ssh -i vagrant-private_key -p 2222 vagrant@localhost

   Note: This VM has only one NIC configured to be NAT. The below test setting indicates that this SSH connection problem is only observed when the NIC is configured as NAT!

Second Test

1. Created an OVA export from the Vagrant VM
2. Imported the OVA to create a separate VM
3. Changed the NIC configuration from "NAT" to "Bridged Adapter"
4. Started VM

5. Successfully established SSH connection

   $ ssh -i vagrant-private_key -p 22 vagrant@<IP Address of VM>

6. Saved the state of the VM by using Virtualbox

   $ VBoxManage controlvm VagrantTesting-Imported savestate

7. Started the VM from the saved state by using Virtualbox

   $ VBoxManage startvm VagrantTesting-Imported

8. Successfully established SSH connection

   $ ssh -i vagrant-private_key -p 22 vagrant@<IP Address of VM>

Third Test

Created a vanilla Ubuntu 16.04 Server VM manually.

1. Downloaded Ubuntu 16.04 Server ISO from http://www.ubuntu.com/download/server
2. Installed Ubuntu 16.04 Server as new VM in Virtualbox
   
   
   
   • Configured VM to have one NIC as NAT
   
   
   • Only installed SSH Server
   
   
   • Did not configure any SSH keys (using username and password only)
   
   
3. Started VM
4. Successfully established SSH connection

$ ssh -p 2222 ubuntu@localhost

1. Saved the state of the VM by using Virtualbox

$ VBoxManage controlvm Ubuntu-Server-16.04 savestate

1. Started the VM from the saved state by using Virtualbox

$ VBoxManage startvm Ubuntu-Server-16.04

1. SSH to the VM but fails with error: ssh: connect to host localhost port 2222: Connection refused

$ ssh -p 2222 ubuntu@localhost

Conclusion

Since the problem is equally observed with Virtualbox 5.0.20 in conjunction with a vanilla Ubuntu 16.04 Server VM, if the NIC is configured as NAT, I have created the Virtualbox bug 15412 https://www.virtualbox.org/ticket/15412.

Virtualbox Bug Reference

Bug: 15412
Name: Virtualbox 5.0.20 Breaks SSH after VM was Saved and Re-Started if NIC is NAT
URL: https://www.virtualbox.org/ticket/15412

[quater]

@Chili-Man @sethvargo It appears that this problem is due to changes with Virtualbox 5.0.20 that concern NAT forwarding rules. Virtualbox 5.0.20 requires a sign of live from the VM after is was started in the shape of a gratuitous ARP request.

Vanilla Ubuntu 14.04 appears to send such requests frequently, which is the reason why this problem is not observed with Ubuntu 14.04. Though, tests showed that it takes almost a minute for Vagrant to be able to connect.

However Ubuntu 16.04 does not appear to send gratuitous ARP requests and thus this problem is observed. Though, this can be worked around by implementing a script that sends ARP requests every so often but that would be far from ideal.

See the https://www.virtualbox.org/ticket/15412 for details.

[quater]

I have now filed an Ubuntu bug concerning the isc-dhclient-4.3.3 package.

Bug: https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/1582163

[DaniloNC]

@quater I have the same problem, let me know if i can help somehow.

Host OS: Ubuntu 16.04
Guest OS: kali linux packed by me (vagrant box danilonc/kalimini)
Vagrant 1.8.1
vboxmanage -V
5.0.20r106931

This will not happen on ArchLinux with the following setup
Host OS: Archlinux
Guest OS: kali linux packed by me (vagrant box danilonc/kalimini)
Vagrant 1.8.1
vboxmanage -V
5.0.20_OSEr106931

[quater]

@DaniloNC

That's interesting that Host OS Archlinux with Guest OS Kali Linux is not having this problem. Methinks that you may potentially have a slightly different problem.

Literally 40 minutes ago, Virtualbox provided a fix to the problem reported here. You can try with their Dev Build and see whether it resolves it for you.

Below is my current conclusion

From what I have established so far, this problem encountered here is a result of two independent changes made in Virtualbox and Ubuntu (i.e. Virtualbox 5.0.20 and Ubuntu 16.04 DHClient).

• Virtualbox 5.0.20 should function in conjunction with Ubuntu 16.04 guest OS, if Ubuntu 16.04 would fire a DHCP lease request after the cable was disconnected and reconnected. This will hopefully be addressed with https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/1582163
• Virtualbox 5.0.20 could be enhanced by performing an initial guest IP address guess (.15) as suggested by Virtualbox developer @vushakov. He has already provided a Dev Build that implements this suggestion and tests showed that it overcomes this problem reported here.

[djcp]

I am seeing similar behavior but with a wrinkle - this is all for 14.04, BTW.

I can bring up a box from a default box from atlas fine and SSH into it. However, if I build a base box in vagrant 1.8.1 / virtualbox 5.0.20 and then try to bring up a box from that new base box, it fails to connect via SSH.

What's weird is base boxes created by an earlier vagrant / virtualbox combo still seem to work fine.

[ethanbeyer]

Glad someone else was having this issue. Downgrading Virtualbox now to test as well!

_EDIT:_ That worked. Virtualbox 5.0.18 allowed me to boot my box without issue.

Thanks again @quater!!!

[sethvargo]

Looks like virtualbox has finally fixed it! Thanks everyone!

[adrianmak]

And the official ubuntu/xenial64 's ssh username is changed to ubuntu instead vagrant

[sethvargo]

Hi @adrianmak

That, unfortunately, is a bug in the box creation process. Vagrant's box-building documentation makes it very clear that the default username should be vagrant. There are ways that you can customize this in your Vagrantfile, but it is not the ideal experience. All boxes should have a default Vagrant user.

[adrianmak]

what is the default password of ubuntu username of ubuntu 16.04 vagrant box ?

[sethvargo]

Hi @quater

The boxes are published by canonical, the makers of Ubuntu. The username should be vagrant and the password should be vagrant. If they are not, that is a bug in the box file and that should be reported to the creators of that box.

[quater]

@sethvargo I guess the previous comment about the default username and password was intended for @adrianmak.

@adrianmak
Note, this issue solely dealt with a SSH connection problem due to Virtualbox not being able to determine the NAT IP and port of the VM.

Any other SSH connection problem observed would need to be dealt with through another issue.

[quater]

@sethvargo The fix for this issue has been included in Virtualbox 5.0.22.

https://www.virtualbox.org/wiki/Changelog