Vagrant-libvirt: vagrant package won't fork/clone/duplicate/generate a similar vagrant-up-able box/image.

[ribamar-santarosa]

Description

vagrant package won't fork/clone/duplicate/generate a similar vagrant-up-able box/image.

Vagrant version

vagrant -v Vagrant 2.0.1

Host operating system

lsb_release -a Distributor ID: Debian Description: Debian GNU/Linux 9.1 (stretch) Release: 9.1 Codename: stretch

Guest operating system

vagrant ssh -c 'cat /etc/debian_version' 9.1

Vagrantfile

fill vagrant image here:
Vagrant.configure("2") do |config| config.vm.box = "mynewbox" end

Debug output

````
vagrant package --output mynew.box

Total bytes written: 1042032640 (994MiB, 26MiB/s)

==> default: Box created

==> default: You can now add the box:

==> default: vagrant box add mynew.box --name any_comfortable_name

````

Full output of vagrant up at:
https://gist.github.com/ribamar-santarosa/7b98edacaada8fc1892140a2aabca0e1

Full output of ssh to the virtual machine at:
https://gist.github.com/ribamar-santarosa/964a8feda145c3ab9602a6dad0aa4888

Expected behavior

After running:
vagrant package --output mynew.box
I would expect the command bellow to finish someday:
vagrant init mynewbox && vagrant up --provider=libvirt

Actual behavior

vagrant init mynewbox && vagrant up --provider=libvirt
Hangs forever (on up part).

Steps to reproduce

````

Install vagrant. Check https://github.com/hashicorp/vagrant/issues/9285 / https://github.com/ribamar-santarosa/champselysees/tree/master/vagrant-debian-stretch

run any image:

vagrant init debian/stretch64
vagrant up --provider=libvirt
vagrant ssh -c pwd
vagrant ssh -c 'cat /etc/debian_version'

9.1

clone/fork/duplicate that image:

vagrant package --output mynew.box
vagrant box add mynewbox mynew.box

==> default: Box created

==> default: You can now add the box:

==> default: vagrant box add mynew.box --name any_comfortable_name

vagrant box list

...

mynewbox (libvirt, 0)

vagrant destroy default
rm -rf .vagrant Vagrantfile

try to use the new image:

vagrant init mynewbox
VAGRANT_LOG=debug
vagrant up --provider=libvirt # full output at: https://gist.github.com/ribamar-santarosa/7b98edacaada8fc1892140a2aabca0e1

...

DEBUG wait_till_up: Searching for IP for MAC address: 52:54:00:19:76:79

....

==> default: Waiting for domain to get an IP address...

...

INFO wait_till_up: Got IP address 192.168.121.65

...

INFO wait_till_up: Time for getting IP: 15.072947263717651

...

INFO ssh: SSH not up: #

...

DEBUG ssh: == Net-SSH connection debug-level log END ==

INFO ssh: SSH not up: #

...

taking too long to reboot. First, try reloading your machine with

...

````

Further Attempts:

Will update this section as things go on/people comment and I try.

1 - Reload: Failed : Same ssh-forever-waiting error.

vagrant reload # ==> leads same error -- wait forever for ssh

2 - manual ssh: Failed: Same ssh-forever-waiting error.

While vagrant up is running:
vagrant ssh -c pwd # no output (not even error output)

3 - host-passsthrough: Failed: Same ssh-forever-waiting error.

Added to Vagrantfile:
config.vm.provider "libvirt" do |lv| lv.cpu_mode = 'host-passthrough' end

4 - manual non-vagrant ssh: Failed: Connection Reset by peer

While vagrant up is running:
````
sudo nmap 192.168.121.65

Starting Nmap 7.40 ( https://nmap.org ) at 2018-01-12 11:18 CET

Nmap scan report for 192.168.121.65

Host is up (0.000036s latency).

Not shown: 998 closed ports

PORT STATE SERVICE

22/tcp open ssh

111/tcp open rpcbind

MAC Address: 52:54:00:19:76:79 (QEMU virtual NIC)

ssh -v -v -v -v [email protected] # full output at: https://gist.github.com/ribamar-santarosa/964a8feda145c3ab9602a6dad0aa4888

...

debug2: fd 3 setting O_NONBLOCK

debug1: Authenticating to 192.168.121.65:22 as 'root'

debug3: hostkeys_foreach: reading file "/home/ribamar/.ssh/known_hosts"

debug3: send packet: type 20

debug1: SSH2_MSG_KEXINIT sent

Connection reset by 192.168.121.65 port 22

````

References

These issues look somehow related, but none really matched/led to a solution:
https://github.com/hashicorp/vagrant/issues/8157
https://github.com/hashicorp/vagrant/issues/8204
https://github.com/vagrant-libvirt/vagrant-libvirt/issues/510
https://github.com/vagrant-libvirt/vagrant-libvirt/issues/560
https://bugzilla.redhat.com/show_bug.cgi?id=1283989
https://github.com/cloudfoundry/bosh-lite/issues/355
https://github.com/mitchellh/vagrant-aws/issues/3

I also openend a ticket on https://github.com/hashicorp/vagrant/issues/9362

[emielmolenaar]

Same issue here, Ubuntu 17.10 host and Debian 8 guest. In the end, the SSH connection is reset by the guest. I have packaged a custom box based on the debian/jessie64 box.

Interesting fact: when converting a custom-packaged VirtualBox image using https://github.com/sciurus/vagrant-mutate , everything works fine.

[pielgrzym]

So mutating the box from, say, virtualbox image solves the issue?

[pielgrzym]

Same here - debian box. Unfortunately vagrant-mutate trick does not work for me.

[mcandre]

I am getting similar behavior when I try to vagrant up / vagrant ssh to a libvirt box which I just vagrant package'd from a 100% working libvirt base box that I built with Packer. Somehow, vagrant package is messing up downstream boxes so they no longer provide SSH authentication / connectivity. Somehow, the SSH host keys originally available in the parent box, become missing in downstream boxes.

[tjanez]

I recently encountered a similar issue.

I was using a multi-machine Vagrant environment and when I ran vagrant up, I would get errors similar to the one below:

... output trimmed ...

==> validator-2: Creating shared folders metadata...
==> validator-2: Starting domain.
==> validator-2: Waiting for domain to get an IP address...
==> validator-1: Creating shared folders metadata...
==> validator-1: Starting domain.
==> validator-1: Waiting for domain to get an IP address...
==> validator-1: Waiting for SSH to become available...
==> validator-1: Setting hostname...
==> validator-1: Configuring and enabling network interfaces...
==> validator-1: [vagrant-hostmanager:host] Updating hosts file on your workstation (password may be required)...
==> validator-2: Removing domain...
==> validator-2: An error occurred. The error will be shown after all tasks complete.
An error occurred while executing multiple actions in parallel.
Any errors that occurred are shown below.

An unexpected error occurred when executing the action on the
'validator-2' machine. Please report this as a bug:

The specified wait_for timeout (2 seconds) was exceeded

... output trimmed ...

Further debugging showed that the libvirtd's dnsmasq-dhcp server would assign all the machines the same IP!?
It turns out this is because they have the same /etc/macine-id: https://everythingshouldbevirtual.com/virtualization/Ubuntu-18.04-Templates-Duplicate-IPs/.

To fix that, one would need to truncate the /etc/machine-id file when a machine is packaged.

Further investigation showed that vagrant-libvirt calls virt-sysprep when the vagrant package command is called (https://github.com/vagrant-libvirt/vagrant-libvirt#package-box-from-vm).
The virt-sysprep command should remove the machine's ID by default.
However, there is a bug in the virt-sysprep command by which the /etc/machine-id file is re-created by the virt-sysprep's 'customize' operation:
https://bugzilla.redhat.com/show_bug.cgi?id=1557042.

One potentially dangerous thing that vagrant-libvirt does when packaging a machine is that it doesn't shut it down beforehand. A PR to fix this was opened recently.

If you use the latest master branch of vagrant-libvirt, you can set the VAGRANT_LIBVIRT_VIRT_SYSPREP_OPERATIONS environment variable and work-around the /etc/machine-id issue by disabling the customize operation.
I also disabled SSH host keys removal on Ubuntu systems since they are not automatically regenerated during boot.

So the end result would be something like:

export VAGRANT_LIBVIRT_VIRT_SYSPREP_OPERATIONS="defaults,-ssh-userdir,-ssh-hostkeys,-customize"
vagrant package --output package.box

[electrofelix]

Recommend using the following script to allow the ssh host keys be regenerated on first boot of a packaged VM

cat <<EOF > sysprep.sh
#!/bin/sh

# repeat what machine-ids does in sysprep as this script needs to run via customize
# which has a bug resulting in the machine-ids being regenerated

if [ -f /etc/machine-id ]
then
    truncate --size=0 /etc/machine-id
fi

if [ -f /var/lib/dbus/machine-id ]
then
    truncate --size=0 /run/machine-id
fi

# for debian based systems ensure host keys regenerated on boot
if [ -e /usr/sbin/dpkg-reconfigure ]
then
    printf "@reboot root command bash -c 'export PATH=$PATH:/usr/sbin ; export DEBIAN_FRONTEND=noninteractive ; export DEBCONF_NONINTERACTIVE_SEEN=true ; /usr/sbin/dpkg-reconfigure openssh-server &>/dev/null ; /bin/systemctl restart ssh.service ; rm --force /etc/cron.d/keys'\n" > /etc/cron.d/keys
fi
EOF
export VAGRANT_LIBVIRT_VIRT_SYSPREP_OPTIONS="--run $(pwd)/sysprep.sh"

vagrant package

I'd hope this is added to be the default behaviour of vagrant-libvirt in the future.

Note that it is necessary to create the initial machine with config.ssh.insert_key = false set to ensure the default vagrant public ssh key is in place if packaging subsequently, so it might be worthwhile adding default behaviour to re-inject the default SSH key via a script like the above when packaging up a box. Thoughts on that as well?

[skhanpersonal]

I had this issue. What I found: I had to ssh inside the box and run 'sudo shutdown -f now' before calling 'vagrant package' This fixed the issue for me. It seems 'vagrant halt' does not do the trick for the vagrant-libvirt plugin.