vagrant up failing for new VM if port 2222 is already taken

[MrTrustor]

Hello,

I have a rather annoying issue where "vagrant up" fails when I am trying to boot up a new VM if another one is already running.
While booting the VM, vagrant detects that there is a port collision on port 2222 for SSH (which is normal, because port 2222 is already taken by another VM) and fixes it. So far, so good.
However, when vagrant is trying to SSH in the new VM to setup a few things (hostname, etc.), it still uses the port 2222, and fails.
Here is an example:

kitchen-dev-01:~/test$ vagrant init dev-01 /srv/vagrant/package_ref.box
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
kitchen-dev-01:~/test$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'dev-01' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Adding box 'dev-01' (v0) for provider: virtualbox
    default: Downloading: file:///srv/vagrant/package_ref.box
==> default: Successfully added box 'dev-01' (v0) for 'virtualbox'!
==> default: Importing base box 'dev-01'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: test_default_1425926412506_64253
==> default: Fixed port collision for 22 => 2222. Now on port 2204.
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 => 2204 (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Authentication failure. Retrying...
    default: Warning: Authentication failure. Retrying...
    default: Warning: Authentication failure. Retrying...
    default: Warning: Authentication failure. Retrying...
    default: Warning: Authentication failure. Retrying...
    default: Warning: Authentication failure. Retrying...
    default: Warning: Authentication failure. Retrying...
    default: Warning: Authentication failure. Retrying...
    default: Warning: Authentication failure. Retrying...
    default: Warning: Authentication failure. Retrying...
    default: Warning: Authentication failure. Retrying...
^C==> default: Waiting for cleanup before exiting...
Vagrant exited after cleanup due to external interrupt.
kitchen-dev-01:~/test$

When I am using vagrant manually, this is only annoying: a vagrant reload fixes the problem. But I need to boot up vagrant VMs through Jenkins and this problem is blocking me. Is there something I missed? Or is it really a bug?

Here is my config:

kitchen-dev-01:~/test$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 7.6 (wheezy)
Release:    7.6
Codename:   wheezy
kitchen-dev-01:~/test$ dpkg -l | grep vagrant
ii  vagrant                            1:1.7.2                       amd64        no description given
kitchen-dev-01:~/test$ dpkg -l | grep Virtual
ii  virtualbox-4.3                     4.3.14-95030~Debian~wheezy    amd64        Oracle VM VirtualBox

Thank you

[semmerson]

I have the same problem with Vagrant 1.7.2 on a Linux 3.6.11 host using VirtualBox 4.2.18.

Here's the workaround in my script:

flock . vagrant up $vmName
trap "vagrant destroy --force $vmName; `trap -p EXIT`" EXIT

# Workaround for concurrent VMs
vagrant ssh -c date $vmName || flock . vagrant reload --provision $vmName

# On the virtual machine:
vagrant ssh $vmName -- -T <<EOF
    ...
EOF

[derek-adair]

Also seeing this, only destroying the vagrant instance seems to work.

[semmerson]

@derek-adair Odd. vagrant reload --provision seems to work for me.

Are you executing the Vagrant commands for the multiple VMs in the same directory?

[MrTrustor]

@semmerson I tried your script, but I have the same problem: vagrant up never returns and is stuck trying to SSH in the VM with either a timeout or an authentication failure. I guess I could hack my way around by launching vagrant up in the background, getting its PID, waiting a while, killing it and then doing a vagrant reload but that seems overkill...

[semmerson]

@MrTrustor I forgot to mention that I also explicitly set the forwarded ssh(1) ports in Vagrantfile:

  config.vm.define "centos64_64" do |centos64_64|
    centos64_64.vm.network "forwarded_port", guest: 22, host: 2200
    ...
  end

  config.vm.define "ubuntu12_32" do |ubuntu12_32|
    ubuntu12_32.vm.network "forwarded_port", guest: 22, host: 2201
    ....
  end

Sorry about that. Give it a try.

[MrTrustor]

@semmerson Well, the server on which I am working can have any number of VMs running at the same time, by different users. Even Jenkins could have different jobs with different VMs running. So Jenkins would have to determine a free port, generate the Vagrantfile and then play with vagrant up and reload. Maybe I'll try that...

If anyone is interested, here is a rough script that does the background and killing thing.

#!/bin/bash

vagrant init dev-01 /srv/vagrant/package_ref.box
vagrant up > up.log 2>&1 &
UP_PID=$!

echo "Vagrant up launched with PID $UP_PID"
while ! grep "SSH auth method: private key" up.log; do
        sleep 1
done

echo "VM has booted, killing vagrant up."
pkill -P $UP_PID
vagrant reload --provision default
vagrant ssh -c hostname default
rm up.log

[semmerson]

@MrTrustor You might try adding port collision auto-correction:

centos64_64.vm.network "forwarded_port", guest: 22, host: 2200, auto_correct: true

See this Vagrant webpage on port forwarding.

[MrTrustor]

Well, in the end, it was not a vagrant issue. The person who created our reference box hardcoded the following: <Forwarding name="SSH" proto="1" hostport="2222" guestport="22"/> in the .ovf file.
Thank you again.

[semmerson]

@MrTrustor Huh. I don't have any entries with Forwarding or SSH (case insensitive) in my .ovf files -- so the cause of the problem must be different for me.

The problem has started-up again -- so my workaround script doesn't, apparently. :-(

[ProxiBlue]

I recently also experienced this issue.

I manually bring up multiple 'machines' in the same folder, so not sure if that is the issue, using docker as my provider.
a workaround I have found is to set a random port, as defined by a random number generated.
Not had this issue since. (granted I could get the same random number twice, but not happened yet)

# Generate a random port number
r = Random.new
ssh_port = r.rand(1000...5000)
config.vm.network "forwarded_port", guest: 22, host: "#{ssh_port}", id: 'ssh', auto_correct: true