Vagrant: Port collision detection broken under VMware Fusion 8.5.1

[sirn]

Vagrant version

$ vagrant -v
Vagrant 1.8.6

$ vagrant plugin list
vagrant-scp (0.5.7)
vagrant-share (1.1.5, system)
vagrant-vmware-fusion (4.0.14)

Host operating system

macOS Sierra 10.12.1

Guest operating system

Ubuntu 12.04 (VMware Fusion 8.5.1)

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "hashicorp/precise64"
  config.vm.network "forwarded_port", guest: 80, host: 8080
end

Debug output

https://gist.github.com/sirn/b885d89d02ec1b426b91beb35a65d34f

Expected behavior

No port collision.

Actual behavior

Vagrant detected port collision.

Steps to reproduce

1. Run vagrant up with the provided Vagrantfile, no port collision should happen in this step.
2. Run vagrant halt to shutdown the machine.
3. Run vagrant up again.

---

In /Library/Preferences/VMware Fusion/vmnet8/nat.conf, this is the [incomingtcp] section after the first vagrant up run (port 8080 successfully forwarded):

[incomingtcp]

# Use these with care - anyone can enter into your VM through these...
# The format and example are as follows:
#<external port number> = <VM's IP address>:<VM's port number>
#8080 = 172.16.3.128:80
# VAGRANT-BEGIN: /Users/sirn/Desktop/foobar/.vagrant/machines/default/vmware_fusion/cb8daaaa-02d8-4707-9708-e8c5e30d17dd/precise64.vmx
8080 = 192.168.229.128:80
2222 = 192.168.229.128:22
# VAGRANT-END: /Users/sirn/Desktop/foobar/.vagrant/machines/default/vmware_fusion/cb8daaaa-02d8-4707-9708-e8c5e30d17dd/precise64.vmx

Then the [incomingtcp] section after the second vagrant up that resulted in an error:

[incomingtcp]

# Use these with care - anyone can enter into your VM through these...
# The format and example are as follows:
#<external port number> = <VM's IP address>:<VM's port number>
#8080 = 172.16.3.128:80
2222 = 192.168.229.128:22
8080 = 192.168.229.128:80

I've tried nuking /Library/Preferences/VMware Fusion/networking* and /Library/Preferences/VMware Fusion/vmnet* and re-run VMware Fusion to have it populate the configuration prior to running, but still the same result.

I can also reproduce this issue in my other machine running exactly the same setup as well.

[vdanen]

I'm seeing the exact same issue here as well.

[rphillips]

Same here as well

[mungler]

Me too. Preventing me from working, currently.

[mungler]

I managed to recover from this by downloading VMWare Fusion 8.5.0 and installing it over the top of 8.5.1, then nuking the .vagrant directory from my project. Upon vagrant up, the box reprovisioned and the port-forwarding worked fine.

[mungler]

Actually, I had to completely remove VMWare, including all the files mentioned here: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017838

Then I rebooted, installed 8.5.0 cleanly (needed to enter my license code etc again) and finally set Vagrant back up. Things are finally back to a working condition.

[sirn]

If anyone has this problem and do not mind countless entries in VMware's nat.conf file, you can actually add auto_correct: true and _use the same port as before_ (instead of the corrected port).

The problem seems to be that either Vagrant or VMware couldn't properly cleaned up the port forwarding entries in nat.conf file, so the staled entries remained there, causing Vagrant to incorrectly detect port collision. But since the staled entries actually pointing to the VM we want to connect to, we can actually use those port as if it were successfully forwarded.

[changx]

Not only nat.conf, look at your vmnet8/../networking , lots of add_nat_portfwd items here too.

[chrisroberts]

Just for reference as I'm trying to get a reproduction of this behavior locally: did this occur after an upgrade to either fusion and/or the vagrant-vmware plugin? If so, can you provide what the previous versions were prior to upgrading? Thanks!

[mungler]

Hi @chrisroberts - for me, everything was working fine on Fusion 8.5.0 - I upgraded to 8.5.1 and the issues began. Blowing away Fusion entirely (plus all supporting files) then reinstalling from an 8.5.0 download and readding my licence key fixed the problem for me, so I definitely think its a change in 8.5.1 which broke it. For reference, i'm on macOS Sierra 10.12.1, and my vagrant plugin version is 4.0.11

[chrisroberts]

Okay, great. Thanks!

[sirn]

Yes, me is the same. It broke after upgrading VMware Fusion to 8.5.1. I reverted VMware Fusion to 8.5.0 and everything back to working normally.

[chrisroberts]

Still not having any luck reproducing the behavior. Currently using:

• OS: macOS Sierra
• Fusion: 8.5.1
• vagrant 1.8.6 (and 1.8.7)
• vagrant-vmware-fusion: 4.0.14

This is a fresh VM, so I'm wondering if perhaps there's other content in the section that is causing problems? Do either of your [incomingtcp] sections contain any other entries (from vagrant or otherwise)?

[chrisroberts]

If it is easily reproducible, can you start with a fresh nat.conf file, and run up, halt, and up with --debug on the commands and provide gists of each of them? That might be the fastest way to identify the problem.

[sirn]

My [incomingtcp] section do not contain any extra entries. (As mentioned in the original post, I did nuke the whole networking configuration and start anew.) I'll try to reproduce it and get back the debug log. Meanwhile, this is the gist for the second up: https://gist.github.com/sirn/b885d89d02ec1b426b91beb35a65d34f

[chrisroberts]

@sirn The debug output from the first up and the halt commands would be really helpful as the final up (which you provided the debug output for) shows the [incomingtcp] section being read and shows no vagrant comment markers. I'm hoping to be able to identify where they are being lost in one of the first two commands run. Thanks!

[zienowicz]

Adding a "Me Too" for this issue in case any data from my configuration would be useful. Same environment as everyone else having the problem (Sierra, latest Vagrant and plugin). In my case the affected machines are Laravel Homestead images that forward a bunch of standard web dev ports (22, 80, 443, 3306, and 5432).

Until recently a vagrant reload fixed the problem some of the time, but that trick seems to have stopped working. I worked around it yesterday by specifying forwarded port numbers rather than letting it detect automatically.

[iamlittle]

Seeing this as well.

• Fusion: 8.5.1
• vagrant: 1.8.7
• vagrant-vmware-fusion: 4.0.14

[chrisroberts]

For anyone adding a "me too" on this issue, I need some more information to help determine what is causing the issue. Please see this comment: https://github.com/mitchellh/vagrant/issues/7948#issuecomment-259966457

[sirn]

@chrisroberts sorry that it took so long, got few stuff to finish that I can't nuke my environment.

Debug logs:

• debug.001_vagrant_up.log — first vagrant up
• debug.002_vagrant_halt.log — vagrant halt
• debug.003_vagrant_up.log — second vagrant up

Networking logs (hopefully I did not commit any sensitive information here):

https://github.com/sirn/vagrant-vmware-networking-logs/commits/master

Also for a reference, I'm using VMware Fusion 8.5.1 Pro.

[zienowicz]

@chrisroberts here is a log from vagrant up

https://gist.github.com/zienowicz/ef14d16175f6c6f1389b5541e52b2682

[ernestonakamura]

I had the same issue with VMware Fusion 8.5.1, but after upgrading to 8.5.2 it started to work again.
Here are my working versions:

• VMware Fusion: 8.5.2
• vagrant: 1.8.4
• vagrant-vmware-fusion: 4.0.9

[zienowicz]

Simply upgrading to Fusion 8.5.2 didn't work for me. I also had to remove a bunch of cruft from the associated nat.conf file. There were a number of entries under [incomingtcp] that were not between VAGRANT-BEGIN and VAGRANT-END comment blocks. See this gist for an excerpt of nat.conf: https://gist.github.com/zienowicz/d414408d2e348b5bb7616fd118339e1d

Removing everything under [incomingtcp] and then running vagrant up a couple of times seems to have fixed nat.conf and allowed the vagrant box to start and forward ports properly. I regularly run three or four different boxes on this machine, so I'll exercise them a bit and report back if I run into additional problems.

[zienowicz]

Nope. Now I'm getting the dreaded "The VMware 'vmnet' devices are failing to start" message. The box is running, and the ports seem to be forwarding, but the shared folder won't mount. My nat.conf has all those (what I thought were) extraneous entries again, plus the ones clearly added by the Vagrant plugin.

[zienowicz]

And now we're back to the port collision error (upon subsequent vagrant reload attempts). Please let me know if there is any additional diagnostic info I can provide.

[chrisroberts]

@sirn Thank you for providing all those logs! While I cannot get this behavior to reproduce on its own, I was able to identify an anomaly within your log files that I was not experiencing. On the second up command, this shows up in the logs:

Hostonly virtual adapter on vmnet1 is disabled
DHCP service on vmnet8 is not running
NAT service on vmnet8 is not running
Hostonly virtual adapter on vmnet8 is disabled
DEBUG subprocess: stderr: DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 1
DEBUG subprocess: stderr:  INFO vmware_driver: NAT config changed but isn't running, so not restarting.

Just after this, the vmnet services are started. To simulate this, I halted, ran vmnet-cli --stop, then uped. Inspecting the nat.conf file I see:

[incomingtcp]

# Use these with care - anyone can enter into your VM through these...
# The format and example are as follows:
# <external port number> = <VM's IP address>:<VM's port number>
#8080 = 172.16.3.128:80
2222 = 172.16.72.131:22
8080 = 172.16.72.131:80

When vmnet is started back up, it's re-adding these entries, even though they were just removed. And now that the vagrant markers are gone, vagrant will not remove them.

Now that I can reliably reproduce this, I'm working on adding some guards to prevent this behavior.

[zienowicz]

I had temporarily given up on this and switched to a different machine so I could get work done. Circling back to this issue, I noticed that /Library/Preferences/VMware Fusion/networking contained what appeared to be extraneous port forwarding entries. I cleared those out, made sure that nat.conf had also been cleared of port forwarding entries, halted the Vagrant boxes (which had been suspended), and rebooted. (I did also upgrade Vagrant from 1.8.6 to 1.8.7 before I started.) I have so far successfully started and suspended my Vagrant boxes several times, so fingers crossed that I'm out of the woods. Maybe the networking file also needs some protection against wayward entries?

[vdanen]

I had cleaned things up before and just now halted my vm, upgraded to vagrant 1.8.7 and vmware fusion pro 8.5.2 and now it's back again. The error looks different than last time though:

vdanen@sif:~/git/otter|master⚡ ⇒  vagrant up
Bringing machine 'otter' up with 'vmware_fusion' provider...
==> otter: Checking if box 'puppetlabs/centos-7.0-64-nocm' is up to date...
==> otter: There was a problem while downloading the metadata for your box
==> otter: to check for updates. This is not an error, since it is usually due
==> otter: to temporary network problems. This is just a warning. The problem
==> otter: encountered was:
==> otter:
==> otter:
==> otter:
==> otter: If you want to check for box updates, verify your network connection
==> otter: is valid and try again.
==> otter: Skipping vmnet device verification, verify_vmnet is set to false.
==> otter: Preparing network adapters...
Vagrant cannot forward the specified ports on this VM, since they
would collide with some other application that is already listening
on these ports. The forwarded port to 5000 is already in use
on the host machine.

To fix this, modify your current project's Vagrantfile to use another
port. Example, where '1234' would be replaced by a unique host port:

  config.vm.network :forwarded_port, guest: 80, host: 1234

Sometimes, Vagrant will attempt to auto-correct this for you. In this
case, Vagrant was unable to. This is usually because the guest machine
is in a state which doesn't allow modifying port forwarding. You could
try 'vagrant reload' (equivalent of running a halt followed by an up)
so vagrant can attempt to auto-correct this upon booting. Be warned
that any unsaved work might be lost.

Debug output is here: https://gist.github.com/vdanen/95c69e4ae6844ad8f391b8ca4dc9d797

[vdanen]

Once I cleaned up the networking file and vmnet8/nat.conf I could get 'vagrant up' to proceed further but there's still something funky going on in that now it reports this:

vdanen@sif:~/git/otter|master⚡ ⇒  vagrant reload
==> otter: Attempting graceful shutdown of VM...
==> otter: Checking if box 'puppetlabs/centos-7.0-64-nocm' is up to date...
==> otter: There was a problem while downloading the metadata for your box
==> otter: to check for updates. This is not an error, since it is usually due
==> otter: to temporary network problems. This is just a warning. The problem
==> otter: encountered was:
==> otter:
==> otter:
==> otter:
==> otter: If you want to check for box updates, verify your network connection
==> otter: is valid and try again.
==> otter: Skipping vmnet device verification, verify_vmnet is set to false.
==> otter: Preparing network adapters...
==> otter: Starting the VMware VM...
==> otter: Waiting for machine to boot. This may take a few minutes...
    otter: SSH address: 172.16.154.159:22
    otter: SSH username: vagrant
    otter: SSH auth method: private key
    otter: Warning: Connection refused. Retrying...
==> otter: Machine booted and ready!
==> otter: Forwarding ports...
    otter: -- 80 => 5000
    otter: -- 3306 => 3306
    otter: -- 22 => 2222
==> otter: Configuring network adapters within the VM...
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

# Restart network
service network restart


Stdout from the command:

Restarting network (via systemctl):  [FAILED]


Stderr from the command:

Job for network.service failed. See 'systemctl status network.service' and 'journalctl -xn' for details.

I have no idea what this means or why the network is not coming up. Going to try rebooting again to see if that helps.

[vdanen]

Well this is disappointing:

==> otter: Forwarding ports...
    otter: -- 80 => 5000
    otter: -- 3306 => 3306
    otter: -- 22 => 2222
failed MSpanList_Insert 0x19b1a0 0x12f75cf22b 0x0
fatal error: MSpanList_Insert

I'm going to leave this today; if you need/want some info please let me know so I can provide it otherwise I'll back this out tomorrow so I can actually get some work done.

[chrisroberts]

This issue has been fantastically frustrating to reproduce. I was able to get an installation to reliably generate previously removed entries in the configuration file, but after rolling back to a fresh setup was unable to again reproduce the issue.

@vdanen If you could just toss your Vagrantfile in that you're using, that would be helpful too. Thanks!

[chrisroberts]

Oh, and one other line I'm following: Is your file system configured as case sensitive or case insensitive?

[vdanen]

Case-insensitive filesystem (so the mac default) and the Vagrantfile can be seen at https://gist.github.com/vdanen/ed8ec7c0be8ec844dc20158045638404

[darthsteven]

I had this issue too, I edited:

/Library/Preferences/VMware Fusion/vmnet8/nat.conf

and commented out all the [incomingtcp] as someone here has suggested, and the machine was able to boot.

On looking at the file now, the vagrant comment markers are back, and the port forwards are within them.

Guess if anything else read this file and wrote it without preserving comments, then we'd see this behavior. Has VMWare just started doing this or something?

[vdanen]

Looks like VMware Fusion 8.5.3 is out now.. has anyone tried it to see if this is fixed?

[f1sherman]

I'm seeing the same issue on Fusion 8.5.3 (Vagrant version 1.9.0, vagrant-vmware-fusion version 4.0.15)

[vincenzoraco]

I confirm. Just updated VMWare Fusion and I have the same issue with the version 8.5.3

[jamsilver]

I updated to 8.5.3 today and saw this issue. I deleted the lines in nat.conf as @darthsteven suggested in https://github.com/mitchellh/vagrant/issues/7948#issuecomment-264835320 and it now works fine.

Perhaps something in VMWare processed this file and rewrote it without vagrant's comments when upgrading?

[ebenoist]

Deleting the lines in nat.conf in addition to killing the networking file is what did the trick for me.

[stevenorell]

Having this issue as well with VMWare Fusion 8.5.3. Deleting lines in nat.conf fixes the issue, but it appears that change gets overridden on restart.

[cswarth]

FWIW Still seeing this issue on a fresh install today.

==> foo: Preparing network adapters...
Vagrant cannot forward the specified ports on this VM, since they
would collide with some other application that is already listening
on these ports. The forwarded port to 2001 is already in use
on the host machine.

To fix this, modify your current project's Vagrantfile to use another
port. Example, where '1234' would be replaced by a unique host port:

  config.vm.network :forwarded_port, guest: 2001, host: 1234

Sometimes, Vagrant will attempt to auto-correct this for you. In this
case, Vagrant was unable to. This is usually because the guest machine
is in a state which doesn't allow modifying port forwarding. You could
try 'vagrant reload' (equivalent of running a halt followed by an up)
so vagrant can attempt to auto-correct this upon booting. Be warned
that any unsaved work might be lost.

$ vagrant --version
Vagrant 1.9.1

$ vagrant plugin list
vagrant-share (1.1.6, system)
vagrant-triggers (0.5.3)
vagrant-vmware-fusion (4.0.15)

Vmware fusion
Version 8.5.3 (4696910)

[vincenzoraco]

I actually reinstall macOS Sierra from scratch and it solved the issue, but we are supposed to do it - we can't simply for an issue reinstall the os.

[valentinhocher]

The newest macOS (10.12.3) fixed this for me

[darthsteven]

@VH0815 interesting, do you know what that has changed in the nat.conf file? Was it just completely cleaned out maybe?

@vincenzoraco I'm fairly certain that you can just edit the nat.conf as described in https://github.com/mitchellh/vagrant/issues/7948#issuecomment-264835320 and it'll work, rather than re-installing.

[valentinhocher]

@darthsteven Not really, sorry. Noticed this more on accident by powering an old machine up. Looks like all entries under incomingtcp got wiped and set to the correct mappings.

[vdanen]

I upgraded to macOS 10.12.3 and then to VMware Fusion 8.5.3 and had to edit nat.conf and restart Fusion, so this is definitely still an issue.

[jchappell82]

This has also been an issue for me on every Fusion 8.5+ release in combination with Sierra. Prior to finding this issue just now, I had worked around it by installing an older 8.0.x release, which has actually seemed to work reasonably well, despite it not being officially supported on Sierra.

[darthsteven]

Just updated OSX and had this issue again.

Is someone from hashicorp looking into this issue at all? Seems a bit poor that this is dragging on without a resolution.

[nishankkhanna]

Quite honestly, it's disheartening to see this bug creep up again and again and again -- especially since the VMware plugin is a paid product.

[rauhryan]

Here to report the same issue. Just rebuilt my laptop and installed everything from scratch.

• VMware Fusion: 8.5.3 build-4696910
• vagrant: 1.8.7
• vagrant-vmware-fusion: 4.0.15
• macOS Sierra 10.12.3 (16D32)

---

I grabbed the debug logs and the nat.conf after each step

1. nat.conf before
2. vagrant up
3. nat.config
4. vagrant halt
5. nat.config
6. vagrant up
7. nat.config

---

Link to the gist https://gist.github.com/rauhryan/fd19c6b46f3e22be11f8333245a57d85

[chronon]

@chrisroberts: we appreciate you looking into this. Do you have any news on an updated release with a fix or at least some improvements? Fighting getting multiple VM's running every day is wearing thin.

[chrisroberts]

@chronon Hi! Yes! I have a fix for this staged in the next release that will be getting pushed out shortly. I'm just wrapping up some testing on Windows hosts. I'll be updating (and closing) this issue as soon as it's available.

[chrisroberts]

The fix for this issue has been released and is included within the 4.0.17 releases of vagrant-vmware-fusion and vagrant-vmware-workstation plugins. If your nat.conf file currently contains invalid entries (vagrant inserted entries no longer wrapped with identifiers) you may need to manually clear them one last time.

I'm going to close this issue as solved. If you encounter the collision behavior again after upgrading, please open a new issue with all the details of your local environment.

Thanks!

[nishankkhanna]

@chrisroberts The issue still remains. Now it's just sporadic -- works fine sometimes, doesn't at other times.

Some notes:

• Upgraded the plugin to v4.0.17 (latest)
• Upgraded vagrant to 1.9.2 (latest)
• Removed any entries from nat.conf and networking
• OS: macOS v10.12.3 (latest)
• VMware: v8.5.3 (latest)

Worked the first time (nat.conf has the vagrant comment block), didn't work the next time.

The problem seems to be that on times it doesn't work, there is no vagrant comment blocks in nat.conf (it has the port entries, but without the vagrant start and end comment markers).

[f1sherman]

I'm also still seeing this issue. Same environment as @nishankkhanna.

[2ndalpha]

Same here. Using Virtualbox for now :(

[chrisroberts]

Are port collision errors still being encountered?

[chronon]

I'm still seeing port collisions. After manually clearing out nat.conf entries vagrant up is successful. On vagrant halt the port forwarding entries remain in nat.conf (wrapped in vagrant markers). A second vagrant up does not make it past Waiting for machine to boot. This may take a few minutes....

Bringing machine 'default' up with 'vmware_fusion' provider...
==> default: Checking if box 'bento/ubuntu-16.04' is up to date...
==> default: Verifying vmnet devices are healthy...
==> default: Pruning invalid NFS exports. Administrator privileges will be required...
==> default: Preparing network adapters...
==> default: Fixed port collision for 80 => 8080. Now on port 10200.
==> default: Fixed port collision for 3306 => 3307. Now on port 10201.
==> default: Fixed port collision for 1080 => 1080. Now on port 10202.
==> default: Fixed port collision for 22 => 2222. Now on port 10203.
==> default: Starting the VMware VM...
==> default: Waiting for machine to boot. This may take a few minutes...
^C==> default: Waiting for cleanup before exiting...
Vagrant exited after cleanup due to external interrupt.

[chrisroberts]

@chronon Can you share the Vagrantfile you are using so I can attempt to reproduce the behavior you are experiencing locally?

[chronon]

@chrisroberts here is one of the effected Vagrantfiles, though I previously was having the same issue with 3 or 4 others (both similar and different configurations). The only thing totally in common is use of nfs for synced folders and the bindfs plugin.

[chrisroberts]

@chronon Where are the extra port forwards defined (80, 3306, etc) and if using --debug does it show what Vagrant is waiting on?

[chronon]

@chrisroberts Sorry, the above output was from a different Vagrantfile (with additional ports defined). Using the one I provided you with results in a collision on 22 only.

In the VMware console, it's waiting on this:

A snippet of the log that repeats showing what Vagrant is waiting on is here.

In nat.conf, the Vagrant markers are now gone:

[incomingtcp]

# Use these with care - anyone can enter into your VM through these...
# The format and example are as follows:
#<external port number> = <VM's IP address>:<VM's port number>
#8080 = 172.16.3.128:80
1080 = 172.16.8.129:1080
3307 = 172.16.8.129:3306
2222 = 172.16.8.129:22
8080 = 172.16.8.129:80

[chrisroberts]

Okay, those collisions are not an issue as they are flagged to auto correct themselves if a collision is detected. When the VM stalls as it has above, what state is it in after you ctrl-c? Does it list it as running? If so, can you vagant halt --force and then vagrant up successfully?

One last thing, when attempting to bring the VM back up, are you doing this after waking the host machine from a sleep?

[chronon]

When it stalls Vagrant reports it as running, and it can be stopped with vagrant halt (without --force). Occasionally the console message will make it to default: SSH auth method: private key and hang there, but the VMware console does not change (per my screenshot above). After a ctrl-c and another vagrant up it still won't proceed. The only way I've found to get it to fully boot is stopping and starting VMware networking multiple times.

I see the issue after freshly booting the host, and then again when I stop one VM and start another. It doesn't seem like waking from sleep effects anything.

[chrisroberts]

@chronon Thanks for the feedback to get at the core of the issue. Once I changed the box I was testing on I was able to reproduce the hanging startup behavior and track it back to settings whitelisting being enforced instead of just warned. Updated plugin is currently released. Thanks again!

[chronon]

@chrisroberts Thanks for working through the issues here, I will do some testing next week and keep my fingers crossed.

[allaire]

I'm still experiencing this issue, using latest version

[chrisroberts]

@allaire Hi! Can you provide version of vmware, vagrant, vagrant-vmware plugin, and the behavior you are experiencing? Thanks!

[allaire]

@chrisroberts Sure!

Vagrant 1.9.4

Fusion Pro Version 8.5.6

vagrant-berkshelf (5.1.1)
vagrant-omnibus (1.5.0)
vagrant-share (1.1.7, system)
vagrant-vmware-fusion (4.0.19)

Bringing machine 'default' up with 'vmware_fusion' provider...
==> default: Verifying vmnet devices are healthy...
==> default: Pruning invalid NFS exports. Administrator privileges will be required...
==> default: Preparing network adapters...
Vagrant cannot forward the specified ports on this VM, since they
would collide with some other application that is already listening
on these ports. The forwarded port to 8080 is already in use
on the host machine.

To fix this, modify your current project's Vagrantfile to use another
port. Example, where '1234' would be replaced by a unique host port:

  config.vm.network :forwarded_port, guest: 80, host: 1234

Sometimes, Vagrant will attempt to auto-correct this for you. In this
case, Vagrant was unable to. This is usually because the guest machine
is in a state which doesn't allow modifying port forwarding. You could
try 'vagrant reload' (equivalent of running a halt followed by an up)
so vagrant can attempt to auto-correct this upon booting. Be warned
that any unsaved work might be lost.
 $ subl /Library/Preferences/VMware\ Fusion/vmnet8/nat.conf
$ vagrant up
Bringing machine 'default' up with 'vmware_fusion' provider...
==> default: Verifying vmnet devices are healthy...
==> default: Preparing network adapters...
==> default: Starting the VMware VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 192.168.157.133:22
    default: SSH username: vagrant
    default: SSH auth method: private key
==> default: Machine booted and ready!
==> default: Forwarding ports...
    default: -- 80 => 8080
    default: -- 22 => 2222

[chrisroberts]

@allaire is this behavior easily reproducible?

[allaire]

@chrisroberts It doesn't happen all the time no. It seems to happen most of the time when I do vagrant reload instead of vagrant halt, wait, vagrant up.

Once I get in this error state, I manually have to edit the nat.conf file.

[incomingtcp]

# Use these with care - anyone can enter into your VM through these...
# The format and example are as follows:
#<external port number> = <VM's IP address>:<VM's port number>
#8080 = 172.16.3.128:80
2222 = 192.168.157.133:22
8080 = 192.168.157.133:80

[jakewan]

The behavior described by the OP seems to be happening pretty consistently right now.

1) Edit /Library/Preferences/VMware\ Fusion/vmnet8/nat.conf to make sure there's nothing in the incomingtcp section except comments.

Verify:

$ grep -A 12 incomingtcp /Library/Preferences/VMware\ Fusion/vmnet8/nat.conf
[incomingtcp]

# Use these with care - anyone can enter into your VM through these...
# The format and example are as follows:
#<external port number> = <VM's IP address>:<VM's port number>
#8080 = 172.16.3.128:80

[incomingudp]

# UDP port forwarding example
#6000 = 172.16.3.0:6001

2) Start the VM. At this point, I see one of two sets of messages.

$ vagrant up
Bringing machine 'default' up with 'vmware_fusion' provider...
==> default: Checking if box 'bento/ubuntu-16.04' is up to date...
==> default: Verifying vmnet devices are healthy...
==> default: Preparing network adapters...
Vagrant cannot forward the specified ports on this VM, since they
would collide with some other application that is already listening
on these ports. The forwarded port to 8080 is already in use
on the host machine.

To fix this, modify your current project's Vagrantfile to use another
port. Example, where '1234' would be replaced by a unique host port:

  config.vm.network :forwarded_port, guest: 80, host: 1234

Sometimes, Vagrant will attempt to auto-correct this for you. In this
case, Vagrant was unable to. This is usually because the guest machine
is in a state which doesn't allow modifying port forwarding. You could
try 'vagrant reload' (equivalent of running a halt followed by an up)
so vagrant can attempt to auto-correct this upon booting. Be warned
that any unsaved work might be lost.

OR

$ vagrant up
Bringing machine 'default' up with 'vmware_fusion' provider...
==> default: Checking if box 'bento/ubuntu-16.04' is up to date...
==> default: Verifying vmnet devices are healthy...
==> default: Preparing network adapters...
Some of the defined forwarded ports would collide with existing
forwarded ports on VMware network devices. This can be due to
existing Vagrant-managed VMware machines, or due to manually
configured port forwarding with VMware. Please fix the following
port collisions and try again:

2222, 8080

3) Check to see what has happened to nat.conf. Something has added entries _without_ the surrounding VAGRANT comments:

$ grep -A 12 incomingtcp /Library/Preferences/VMware\ Fusion/vmnet8/nat.conf
[incomingtcp]

# Use these with care - anyone can enter into your VM through these...
# The format and example are as follows:
#<external port number> = <VM's IP address>:<VM's port number>
#8080 = 172.16.3.128:80
8081 = 192.168.223.169:8081
2222 = 192.168.223.169:22
8080 = 192.168.223.169:80
2200 = 192.168.223.169:22

[incomingudp]

As an experiment, I tried deleting the line in my Vagrantfile that sets up port forwarding. Upon doing vagrant up, it still complained about a conflicting entry for port 2222, so it _seems_ like a bug somewhere in the area of whatever does the automatic port forwarding for SSH, but I'm only speculating.

Versions:

• VMWare Fusion 8.5.6
• Vagrant 1.9.4
• vagrant-vmware-fusion plugin 4.0.19

[jakewan]

And here we are 10 minutes later and I'm back to not being able to reproduce it. All entries in nat.conf are surrounded by VAGRANT comments. They survive vagrant halt and vagrant reload just fine. vagrant destroy -f removes them and vagrant up works again.

The only thing I changed was to go to a different directory and start a new Vagrantfile using the one that the OP posted. As soon as I tried vagrant up on it, I got the collision error because I had left the previous entries in nat.conf. Then I cleared them out and did vagrant up again. After that, it's been smooth sailing.

[joergRossdeutscher]

Seems killing the virtualbox networking fixed the fusion network for me.
sudo /Library/Application\ Support/VirtualBox/LaunchDaemons/VirtualBoxStartup.sh stop

[jakewan]

To be clear, this happens even when Virtualbox and VMWare Fusion are not installed side-by-side. 😄

[joergRossdeutscher]

For me it looks like this are two problems.

1. Some bug causes the problem, nat.conf contains unwanted entries then. Not connected to Virtualbox.
2. Virtualbox seems responsible for the part “I fixed the nat.conf file, but something made the old settings re-appear in that file magically“

[jakewan]

@joergRossdeutscher I agree there's some bug causing the unwanted "unmanaged" entries to appear in nat.conf. What entity contains that bug is unclear however. The leftover entries were definitely ones originally managed by Vagrant, but somehow they now appear without the surrounding comments. Maybe it's something in VMWare doing that automatically and the plugin has no control over it, but it's hard to speculate.

Whether some people experience a similar issue when they are running both Virtualbox and VMWare may cloud the issue unnecessarily, since it's common to experience such various issues when doing that. I would recommend focusing attention on the use case where only VMWare Fusion is being used, since we know this behavior is unexpected then.

[arnoleo]

Today, I encountered the same problem after a vagrant destroy -f && vagrant up

I looked into the /Library/Preferences/VMware\ Fusion/networking and remove an _add_nat_portfwd_ rule set on 2222 port and restarted Fusion.

It seemed to have fixed the problem.

[BryanGullan]

I've been banging my head against persistent issues with a failure for the network connection to become available: I was looking for things like "SSH timeout" and "Vagrant/VMWare networking does not start". In the VMWare GUI, things were hanging at "A start job is running for Raise network interfaces", and the vagrant sequence could not SSH in.

Finally I came across this thread, and now can not only reproduce this issue every time (that was no problem!) but with the information above, can get the box back up with relatively little hassle. Hopefully this will help others, and even better help figure out a fix.

I'm running:

• VMWare Fusion 8.5.6
• Vagrant 1.9.4
• vagrant-vmware-fusion 4.0.19
• OS X 10.12.4

Steps taken:
After a successful vagrant up, the following lines are in my /Library/Preferences/VMware\ Fusion/vmnet8/nat.conf file:

[incomingtcp]
#8080 = 172.16.3.128:80
# VAGRANT-BEGIN: ...vmx
2222 = 192.168.25.128:22
# VAGRANT-END: ...vmx

Now, if I run vagrant reload:

During the boot sequence - it seems to happen at "verifying vmnet devices are healthy appears on screen" - the file briefly reverts to

[incomingtcp]
#8080 = 172.16.3.128:80

and then becomes - note the lack of VAGRANT start and end tags:

[incomingtcp]
#8080 = 172.16.3.128:80
2222 = 192.168.25.128:22

And it all gets stuck at the following vagrant step:

vm: SSH address: 192.168.25.128:22
vm: SSH username: vagrant
vm: SSH auth method: private key

Hit Ctrl-C then put the nat.conf file back to

[incomingtcp]
#8080 = 172.16.3.128:80

vagrant reload

Box comes up just fine.

Incidentally, I did have the following line in /Library/Preferences/VMware\ Fusion/networking:

add_nat_portfwd 8 tcp 2222 192.168.25.128 22

but did not have to remove this (it'll be correct for my setup, as long as I've sorted out the nat.conf file as above).

[edgan]

I see this problem. The command "vagrant up" fails the first time, and works the second time, without changing anything.

vagrant --version
Vagrant 1.9.5

vagrant plugin list
vagrant-share (1.1.8, system)
vagrant-vmware-fusion (4.0.19)

[kiere]

This has suddenly become a problem with me with a set of version numbers matching @BryanGullan. This didn't happen until I upgraded from Vagrant v1.9.0 and the VMWare plugin from an earlier 4.0.x version (sorry I don't remember what it was). I also updated VMWare Fusion to the latest 8.5.7.

Vagrant 1.9.5
VMware Plugin 4.0.19
VMware Fusion 8.5.7

UPDATE: Perhaps this is because I didn't follow @chrisroberts advice about clearing the port forwarding line one last time in vmnet8/nat.conf because I did that and now see the "wrapped" version of the line.

[bolso]

Some investigation after going through port forwarding issues... Note that there are no forwarding rules in the vagrantfile

1. Cleared out nat.conf's incomingtcp section

vagrant up works normally. At some point it outputs

==> default: Machine booted and ready!
==> default: Forwarding ports...
    default: -- 5985 => 55985
    default: -- 5986 => 55986
    default: -- 22 => 2222

The contents of the incomingtcp section (comments omitted) are:

# VAGRANT-BEGIN: /Users/xxx/.vagrant/machines/default/vmware_fusion/yyy/zzz.vmx
55985 = 192.168.83.128:5985
55986 = 192.168.83.128:5986
2222 = 192.168.83.128:22
# VAGRANT-END: /Users/xxx/.vagrant/machines/default/vmware_fusion/yyy/zzz.vmx

2. vagrant halt doesn't change the contents of nat.conf

3. vagrant up again

Output:

Bringing machine 'default' up with 'vmware_fusion' provider...
==> default: Verifying vmnet devices are healthy...
==> default: Preparing network adapters...
==> default: Fixed port collision for 5985 => 55985. Now on port 2200.
==> default: Fixed port collision for 5986 => 55986. Now on port 2201.
==> default: Fixed port collision for 22 => 2222. Now on port 2202.
==> default: Starting the VMware VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: WinRM address: 192.168.83.128:5985
    default: WinRM username: vagrant
    default: WinRM execution_time_limit: PT2H
    default: WinRM transport: negotiate

At this point the process hangs, I believe because the RDP port rules are now messed up.

The contents of nat.conf:

55985 = 192.168.83.128:5985
2222 = 192.168.83.128:22
55986 = 192.168.83.128:5986

At this point I need to use VMWare to get into the machine and shut it down. I can't RDP with the microsoft client.

macOS Sierra 10.12.5
Vagrant 1.9.5
VMWare Fusion 8.5.8
vagrant-vmware-fusion 4.0.19

[AnonymousArthur]

@BryanGullan Thank you so much! This method works.

[stevenwaskey]

Add me to the party.
I'm working on a fresh OSX install.
I'm repeatedly experiencing a SSH port collision preventing vagrant up from completing.

Environment:

OSX 10.11.6
VMWare Fusion 8.5.8
Vagrant 1.9.7
vagrant-vmware-fusion 4.0.24
DHCP private network

Step to Reproduce:

vagrant up
vagrant halt
vagrant up

Read: Error message indicating there is a port collision on 2222.

Remedy:

1. Remove add_nat_portfwd entries in /Library/Preferences/VMware\ Fusion/networking that contain the suspect port.
2. Remove suspect port declaration within the [incomingtcp] block of /Library/Preferences/VMware\ Fusion/vmnet8/nat.conf

In my case, I've seen it look like one of the two following examples:

2222 = 192.168.106.128:22

&

# VAGRANT-BEGIN: /Users/swaskey/Sites/NSI/.vagrant/machines/default/vmware_fusion/b82b768b-8ce3-4b77-bfa4-01cad57ebe00/ubuntu-14.04-amd64-vmwarefusion.vmx
2222 = 192.168.106.128:22
# VAGRANT-END: /Users/swaskey/Sites/NSI/.vagrant/machines/default/vmware_fusion/b82b768b-8ce3-4b77-bfa4-01cad57ebe00/ubuntu-14.04-amd64-vmwarefusion.vmx

When I clear those entries, really quite similar to @BryanGullan experience, I am able to boot without issue.
(Though in my case, I had to remove entries from both files.)

I've started working up a little bash script to help this in the future. It's not quite complete, but for anyone whose interested - here it is.

[jhogendorn]

So theres a new version of VMWare on the horizon, is there any chance of this bug being resolved any time soon? @chrisroberts @mitchellh can we get an update?

[jdelaune]

Thanks @stevenwaskey that solved the issue for me as well. This bug sure is annoying

[colepeters]

Thanks so much @stevenwaskey, this fixed the issue for me as well. What a relief.

[jen20]

I am still seeing this on High Sierra with VMWare 8.5.2 - it should be reopened. Requiring manual editing of files is not an acceptable workaround, and nor is requiring an upgrade to VMWare Fusion 10.0 (and therefore also a plugin license upgrade).

[chrisroberts]

@jen20 Hi James, what version of the VMware plugin do you currently have installed? Do you have the latest version installed (currently 5.0.2)? There is no upgrade required (VMware or plugin license) to use the latest version of the plugin. The plugin license would only need to be upgraded if you are using Fusion 10.

[sigil66]

I still experience the issue with:
OS X: 10.12.6
VMWare: 8.5.8
vagrant-vmware-fusion: 5.0.2

[jen20]

I experience it with the same versions of VMWare and the vagrant-vmware-fusion plugin as @sigil66, but with OSX 10.13.

[chrisroberts]

Has the nat.conf file been cleaned of any old forwards? Are these collisions newly generated? I was able to isolate the source of the collision to the vmnet service rewriting the nat.conf file after a service restart. Updates to the service interaction and networking file configuration resolved the collisions issues (and this was repro + implementation fix was on Fusion 8/Workstation 12).

If your nat file is clean and you are still getting collisions, would you gist a debug run so I can take a look at the behavior. I've got a fresh macos 10.13 + fusion 8 test instance running to see if I can force a reproduction. Some debug output may help me track down a difference to force a collision state.

Thanks!

[jen20]

Yes, nat.conf has no other forwards (I've been cleaning it out regularly as a workaround for this). Next time I see it I'll gist debug output, but it's fairly non-deterministic. FWIW it appears vastly more frequently on my "fast" machine than my laptop, re-enforcing the idea that it is a race condition. For most work (at least the bits that don't require nested virtualisation that actually works) I've switched to using the Parallels provider as it is open source and I can fix it myself when it breaks.

[jen20]

OK here's a new one. A box running the latest versions of everything with Fusion _10_ instead of 8.5:

❯ vagrant up
Bringing machine 'compile' up with 'vmware_fusion' provider...
==> compile: Cloning VMware VM: 'FreeBSD-12.0-CURRENT-BHYVE-NODEBUG'. This can take some time...
==> compile: Verifying vmnet devices are healthy...
The VMware "vmnet" devices are failing to start. The most common
reason for this is collisions with existing network services. For
example, if a hostonly network space collides with another hostonly
network (such as with VirtualBox), it will fail to start. Likewise,
if forwarded ports collide with other listening ports, it will
fail to start.

Vagrant does its best to fix these issues, but in some cases it
cannot determine the root cause of these failures.

Please verify you have no other colliding network services running.
As a last resort, restarting your computer often fixes this issue.

The second time (with debug logging) it worked though. I think it's safe to say this isn't fixed.

[sigil66]

@jen20 same, if I run vagrant up enough times it will work eventually.

[jen20]

@chrisroberts I have a new theory here. I've now seen it fail two or three times in a row, then succeed as soon as I enable debug output. I think the additional work of debug logging is sufficient to make the race not happen. I would suggest trying faster hardware to reproduce this.

[jen20]

@chrisroberts Here is a complete debug log (with only the some directory names redacted in a way which doesn't alter semantics).

https://gist.github.com/jen20/6121900158a5388576c9628150b85db5

This is with Vagrant 2.0.0 rather than 2.0.1 but nothing in the change log suggests that a fix has been applied in the point release.

[jen20]

The subsequent vagrant up completed successfully with no other operation carried out, lending further credence to the idea that this is a race condition.

[ice799]

I can confirm that I recently upgraded my vagrant and vagrant plugins and see this issue on:

Vagrant 2.0.2
VMWare Fusion Version 8.5.10 (7527438)
vagrant-vmware-fusion (5.0.4)

running OSX 10.13.3 High Sierra

with a cleared nat.conf (which I have previously cleared somewhat regularly to avoid this issue, prior to this upgrade) containing no conflicting port forwards.

Running vagrant up multiple times eventually succeeds.

[sigil66]

Wondering why this is closed, this is the issue that continues to give ...

[davosian]

I am having the same issue using the newer vmware-desktop plugin, version 1.0.2. Clearing out nat.conf manually resolves the issue temporarily.

Vagrant 2.0.3
vagrant-vmware-desktop (1.0.2)
VMWare Fusion Pro 10.1.1

Vagrant installation performed through the official installers and cleared out ~/.vagrant.d to start fresh.

[chrisroberts]

@davosian Would you halt/destroy all VMware VMs and follow these steps:

• Remove any port forward entries in /Library/Preferences/VMware Fusion/vmnet8/nat.conf
• Remove any port forward directives in /Library/Preferences/VMware Fusion/networking
• Stop VMware networking: sudo /Applications/VMware Fusion/Contents/Library/vmnet-cli --stop
• Start VMware networking: sudo /Applications/VMware Fusion/Contents/Library/vmnet-cli --start
• Validate port forward entries/directives do not exist in networking or nat.conf files

After this, do you still collisions occurring?

Thanks!

[davosian]

@chrisroberts I followed your instructions and so far I am not having any more collisions. networking did not have any port forwarding set up.

Thanks for your support!

[davosian]

@chrisroberts I followed your instructions and so far I am not having any more collisions. networking did not have any port forwarding set up.

Thanks for your support!

[davosian]

I spoke too soon - getting port collisions again when running these steps:

vagrant init centos/7
vagrant up
vagrant halt
vagrant destroy
rm -rf .vagrant
vagrant global-status --> no machine listed

When I then check /Library/Preferences/VMware Fusion/vmnet8/nat.conf I still see a port entry for ssh (2222) and running vagrant up again results in collisions.

VMWare as default provider set in my .zshrc file:

# Vagrant default provider
export VAGRANT_DEFAULT_PROVIDER='vmware_desktop';

Any idea what might be causing this?

[jen20]

This is still occurring for me with the new vagrant-desktop plugin:

Bringing machine 'linux1' up with 'vmware_desktop' provider...
Bringing machine 'linux2' up with 'vmware_desktop' provider...
==> linux1: Cloning VMware VM: 'bento/ubuntu-18.04'. This can take some time...
==> linux1: Checking if box 'bento/ubuntu-18.04' is up to date...
==> linux1: Verifying vmnet devices are healthy...
==> linux1: Preparing network adapters...
==> linux1: Starting the VMware VM...
==> linux1: Waiting for the VM to receive an address...
Some of the defined forwarded ports would collide with existing
forwarded ports on VMware network devices. This can be due to
existing Vagrant-managed VMware machines, or due to manually
configured port forwarding with VMware. Please fix the following
port collisions and try again:

2222

The Vagrantfile is:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
    config.vm.define "linux1" do |vmCfg|
        vmCfg.vm.box = "bento/ubuntu-18.04"
        vmCfg.vm.box_check_update = true
        vmCfg.vm.hostname = "linux1"

        vmCfg.vm.network "private_network", ip: "10.16.0.10"

        vmCfg.vm.provider "vmware_desktop" do |v|
            v.vmx["memsize"] = "4096"
            v.vmx["numvcpus"] = "2"
        end
    end

    config.vm.define "linux2" do |vmCfg|
        vmCfg.vm.box = "bento/ubuntu-18.04"
        vmCfg.vm.box_check_update = true
        vmCfg.vm.hostname = "linux2"

        vmCfg.vm.network "private_network", ip: "10.16.0.11"

        vmCfg.vm.provider "vmware_desktop" do |v|
            v.vmx["memsize"] = "4096"
            v.vmx["numvcpus"] = "2"
        end
    end
end

$ vagrant version
Installed Version: 2.0.3
Latest Version: 2.0.3

You're running an up-to-date version of Vagrant!

$ vagrant plugin list
vagrant-vmware-desktop (1.0.0)

VMWare Fusion is Professional Version 10.1.1 (7520154)

Looks like this issue needs to be reopened.

[jen20]

This actually seems to cause numerous issues, including SSH'ing into the wrong box when multi-box setups are present. For example, in the case above, I can vagrant ssh linux1, but get prompted for a password on linux2, and if I enter the password baked into the box, I find myself on linux2.

Not really sure how to proceed at this point - I must use VMware (I need nested virtualization) but cannot fix the bugs in the provider as the source is not available.

[jchappell82]

I can also confirm that this is _still_ an issue in the latest vagrant-vmware-desktop plugin, running against the most recent version of VMware Fusion 10 on macOS 10.13.4 here as well.

[WizBangCrash]

I can also confirm that this is still an issue.
I'm using VMWare Fusion 10.1.1, macOS 10.13.4, vagrant 2.0.4, plugin vagrant-share 1.1.9 & plugin vagrant-vmware-desktop 1.0.3

[jscheid]

Still an issue for me as well. VMware Fusion 10.1.5, macOS 10.12.6, Vagrant 2.2.3, plugin vagrant-share 1.1.9, plugin vagrant-vmware-desktop 2.0.3. Is there anything else I can do to help fix this? It's quite annoying, takes me around 15 minutes to sort out every time I need to restart the VM (which, for unrelated reasons, is quite often, pretty much once a day).

[Fossil01]

Still an issue for me too.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.