Vagrant: Vagrant vmware error Port forward conflict on host port 2222

[fasmat]

Vagrant version

vagrant 2.2.9 (installed via homebrew)
vagrant-vmware-utility 1.0.9 (installed via homebrew)
VMware Fusion 11.5.3 (installed via homebrew)
vagrant-vmware-desktop 2.1.0, global

Host operating system

macOS Catalina 10.15.4

Guest operating system

Any

Vagrantfile

Any with port forwarding

Expected behavior

Vagrant setting up port forwarding and starting my VM.

Actual behavior

Vagrant refuses to start my VM.

Steps to reproduce

1. vagrant up

Fails with an error message similar to:

Vagrant cannot forward the specified ports on this VM, since they
would collide with some other application that is already listening
on these ports. The forwarded port to 3000 is already in use
on the host machine.

To fix this, modify your current project's Vagrantfile to use another
port. Example, where '1234' would be replaced by a unique host port:

  config.vm.network :forwarded_port, guest: 3000, host: 1234

Sometimes, Vagrant will attempt to auto-correct this for you. In this
case, Vagrant was unable to. This is usually because the guest machine
is in a state which doesn't allow modifying port forwarding. You could
try 'vagrant reload' (equivalent of running a halt followed by an up)
so vagrant can attempt to auto-correct this upon booting. Be warned
that any unsaved work might be lost.

References

This is a copy of the issue #10575 since this bug keeps reappearing. I have the feeling it got progressively worse for me with the last minor releases of vagrant. By now I have to delete nat.conf nat.json and restart the vagrant-vmware-utility every time I start a VM which has port-forwarding set up, while a year ago this was just an occasional problem.

[simon-4c]

I have the same issue, but on Windows 10. If I shut down machines and try to restart, the port forwards are hanging around in the VMNat config and I have to manually remove them

[jbonhag]

Hi there,

Thanks for opening an issue with Vagrant. This error message indicates that the something is running on port 3000 on the host machine, and Vagrant cannot create a forward to that port.

You should be able to see what's using port 3000 on the host by using the following command:

lsof -nP -iTCP:3000 | grep LISTEN

This message can also show up if you have multiple guests forwarding to port 3000 on the host. Would you mind trying the port forward with auto_correct: true to see if that solves the issue?

config.vm.network :forwarded_port, guest: 3000, host: 3000, auto_correct: true

Thanks!

[fasmat]

@jbonhag:

lsof shows no processes listening on port 3000 and there is no VM running.

The problem is a nat.json and a nat.conf file that contain incorrect entries for port forwarding. After deleting nat.json, nat.conf and restarting vagrant-vmware-utility I can start my VM. I have to go through that process every time I want to start a VM.

[magpoc]

How about if you try after you launch the VMware fusion app (that is, the app icon on your Mac's taskbar)?
In my case, the port conflict issue does not occur when the app is running.

[fasmat]

I just booted my mac, started VMWare fusion and tried to vagrant up (top left terminal window). It didn't work. Nothing is listening on port 3000 (top right terminal). My nat.json file is polluted with invalid entries again, and my nat.conf file again contains a bunch of invalid port forward settings.

The only way to recover from that is:

sudo rm /opt/vagrant-vmware-desktop/settings/nat.json
sudo rm /Library/Preferences/VMware\ Fusion/vmnet8/nat.conf
sudo killall vagrant-vmware-utility

Sometimes I have to repeat this process 2 or 3 times but eventually it recovers and I can start my VM with vagrant up.

This happens with all my vagrant VMs, not just with the one I tried to start today. The optman-test VM that is referenced in the nat.json doesn't even exist anymore, I destroyed it yesterday with vagrant destroy.

EDIT: again, this used to be an occasional issue a few weeks ago and has been bothering me since 2018, but recently it really happens every time I try to start a VM.

[fasmat]

Would you mind trying the port forward with auto_correct: true to see if that solves the issue?

config.vm.network :forwarded_port, guest: 3000, host: 3000, auto_correct: true

Thanks!

auto_correct works, but it changes my ports every time I use vagrant up, breaking some of my apps that expect the service to be available at a specific port.

[mattgagliardi]

Same issue here...
OSX 10.15.4
Vagrant 2.2.9
vagrant-vmware-desktop 2.1.0
Fusion Pro 11.5.3
I can confirm this is not a matter of other VMs running, etc. The nat.json file and the Fusion network config. are retaining the Vagrant-established port forwards after destroying VMs. If i clear out those files and recycle vagrant-vmware-utility I'm able to successfully "vagrant up" the VM...then the cycle repeats and I have to delete everything all over again.

[simon-4c]

Further to this (latest version of everything, Windows 10):

My host crashed without shutting down the vagrant box cleanly. When I restarted, I could see that vmnat was holding the forwarded ports open. I used the VMWare virtual network manager to clear the ports down, but it instantly stuck them back in. I couldn't do anything with the vagrant box until I destroyed and re-created it. It was as if 'something' (I'm suspecting the vagrant vmware utility) was telling VMWare it still had forwarded ports, and it was impossible to recover.

I get the impression that - despite the fact that the vmware plugin is a paid for, commercial piece of software, Hashicorp seem to constantly be on the back foot whenever something changes in VMWare. This means there are lots of issues that don't exist on VirtualBox. It's a shame, because from a commercial, cross-platform, performance standpoint it's very useful but boy is it flaky. It's disappointing.

[fasmat]

@jbonhag:

Something I also noticed today is that vagrant global-status shows incorrect states of VMs (running when they are powered off). Maybe this is related?

[jbonhag]

Hi @fasmat -- thanks for the extra details. Does running vagrant global-status --prune mitigate the issue at all?

[fasmat]

Hi @jbonhag

not really. The first time I run it some VMs show as "running" if I run it again they show as "not running" but the port forwarding issue still persists :-(

[jbonhag]

Sorry to hear that -- thanks for checking that out. So just to clarify, can you see the state of the machine that you're trying to vagrant up?

Digging into the screenshot a bit, I saw the forwards that remained in both nat.json and nat.conf. I tried "stealing" port 3000 by adding some invalid forwards to both files but they were cleaned up by Vagrant when I did a vagrant up.

Eventually I was able to get an invalid forward to stick in nat.json by setting guestip to an actual (unrelated) VM, which raised a slightly different error:

Vagrant failed to apply the requested port forward. The following
error message was generated while attempting to apply the port
forward rule:

  Port forward conflict on host port 3000

Please resolve any problems reported in the error message above and
try again.

However, I couldn't get the bad entry to stick in nat.conf. I'll keep at it and see if I can find anything else.

[fasmat]

Yes, I can see the state of the machine I try to vagrant up. When I have the problem, it shows first as running and if I call vagrant global-status --prune again it shows as not running.

Since you have trouble reproducing the issue I tried uninstalling and reinstalling vagrant, vagrant-vmware-utility and vmware fusion. But it didn't help, the issue persists. My actuall port forwards are configured like this for the VM i just tried to start:

cfg.vm.network "forwarded_port", :guest => 3000, :host => 3000
cfg.vm.network "forwarded_port", :guest => 5432, :host => 5432

[justinatomatic]

Same issue here...
OSX 10.15.4
Vagrant 2.2.9
vagrant-vmware-desktop 2.1.0
Fusion 11.5.3

I've found sometimes the workaround works and then sometimes it don't and I can't start with any port forwarding enabled.

[jbonhag]

Hi there,

Thanks for the additional details. It definitely sounds like nat.conf and nat.json are not quite in sync, causing VMware to hold onto those forwarded ports. There is a way to scrub all the port forwards and reset both files to a pristine state. Please note that this will remove _all_ VMware forwarded ports, including those not registered with Vagrant, so you would need to reload any running VMs to re-establish the port forwarding rules.

To remove all forwarded ports, run:

vagrant cap provider scrub_forwarded_ports

After that, you should be able to vagrant up or vagrant reload VMs as needed.

I appreciate your patience working through this issue!

[fasmat]

@jbonhag: thanks for the tip! This does seem to unblock ports when vagrant gives port forward errors.

It also seems like the vagrant command line utility often fails to communicate with the VMware Utility driver recently:

$ vagrant up
Bringing machine 'lin-vm' up with 'vmware_desktop' provider...
==> lin-vm: Verifying vmnet devices are healthy...
Vagrant encountered an unexpected communications error with the
Vagrant VMware Utility driver. Please try to run the command
again. If this error persists, please contact [email protected]

The only way to solve this issue seems to be to start vmware fusion manually and then call vagrant up from the command line again. I don't know if these issues are related, but if I can provide some logs or other data to narrow down the issue(s) I'd be happy to do so.

[jbonhag]

@fasmat: Interesting! Did you start the vagrant-vmware-utility with launchctl, e.g.:

sudo launchctl load -w /Library/LaunchDaemons/com.vagrant.vagrant-vmware-utility.plist

[fasmat]

Update on the issue: I still get port forwarding errors every time I try to vagrant up a VM. The "unexpected communications error" in my last response has become very rare, but rebooting has so far always solved the issue.

vagrant cap provider scrub_forwarded_ports works most of the time to solve the port forwarding issue. Sometimes it gives me this error:

Vagrant encountered an error while attempting to prune unused
port forward entries:

  Failed to update device NAT

In which case I just run the script a second time, which so far always made vagrant up work again.

Since my last comment I updated vagrant to 2.2.10, VMware Fusion to 11.5.6 and vagrant-vmware-desktop to 2.1.2, but the port forwarding issues remain.

[fasmat]

The most recent update seems to have fixed the error for me. I'm now using the following versions of the tools:

vagrant 2.2.10 (installed via homebrew)
vagrant-vmware-utility 1.0.13 (installed via homebrew)
VMware Fusion 11.5.6 (installed via homebrew)
vagrant-vmware-desktop 2.1.4, global

Thanks for the effort! :-)

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.