Vagrant: Catalina filesystem changes appear to introduce NFS sync issues

[bjfresh]

Vagrant version

2.2.5

Host operating system

Mac OS X Catalina Public Beta 2

Guest operating system

Ubuntu 16.04.1 LTS

Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

ANSIBLE_PATH = __dir__ # absolute path to Ansible directory on host machine
ANSIBLE_PATH_ON_VM = '/home/vagrant/trellis'.freeze # absolute path to Ansible directory on virtual machine

require File.join(ANSIBLE_PATH, 'lib', 'trellis', 'vagrant')
require File.join(ANSIBLE_PATH, 'lib', 'trellis', 'config')
require 'yaml'

vconfig = YAML.load_file("#{ANSIBLE_PATH}/vagrant.default.yml")

if File.exist?("#{ANSIBLE_PATH}/vagrant.local.yml")
  local_config = YAML.load_file("#{ANSIBLE_PATH}/vagrant.local.yml")
  vconfig.merge!(local_config) if local_config
end

ensure_plugins(vconfig.fetch('vagrant_plugins')) if vconfig.fetch('vagrant_install_plugins')

trellis_config = Trellis::Config.new(root_path: ANSIBLE_PATH)

Vagrant.require_version '>= 2.0.1'

Vagrant.configure('2') do |config|
  config.vm.box = vconfig.fetch('vagrant_box')
  config.vm.box_version = vconfig.fetch('vagrant_box_version')
  config.ssh.forward_agent = true
  config.vm.post_up_message = post_up_message

  # Fix for: "stdin: is not a tty"
  # https://github.com/mitchellh/vagrant/issues/1673#issuecomment-28288042
  config.ssh.shell = %(bash -c 'BASH_ENV=/etc/profile exec bash')

  # Required for NFS to work
  if vconfig.fetch('vagrant_ip') == 'dhcp'
    config.vm.network :private_network, type: 'dhcp', hostsupdater: 'skip'

    cached_addresses = {}
    config.hostmanager.ip_resolver = proc do |vm, _resolving_vm|
      if cached_addresses[vm.name].nil?
        if vm.communicate.ready?
          vm.communicate.execute("hostname -I | cut -d ' ' -f 2") do |_type, contents|
            cached_addresses[vm.name] = contents.split("\n").first[/(\d+\.\d+\.\d+\.\d+)/, 1]
          end
        end
      end
      cached_addresses[vm.name]
    end
  else
    config.vm.network :private_network, ip: vconfig.fetch('vagrant_ip'), hostsupdater: 'skip'
  end

  main_hostname, *hostnames = trellis_config.site_hosts_canonical
  config.vm.hostname = main_hostname

  if Vagrant.has_plugin?('vagrant-hostmanager') && !trellis_config.multisite_subdomains?
    redirects = trellis_config.site_hosts_redirects

    config.hostmanager.enabled = true
    config.hostmanager.manage_host = true
    config.hostmanager.aliases = hostnames + redirects
  elsif Vagrant.has_plugin?('landrush') && trellis_config.multisite_subdomains?
    config.landrush.enabled = true
    config.landrush.tld = config.vm.hostname
    hostnames.each { |host| config.landrush.host host, vconfig.fetch('vagrant_ip') }
  else
    fail_with_message "vagrant-hostmanager missing, please install the plugin with this command:\nvagrant plugin install vagrant-hostmanager\n\nOr install landrush for multisite subdomains:\nvagrant plugin install landrush"
  end

  bin_path = File.join(ANSIBLE_PATH_ON_VM, 'bin')

  vagrant_mount_type = vconfig.fetch('vagrant_mount_type')

  if vagrant_mount_type != 'nfs' || Vagrant::Util::Platform.wsl? || (Vagrant::Util::Platform.windows? && !Vagrant.has_plugin?('vagrant-winnfsd'))
    vagrant_mount_type = nil if vagrant_mount_type == 'nfs'
    trellis_config.wordpress_sites.each_pair do |name, site|
      config.vm.synced_folder local_site_path(site), remote_site_path(name, site), owner: 'vagrant', group: 'www-data', mount_options: ['dmode=776', 'fmode=775'], type: vagrant_mount_type
    end

    config.vm.synced_folder ANSIBLE_PATH, ANSIBLE_PATH_ON_VM, mount_options: ['dmode=755', 'fmode=644'], type: vagrant_mount_type
    config.vm.synced_folder File.join(ANSIBLE_PATH, 'bin'), bin_path, mount_options: ['dmode=755', 'fmode=755'], type: vagrant_mount_type
  elsif !Vagrant.has_plugin?('vagrant-bindfs')
    fail_with_message "vagrant-bindfs missing, please install the plugin with this command:\nvagrant plugin install vagrant-bindfs"
  else
    trellis_config.wordpress_sites.each_pair do |name, site|
      config.vm.synced_folder local_site_path(site), nfs_path(name), type: 'nfs', nfs_version: 4, nfs_udp: false
      config.bindfs.bind_folder nfs_path(name), remote_site_path(name, site), u: 'vagrant', g: 'www-data', o: 'nonempty'
    end

    config.vm.synced_folder ANSIBLE_PATH, '/ansible-nfs', type: 'nfs', nfs_version: 4, nfs_udp: false
    config.bindfs.bind_folder '/ansible-nfs', ANSIBLE_PATH_ON_VM, o: 'nonempty', p: '0644,a+D'
    config.bindfs.bind_folder bin_path, bin_path, perms: '0755'
  end

  vconfig.fetch('vagrant_synced_folders', []).each do |folder|
    options = {
      type: folder.fetch('type', 'nfs'),
      create: folder.fetch('create', false),
      mount_options: folder.fetch('mount_options', [])
    }

    destination_folder = folder.fetch('bindfs', true) ? nfs_path(folder['destination']) : folder['destination']

    config.vm.synced_folder folder['local_path'], destination_folder, options

    if folder.fetch('bindfs', true)
      config.bindfs.bind_folder destination_folder, folder['destination'], folder.fetch('bindfs_options', {})
    end
  end

  provisioner = local_provisioning? ? :ansible_local : :ansible
  provisioning_path = local_provisioning? ? ANSIBLE_PATH_ON_VM : ANSIBLE_PATH

  config.vm.provision provisioner do |ansible|
    if local_provisioning?
      ansible.install_mode = 'pip'
      ansible.provisioning_path = provisioning_path
      ansible.version = vconfig.fetch('vagrant_ansible_version')
    end

    ansible.compatibility_mode = '2.0'
    ansible.playbook = File.join(provisioning_path, 'dev.yml')
    ansible.galaxy_role_file = File.join(provisioning_path, 'requirements.yml') unless vconfig.fetch('vagrant_skip_galaxy') || ENV['SKIP_GALAXY']
    ansible.galaxy_roles_path = File.join(provisioning_path, 'vendor/roles')

    ansible.groups = {
      'web' => ['default'],
      'development' => ['default']
    }

    ansible.tags = ENV['ANSIBLE_TAGS']
    ansible.extra_vars = { 'vagrant_version' => Vagrant::VERSION }

    if (vars = ENV['ANSIBLE_VARS'])
      extra_vars = Hash[vars.split(',').map { |pair| pair.split('=') }]
      ansible.extra_vars.merge!(extra_vars)
    end
  end

  # Virtualbox settings
  config.vm.provider 'virtualbox' do |vb|
    vb.name = config.vm.hostname
    vb.customize ['modifyvm', :id, '--cpus', vconfig.fetch('vagrant_cpus')]
    vb.customize ['modifyvm', :id, '--memory', vconfig.fetch('vagrant_memory')]
    vb.customize ['modifyvm', :id, '--ioapic', vconfig.fetch('vagrant_ioapic', 'on')]

    # Fix for slow external network connections
    vb.customize ['modifyvm', :id, '--natdnshostresolver1', vconfig.fetch('vagrant_natdnshostresolver', 'on')]
    vb.customize ['modifyvm', :id, '--natdnsproxy1', vconfig.fetch('vagrant_natdnsproxy', 'on')]
  end

  # VMware Workstation/Fusion settings
  %w(vmware_fusion vmware_workstation).each do |provider|
    config.vm.provider provider do |vmw, _override|
      vmw.name = config.vm.hostname
      vmw.vmx['numvcpus'] = vconfig.fetch('vagrant_cpus')
      vmw.vmx['memsize'] = vconfig.fetch('vagrant_memory')
    end
  end

  # Parallels settings
  config.vm.provider 'parallels' do |prl, _override|
    prl.name = config.vm.hostname
    prl.cpus = vconfig.fetch('vagrant_cpus')
    prl.memory = vconfig.fetch('vagrant_memory')
    prl.update_guest_tools = true
  end
end

Debug output

https://gist.github.com/bjfresh/673d66b9ab814e6ca66404ca6755a885

Expected behavior

NFS syncs the folder

Actual behavior

Catalina introduces separate volumes for system and user data (Root changed from "/" to "System/Volumes/Data/" ), which appears to be the cause of this unexpected mismatch:

NFS is reporting that your exports file is invalid. Vagrant does
this check before making any changes to the file. Please correct
the issues below and execute "vagrant reload":

exports:2: exported dir/fs mismatch: /Users/bj/Sites/sitename.com/site /System/Volumes/Data
exports:3: exported dir/fs mismatch: /Users/bj/Sites/sitename.com/trellis /System/Volumes/Data

Steps to reproduce

1. Install MacOS Catalina Public Beta 2
2. Run 'vagrant up' on an existing Vagrant + Trellis installation using NFS

FWIW I've updated Virtualbox, Vagrant, and relevant plugins

[briancain]

Quick question @bjfresh - Can you bring up new guests, or does this only affect guests that were created on a previous version of macOS? Thanks.

[bjfresh]

@briancain seems a fresh, unmodified clone of the Roots Example Project is able to launch without issue for me.

[briancain]

Interesting.... @bjfresh well that is good news. :pray: We might be able to get some sort of handling around old guests on newer upgrades of macOS, but if fresh guests seem to work at least there's a path forward currently and isn't completely busted.

[musikov]

It's not a vagrant issue but nfsd/macos issue.
If you add to /etc/exports line:
/Users/USERNAME/Documents
and then run nfsd checkexports, you'll receive:
exports:22: exported dir/fs mismatch: /Users/USERNAME/Documents /System/Volumes/Data

But if you explicitly add path to data volume to /etc/exports:
/System/Volumes/Data/Users/USERNAME/Documents
And run nfsd checkexports it will pass.

So i think it's wrong firmlink handling done by either macos of nfsd

[cyberkurumin]

I'm facing the same problem but I just can't make it work. Everything has been updated, but the _mismatch_ message always come back.

Do you know if there is any solution or workaround?

[musikov]

I'm facing the same problem but I just can't make it work. Everything has been updated, but the _mismatch_ message always come back.

Do you know if there is any solution or workaround?

workaround is to add /System/Volumes/Data/... to your paths in VagrantFile and /etc/exports

[telcy]

same problem here. not able to get around the mismatch, as vagrant up always adds it again

EDIT:
I have added /System/Volumes/Data to the ANSIBLE_PATH and now I stuck at:

==> default: Cannot create bind mount from '/vagrant-nfs-project1' to '/srv/www/project1/current': Source path '/vagrant-nfs-project1' doesn't exist
==> default: Cannot create bind mount from '/vagrant-nfs-project2' to '/srv/www/project2/current': Source path '/vagrant-nfs-project2' doesn't exist

[RawDawg]

workaround is to add /System/Volumes/Data/... to your paths in VagrantFile and /etc/exports

Where exactly would I add this to Vagrantfile? I have added it to my etc/exports but it keeps changing back with every vagrant up.

Another option might be to use nfs_export: false but I can’t figure out where to put it in my Vagrantfile or vagrant.default.yml.

[musikov]

Where exactly would I add this to Vagrantfile? I have added it to my etc/exports but it keeps changing back with every vagrant up.

Where you describe path to source folder:

vms.vm.synced_folder "~/dev/myproject/", "/src/myproject", type: "nfs"

change to:

vms.vm.synced_folder "/System/Volumes/Data/Users/USERNAME/dev/myproject/", "/src/myproject", type: "nfs"

but, if you commit this file, this will crash vagrant for all other users.

Another option might be to use nfs_export: false but I can’t figure out where to put it in my Vagrantfile or vagrant.default.yml.

Here:

vms.vm.synced_folder "~/dev/myproject/", "/src/myproject", type: "nfs", nfs_export: false

[brablc]

I have tried rather naive:

  nfsPath = "/System/Volumes/Data" + Dir.pwd

  config.vm.synced_folder nfsPath + "../src",   "/vagrant-src", :nfs => !RUBY_PLATFORM.downcase.include?("w32"), :linux__nfs_options => ["async,rw,no_subtree_check,all_squash"]

Which seems to construct correct path. Before running vagrant up again, please remove lines added by vagrant to /etc/exports.

Btw, the template that is used to write to the exports file is located in /opt/vagrant/embedded/gems/2.2.5/gems/vagrant-2.2.5/templates/nfs/exports_bsd.erb and the file which reads it is /opt/vagrant/embedded/gems/2.2.5/gems/vagrant-2.2.5/plugins/hosts/bsd/cap/nfs.rb.

[ryscript]

There's even a much simpler approach, make sure the vagrant is up

1. Open VirtualBox
2. Click on "Shared Folders" Tab
3. on Machine Folders list view, double click/tap the configuration
4. Navigate to the new Catalina path

restart vagrant and you're done.

[slkxmail]

resolving this problem https://stackoverflow.com/questions/20726248/vagrant-error-nfs-is-reporting-that-your-exports-file-is-invalid

[bobbytables]

If you're here like I was because of Docker For Mac and NFS, I've got you: https://www.firehydrant.io/blog/nfs-with-docker-on-macos-catalina/

[pingram3030]

_edited to make the config backwards compatible_

@brablc, try this, it worked for us:

  config.vm.synced_folder "/System/Volumes/Data/Users/#{ENV['USER']}/dev/myproject", '/src/myproject',
    id: 'myproject',
    type: 'nfs',
    nfs_version: 3,
    nfs_udp: false

• Vagrantfile is Ruby
• Ruby, like Bash, can expand variables within double quotes, but not within single quotes.
• Any Env var is accessible in Ruby with ENV['VARIABLE_NAME'], or within double quotes as "#{ENV['VARIABLE_NAME']}"
• nfs_udp: false forces TCP to be used which will reduce failures when dealing with thousands of files.
• You will need to halt all VMs and remove all offending entries from /etc/exports by hand before executing vagrant up with this work-around.
• Variablising:
  
  
  
  • NFSv4 is not supported on MacOS, so to make your config Linux compatible, you will need to variablise nfs_verison.
  
  
  • To make your config backwards compat, you need to change the base depending on MacOS version
    
    case RUBY_PLATFORM when 'x86_64-linux' export_base = '/opt' nfs_version = 4.1 when 'x86_64-darwin13' case %x('uname' '-r').split('.').first when '18' export_base = ENV['HOME'] nfs_version = 3 when '19' export_base = "/System/Volumes/Data/Users/#{ENV['USER']}" nfs_version = 3 else fail("Unsupported version of MacOS") end else fail("Unsupported RUBY_PLATFORM '#{RUBY_PLATFORM}'") end ... config.vm.synced_folder "#{export_base}/dev/myproject"... nfs_version: nfs_version, linux__nfs_options: ['crossmnt', 'rw', 'no_subtree_check', 'all_squash'] ...
  
  

[jeroenmoons]

For me the workaround looks like this now, since my Vagrantfile uses "." to mount the current working directory:

nfsPath = "."
if Dir.exist?("/System/Volumes/Data")
    nfsPath = "/System/Volumes/Data" + Dir.pwd
end
config.vm.synced_folder nfsPath, "/vagrant", type: "nfs"

This provides a workaround without breaking the config for non-Catalina hosts. (This still needs existing exports in /etc/exports to be removed first or vagrant will complain about a path mismatch.)

[mmoollllee]

For me the workaround looks like this now, since my Vagrantfile uses "." to mount the current working directory:

nfsPath = "."
if Dir.exist?("/System/Volumes/Data")
    nfsPath = "/System/Volumes/Data" + Dir.pwd
end
config.vm.synced_folder nfsPath, "/vagrant", type: "nfs"

This provides a workaround without breaking the config for non-Catalina hosts. (This still needs existing exports in /etc/exports to be removed first or vagrant will complain about a path mismatch.)

Great! Do you know how to implement this in the Trellis Vagrantfile?

[deb1990]

The above solutions work to start the vagrant box, but when I do vagrant ssh and try to access the shared folder , I get cannot open directory '.': Stale file handle error.
Any one faced this issue?

[aglarsson]

After som hours of struggling I found out that you can not have your vagrant projects in the Documents folder anymore, because nfs reports "stale file" when trying to access the nfs mount.

Solutiuon: create a new folder in your home directory, ie. /Users/[user]/Development and move your projects there. Delete your cached server: rm -rf .vagrant. Then your can start your vagrant server without problem.

I do not know what happened with the Documents folder when upgrading to OS X Catalina.

[venanciorodrigo]

The temporary solution for my case was to change the sync from NFS to native VirtualBox.

config.vm.synced_folder ".", "/vagrant", type: "virtualbox"

Also as people described above you have to change the folder map to "map" => "/System/Volumes/Data/Users/{username}/Documents/{project-folder}"

[mmoollllee]

For me the workaround looks like this now, since my Vagrantfile uses "." to mount the current working directory:

nfsPath = "."
if Dir.exist?("/System/Volumes/Data")
    nfsPath = "/System/Volumes/Data" + Dir.pwd
end
config.vm.synced_folder nfsPath, "/vagrant", type: "nfs"

This provides a workaround without breaking the config for non-Catalina hosts. (This still needs existing exports in /etc/exports to be removed first or vagrant will complain about a path mismatch.)

Great! Do you know how to implement this in the Trellis Vagrantfile?

Okay, found it out for the trellis Vagrantfile!
It's as easy as changing the first line of the trellis Vagrantfile. With backwarts compatibility it looks like this:

if Dir.exist?("/System/Volumes/Data")
    ANSIBLE_PATH = "/System/Volumes/Data" + __dir__ # absolute path to Ansible directory on host machine for macos Catalina
else
    ANSIBLE_PATH = __dir__ # absolute path to Ansible directory on host machine
end

[venanciorodrigo]

After som hours of struggling I found out that you can not have your vagrant projects in the Documents folder anymore, because nfs reports "stale file" when trying to access the nfs mount.

Solutiuon: create a new folder in your home directory, ie. /Users/[user]/Development and move your projects there. Delete your cached server: rm -rf .vagrant. Then your can start your vagrant server without problem.

I do not know what happened with the Documents folder when upgrading to OS X Catalina.

Had the same issue as you described here, the solution to make NFS work again was the combination of @jeroenmoons and yours (Moving the project outside the Documents folder) otherwise I'll have the cannot open directory '.': Stale file handle error.

[TechN8]

It's possible to get this working without moving files out of Documents.

Add /sbin/nfsd in System Preferences > Security & Privacy > Privacy > Full Disk Access

[charlieartist]

@TechN8 How is this done? I"m not seeing how to add /sbin/nsfd there.

[TechN8]

@TechN8 How is this done? I"m not seeing how to add /sbin/nsfd there.

Click on the plus sign to add a program and then you can navigate to the /sbin folder by pressing Command-Shift-G. (edit) I noticed that I didn't specify that this is on the Privacy tab.

[charlieartist]

@TechN8 Thanks—I did get that set up, and did a restart to confirm. Still having access.denied issues with mount.nfs, so will keep digging.

[nusserstudios]

Full Disk Access to /sbin/nfsd didn't work for me either.

[soareswallace]

I was able to make work by doing:
1 - sudo rm /etc/exports
2 - On the Vagrantfile change the configuration on synced_folder to use the absolute path in Catalina's new pattern.
ex.:
directory = "<project_path>" config.vm.synced_folder "/System/Volumes/Data#{directory}", "/vagrant", :nfs => true, :mount_options => ['actimeo=2']
3 - Run vagrant destroy -f && vagrant up

And you are good to go!

[f0rk]

Vagrant has already fixed this issue, but has not released the fix. The files in question are the cap/path.rb and bsd/darwin plugin ones listed below. It seemed (and apparently was) safe to add/replace the existing versions locally. A couple of internal users (including myself) have run this script and it fixes the problem and does not seem to break anything else.

```!sh

!/usr/bin/env sh

vagrant_version=vagrant --version | perl -ne 'print $1 if /Vagrant (.+)/'
vagrant_path="/opt/vagrant/embedded/gems/$vagrant_version/gems/vagrant-$vagrant_version//"
​
for f in plugins/hosts/bsd/cap/path.rb plugins/hosts/bsd/plugin.rb plugins/hosts/darwin/cap/path.rb plugins/hosts/darwin/plugin.rb; do
curl --silent "https://raw.githubusercontent.com/hashicorp/vagrant/b12a23273ed9f17442506996c0f5ba8844450a4b/$f" -o "vagrant_path/$f"
done
```

[icetee]

Vagrant has already fixed this issue, but has not released the fix. The files in question are the cap/path.rb and bsd/darwin plugin ones listed below. It seemed (and apparently was) safe to add/replace the existing versions locally. A couple of internal users (including myself) have run this script and it fixes the problem and does not seem to break anything else.

#!/usr/bin/env sh
vagrant_version=`vagrant --version | perl -ne 'print $1 if /Vagrant (.+)/'`
vagrant_path="/opt/vagrant/embedded/gems/$vagrant_version/gems/vagrant-$vagrant_version//"
​
for f in plugins/hosts/bsd/cap/path.rb plugins/hosts/bsd/plugin.rb plugins/hosts/darwin/cap/path.rb plugins/hosts/darwin/plugin.rb; do
  curl --silent "https://raw.githubusercontent.com/hashicorp/vagrant/b12a23273ed9f17442506996c0f5ba8844450a4b/$f" -o "vagrant_path/$f"
done

HTTP 400 the raw link :(

[f0rk]

You can't click the raw link directly. Each of the files is specified in the for loop, i.e.:
https://raw.githubusercontent.com/hashicorp/vagrant/b12a23273ed9f17442506996c0f5ba8844450a4b/plugins/hosts/bsd/cap/path.rb
https://raw.githubusercontent.com/hashicorp/vagrant/b12a23273ed9f17442506996c0f5ba8844450a4b/plugins/hosts/bsd/plugin.rb
https://raw.githubusercontent.com/hashicorp/vagrant/b12a23273ed9f17442506996c0f5ba8844450a4b/plugins/hosts/darwin/cap/path.rb
https://raw.githubusercontent.com/hashicorp/vagrant/b12a23273ed9f17442506996c0f5ba8844450a4b/plugins/hosts/darwin/plugin.rb

[eli007s]

Vagrant has already fixed this issue, but has not released the fix. The files in question are the cap/path.rb and bsd/darwin plugin ones listed below. It seemed (and apparently was) safe to add/replace the existing versions locally. A couple of internal users (including myself) have run this script and it fixes the problem and does not seem to break anything else.

#!/usr/bin/env sh
vagrant_version=`vagrant --version | perl -ne 'print $1 if /Vagrant (.+)/'`
vagrant_path="/opt/vagrant/embedded/gems/$vagrant_version/gems/vagrant-$vagrant_version//"
​
for f in plugins/hosts/bsd/cap/path.rb plugins/hosts/bsd/plugin.rb plugins/hosts/darwin/cap/path.rb plugins/hosts/darwin/plugin.rb; do
    curl --silent "https://raw.githubusercontent.com/hashicorp/vagrant/b12a23273ed9f17442506996c0f5ba8844450a4b/$f" -o "vagrant_path/$f"
done

HTTP 400 the raw link :(

you can try

https://www.vagrantup.com/docs/installation/source.html

[charlieartist]

@icetee Just tried running the script, got syntax errors.

[icetee]

@f0rk The last release (v2.2.5), no exists "path.rb" file, I think no call the file.(https://github.com/hashicorp/vagrant/tree/ae0d574e8aff7aa4cf2eef4c29955de909879290/plugins/hosts/bsd/cap).

@eli007s I try build Vagrant. I installed rvm and I try Ruby 2.5.7 and 2.6.5, but I always get something error. I got the latest version the longest.

vagrant -v

Vagrant 2.2.6.dev

I run vagrant up command and get this error:

Traceback (most recent call last):
    2: from /Users/icetee/.rvm/gems/ruby-2.6.5/bin/vagrant:23:in `<main>'
    1: from /Users/icetee/.rvm/rubies/ruby-2.6.5/lib/ruby/site_ruby/2.6.0/rubygems.rb:303:in `activate_bin_path'
/Users/icetee/.rvm/rubies/ruby-2.6.5/lib/ruby/site_ruby/2.6.0/rubygems.rb:284:in `find_spec_for_exe': can't find gem vagrant (>= 0.a) with executable vagrant (Gem::GemNotFoundException)

I run gem build vagrant.gemspec.

 Successfully built RubyGem
  Name: vagrant
  Version: 2.2.6.dev
  File: vagrant-2.2.6.dev.gem

gem update --system

Latest version already installed. Done.

bundle -> Same Traceback issue.

Perhaps there is no more detailed description of the installation?

[d0p3fish]

Vagrant has already fixed this issue, but has not released the fix. The files in question are the cap/path.rb and bsd/darwin plugin ones listed below. It seemed (and apparently was) safe to add/replace the existing versions locally. A couple of internal users (including myself) have run this script and it fixes the problem and does not seem to break anything else.

#!/usr/bin/env sh
vagrant_version=`vagrant --version | perl -ne 'print $1 if /Vagrant (.+)/'`
vagrant_path="/opt/vagrant/embedded/gems/$vagrant_version/gems/vagrant-$vagrant_version//"
​
for f in plugins/hosts/bsd/cap/path.rb plugins/hosts/bsd/plugin.rb plugins/hosts/darwin/cap/path.rb plugins/hosts/darwin/plugin.rb; do
  curl --silent "https://raw.githubusercontent.com/hashicorp/vagrant/b12a23273ed9f17442506996c0f5ba8844450a4b/$f" -o "vagrant_path/$f"
done

When pasting this into my console with vim, there is a <200b> between the lines.
After removing this, i can execute the script.

[mark1282]

Looks like we will a release of 2.2.6 very soon soon

https://github.com/hashicorp/vagrant/blob/v2.2.6/CHANGELOG.md

[jakedowns]

Just tested Vagrant 2.2.6 after seeing https://github.com/hashicorp/vagrant/commit/ae9c3e28d671f9230c18639f452d573914fabee0

i'm getting this error when attempting to use an old box https://cl.ly/f271e8a02b9e

i'm getting this error when trying a fresh one: https://cl.ly/27da040ae23b

Vagrantfile: https://gist.github.com/jakedowns/f2218e20378886c8330fdf9977cba5da

[icetee]

I install new version, and works. Thanks!

@jakedowns You seem to have improved too. You cannot install Guest additions. No exists linux-headers-4.9.0-9-amd64 in Debian repository and no installable in the box.

[tychay]

Well it looks like it will be fixed soon.

In any case, I ran into this issue because I need NFS mounting (for symlinking in npm which virtualbox shared folders do not support) so I need to override the default shared folder with the NFS version.

config.vm.synced_folder "./", "/vagrant", type: "nfs"

My hack was to use ruby in the Vagrantfile to

vagrant_root = File.dirname(__FILE__) 
if Dir.exist?('/System/Volumes/Data')
  vagrant_root  = '/System/Volumes/Data' + vagrant_root
end
config.vm.synced_folder vagrant_root, "/vagrant", type: "nfs"

It seems to work except I'm no longer seeing the root files. This might help people get up and running until the patch is released.

[nikunjkotecha]

I confirm it works after updating to Vagrant 2.2.6

For me it required deleting the file /etc/exports and provisioning again once.

[charlieartist]

I can also confirm that it works. To do so, this is what I did:

• Update Vagrant to 2.2.6
• Update the Vagrant VBGuest and Hostsupdater plugins
• Make sure VirtualBox is updated
• Delete all existing VMs
• Delete the ~/.vagrant.d folder
• Delete the ~/Vagrant folder (if there)
• Delete the /private/etc/exports file
• Run vagrant up

[musikov]

you DO NOT need to delete any VMs.

• stop VMs
• install vagrant 2.2.6
• clear /etc/exports
• restore Vagrantfile (if you added workaround with /System/Volumes/blablabla or nfs_export: false)
• vagrant up

the vagrant up command will add to /etc/exports correct paths

[petertenhoor]

The suggested fix by @musikov is the most clean. Just make sure to backup your /etc/exports file before clearing it!

[oddnoc]

I understand this is consider fixed in 2.2.6, but I cannot confirm the solution by @musikov. vagrant up still hangs on Catalina at "Mounting NFS shared folders ...".

[oddnoc]

Hopefully this is helpful. While vagrant up is hanging, the Vagrantfile NFS configuration is config.vm.synced_folder ".", "/usr/home/vagrant/Project", :nfs => true. The guest OS (FreeBSD 11.3) is doing this:

  ├─sshd
  │   ├─sshd
  │   │   └─sshd,vagrant
  │   │       └─sudo,root -E -H sh
  │   │           └─sh
  │   │               └─mount -t nfs -o nfsv3,mntudp 172.17.0.1:/System/Volumes/Data/Volumes/Sensitive/Sites/QI/sysadmin /usr/home/vagrant/Project
  │   │                   └─mount_nfs -o nfsv3 -o mntudp 172.17.0.1:/System/Volumes/Data/Volumes/Sensitive/Sites/QI/sysadmin /usr/home/vagrant/Project

nmap -sU 172.17.0.1 reports:

PORT     STATE         SERVICE
111/udp  open          rpcbind
137/udp  open          netbios-ns
138/udp  open|filtered netbios-dgm
1014/udp open|filtered unknown
2049/udp open          nfs
5353/udp open          zeroconf

VirtualBox version: 6.0.14 r133895
Vagrant version: 2.2.6
MacOS version: 10.15 (19A602)

[paoga87]

For me the workaround looks like this now, since my Vagrantfile uses "." to mount the current working directory:

nfsPath = "."
if Dir.exist?("/System/Volumes/Data")
    nfsPath = "/System/Volumes/Data" + Dir.pwd
end
config.vm.synced_folder nfsPath, "/vagrant", type: "nfs"

This provides a workaround without breaking the config for non-Catalina hosts. (This still needs existing exports in /etc/exports to be removed first or vagrant will complain about a path mismatch.)

@jeroenmoons thanks for providing this information. I would like to try this workaround, but which file is the one that I'd need to edit? Thanks

[oddnoc]

I have figured out the problem.

The workaround in vagrant 2.2.6 for Catalina does not work when the shared folder is not on the boot volume. I use a separate, case-sensitive partition for all my projects. It shows up on Catalina at /Volumes/Sensitive as well as at /System/Volumes/Data/Volumes/Sensitive.

Vagrant dutifully creates an entry in the exports file like this:

/System/Volumes/Data/Volumes/Sensitive/Sites/example -alldirs -mapall=501:80 172.17.0.19

The MacOS console reveals this:

exports:2: exported dir/fs mismatch: /System/Volumes/Data/Volumes/Sensitive/Sites/example /Volumes/Sensitive

The workaround:

1. Leave vagrant hanging
2. Edit /etc/exports to remove the /System/Volumes/Data prefix
3. sudo nfsd restart
4. Now just wait. Vagrant's attempt to mount the shared folder will time out and it will retry with the correct exports.

Vagrant up will unwedge as soon as the shared folder mounts.

[nikunjkotecha]

IMO this deserves a separate ticket and proper fix.

[charlieartist]

For those still having issues on Catalina, there may some things to try besides workarounds:

1. Update Xcode, then xcode-select --install and brew upgrade
2. Under System Preferences > Security & Privacy > Files and Folders. Enable access for your VM environment.
3. Get a new exports file to generate by running: sudo mv /etc/exports /etc/exports_old
4. Run vagrant up

[evancalkins]

For those still having issues on Catalina, there may some things to try besides workarounds:

1. Update Xcode, then xcode-select --install and brew upgrade
2. Under System Preferences > Security & Privacy > Files and Folders. Enable access for your VM environment.
3. Get a new exports file to generate by running: sudo mv /etc/exports /etc/exports_old
4. Run vagrant up

I ran through these 4 suggestions and it solved the problem for me. Thanks!

[simkimsia]

What does it mean to "enable access for your VM environment"?

[OZZlE]

I went back to vagrant 2.2.5 and used the default virtualbox mounting (nfs wouldn't work).

I had to destroy + restart mac before vagrant became accessible from outside though.. I found it by chance here: https://stackoverflow.com/a/28337424/846348

It seems like latest Catalina doesn't want the /System/Volumes/Data/ prefix like 2.2.6 introduces as a fix in the first place.. good damn Apple. Or could it be related to that I am not using zsh?

[mpukit]

@OZZlE I'm not using zsh and i'm still having issues with the Vagrant 2.2.6 fix. I'm ready to roll back now. Where did you set Virtualbox mounting?

[OZZlE]

@mpukit It's in the Vagrantfile search for 'synced_folder' and remove nfs, ours looks like, I added # to comment it out,

  machine.vm.synced_folder "/Volumes/Projects/apache", "/vagrant", 
    automount: true # added so virtualbox mounting happens directly
    # nfs: true, # disabled nfs
    # nfs_udp: false # disabled nfs
    # mount_options: ['rw', 'tcp', 'fsc', 'actimeo=2']  # disabled nfs

It's slow as fuck though but at least it works :)

[mpukit]

Hmm, I must have a different Vagrant file - I don't have the nfs: true option

[Girres]

It's possible to get this working without moving files out of Documents.

Add /sbin/nfsd in System Preferences > Security & Privacy > Privacy > Full Disk Access
RE: https://github.com/hashicorp/vagrant/issues/10961#issuecomment-541070382
Brilliant!!

[OZZlE]

Add /sbin/nfsd in System Preferences > Security & Privacy > Privacy > Full Disk Access

This doesn't seem to help when using a separate (case sensitive) partition that you use as the shared folder :'(

[OZZlE]

Hmm, I must have a different Vagrant file - I don't have the nfs: true option

I guess either it's being set programmatically somehow (?) via yaml files maybe? or perhaps you aren't even using nfs?

I had issues even without nfs that the box started and everything inside it seemed to work properly but I couldn't access it from outside, the solution that worked for me then was to do vagrant destroy, restart MAC and vagrant up again (this will remove any database or files that aren't stored in your host, so backup those if need be first)

[lbabayigit]

I updated my osx to 10.15.1 and now im getting a mount timeout error.

`The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

mount -o vers=3,udp,actimeo=2,nolock,fsc x.x.254.1:/System/Volumes/Data/Users/xxx/dir /opt/xxx/dir

Stdout from the command:

Stderr from the command:

mount.nfs: Connection timed out`

[ErwinSteffens]

Same here on macOS 10.15.1. Getting a mount time-out.

Seems like the nfsd is crashing:

default 19:40:02.928287+0100    nfsd    open(/etc/exports): No such file or directory (2)
default 19:40:06.133783+0100    sudo        root : TTY=ttys003 ; PWD=/Users/erwin/Workspace/devvm ; USER=root ; COMMAND=/sbin/nfsd restart
default 19:40:13.609863+0100    ReportCrash Parsing corpse data for process nfsd [pid 46376]
default 19:40:15.009535+0100    ReportCrash Saved crash report for nfsd[46376] version 136.40.2 to nfsd_2019-11-07-194015_Erwins-MacBook-Pro.crash
default 19:40:15.013571+0100    ReportCrash Removing excessive log: nfsd_2019-11-07-104554_Erwins-MacBook-Pro.crash

Any ideas on a workaround?

[lbabayigit]

i manage to fix this issue @ErwinSteffens can you post your mount section of your Vagrantfile?

[naprirfan]

Hello @lbabayigit , I also have this mount time-out problem. How did you solve it?

This is my mount section:

# Project root
vagrant_root = File.dirname(__FILE__)

# Mount
$mount_type = 'nfs'
if OS.linux?
  $mount_options = %w(rw vers=4 tcp noacl nolock actimeo=1)
  if OS.wsl?
    $mount_type = 'virtualbox'
    $mount_options = []
  end
else
  $mount_options = %w(rw vers=3 tcp actimeo=1)
end

# Folders
  config.vm.synced_folder "#{vagrant_root}", "/vagrant", type: $mount_type, mount_options: $mount_options, automount: true
  config.vm.synced_folder "#{vagrant_root}/../projectA", "/ww-shop", type: $mount_type, mount_options: $mount_options, automount: true 
  config.vm.synced_folder "#{vagrant_root}/..", "/projectB", type: $mount_type, mount_options: $mount_options

[kwn]

@lbabayigit

Any hints?

[paco3346]

I too am having the timeout issue on Catalina with 2.2.6

If I SSH into my VM I can run

sudo mount 192.168.99.1:/System/Volumes/Data/Volumes... /some/local/path

and it works perfectly. It appears that the core of NFS works but there's some other detail vagrant must add to the mount command.

[ErwinSteffens]

@lbabayigit Here it is. No special config I think:

  config.vm.synced_folder ".", "/vagrant", type: "nfs"
  config.vm.synced_folder "../com.folder1", "/opt/folder1", type: "nfs"
  config.vm.synced_folder "../com.folder1", "/var/www/folder2", type: "nfs"
  config.vm.synced_folder "../com.folder3", "/var/www/folder3", type: "nfs"
  config.vm.synced_folder "..", "/workspace", type: "nfs"

I have updated the logs in my post above as I copied/pasted the wrong nfsd logs.

[lbabayigit]

@ErwinSteffens you first need to change your folder structure to /System/Volumes/Data/
i.e.
config.vm.synced_folder ".", " /System/Volumes/Data/vagrant", type: "nfs"

[kenji21]

@lbabayigit Here it is. No special config I think:

  config.vm.synced_folder ".", "/vagrant", type: "nfs"
  config.vm.synced_folder "../com.folder1", "/opt/folder1", type: "nfs"
  config.vm.synced_folder "../com.folder1", "/var/www/folder2", type: "nfs"
  config.vm.synced_folder "../com.folder3", "/var/www/folder3", type: "nfs"
  config.vm.synced_folder "..", "/workspace", type: "nfs"

I have updated the logs in my post above as I copied/pasted the wrong nfsd logs.

Have you tried with a single nfs mount ? maybe having multiple ones can cause the issue

[ErwinSteffens]

@kenji21 Tried a single NFS mount and this works.

Hmm, is there a way to work around this?

[OZZlE]

It seems like restarting your mac installs patches ( https://semver.org/ ) without even informing us about it. So what I did was. 1) vagrant up => if fail 2) restart Mac and over again during a couple of days. One day it finally worked with nfs, I haven't restarted my computer since and I don't dare :D

It is now working without the prefix /System/Volumes/Data/, it wasn't allowing this prefix.

I have:

$ system_profiler SPSoftwareDataType
System Version: macOS 10.15 (19A602)

It's Apple's fault but why the heck is this issue closed here? Shouldn't the maintainers try to talk with Apple that they have destroyed Vagrant and that this is still an issue depending on which patch version of Catalina you use?

[ErwinSteffens]

@OZZlE I think the original problem was that absolute paths should be used /etc/exports on macOS 10.15. This was fixed in vagrant 2.2.6, so the issue was closed.

The issue with the time-outs is only occurring for us on macOS 10.15.1 (19B88).

[OZZlE]

@ErwinSteffens alright but in 19A602 using the absolute path /System/Volumes/Data/[something] doesn't work, it refuses to mount then even when I try manually, when I removed this prefix/absolut path it works fine. So I had to use vagrant version 2.2.5 for vagrant to work at all for me in Catalina.

Probably Apple doesn't push out system updates to all users in the world at the same time. So even when I though I had the latest Catalina due to accepting all updates, I did infact not have it and no possibility to get it other than wait. Meanwhile this was fixed in vagrant 2.2.6 for the latest Catalina but not backward compatible with the latest Catalina that I could actually install..... or that some users had installed...

[Nilz11]

For me it works when i have one NFS mount, but if I have two mounts it hangs at "Mounting nfs folders"

[kwn]

Thanks @Nilz11. This temporarily solved the issue. We used NFS for mounting multiple directories that were not performance critical (ansible scripts etc.). We removed them the NFS option, so now they use the native VirtualBox folder share capability. We use NFS to keep the performance high for the application code only.

[rfay]

Just an extreme oddity of macOS Catalina: It seems that the ~/Documents directory cannot work with NFS, even though any other directory works (assuming homedir shared). It always shows as a stale filehandle. I have no idea why this would be.

Edit: The ~/Documents NFS problem
is solved by https://github.com/hashicorp/vagrant/issues/10961#issuecomment-541070382 - You have to grant "Full Disk Access" to /sbin/nfsd

[mpukit]

I fought this for over a week and in the end the only solid solution was downgrading my OS from Catalina. She's a real nuisance.

[ErwinSteffens]

What is the process to get this fixed? Is there anything vagrant can do to work around this?

Should we file a bug report at Apple (https://developer.apple.com/bug-reporting/)?

[kwn]

@ErwinSteffens

I already did (FB7433067) however my report doesn't explain that the issue occurs only when there are multiple NFS mounts. It would be great if you could also report it. More reports they get - higher chance they will fix it.

BTW - 2 days ago they announced 10.15.2 (19C39d) release, but they didn't mention NFS problem.

[JensBourgeois]

I can confirm the multiple NFS mounts problem.

I managed to fix it by manually changing the mapping in the /etc/exports file.
The cause seems to be by 2 exports on the same line
"/System/Volumes/Data/Users/xxx/Documents/Vagrant/xxx-shop-m2-vagrant/data/web/magento2 /System/Volumes/Data/Users/xxx/Documents/Vagrant/xxx-shop-m2-vagrant/data/web/magento2/config/nginx -alldirs -mapall=501:20 172.28.128.3"

I removed the nginx mapping part and now it's working again.
So multiple NFS exports seem to work, but not when they are combined on 1 line.

FIXED

# VAGRANT-BEGIN: 501 41963a7b-784f-435c-8669-8709e7ccf0eb
/System/Volumes/Data/Users/xxx/.composer -alldirs -mapall=501:20 172.28.128.3
/System/Volumes/Data/Users/xxx/Documents/Vagrant/xxx-shop-m2-vagrant/data/web/magento2 -alldirs -mapall=501:20 172.28.128.3
# VAGRANT-END: 501 41963a7b-784f-435c-8669-8709e7ccf0eb

BEFORE (BROKEN)

# VAGRANT-BEGIN: 501 432726b2-afcf-4aad-b58c-aa7c17a1d797
/System/Volumes/Data/Users/xxx/.composer -alldirs -mapall=501:20 172.28.128.3
/System/Volumes/Data/Users/xxx/Documents/Vagrant/xxx-shop-m2-vagrant/data/web/magento2 /System/Volumes/Data/Users/xxx/Documents/Vagrant/xxx-shop-m2-vagrant/data/web/magento2/config/nginx -alldirs -mapall=501:20 172.28.128.3
# VAGRANT-END: 501 432726b2-afcf-4aad-b58c-aa7c17a1d797

[OZZlE]

I was getting stuck with Inappropriate ioctl for device now instead after my Mac crashed,

this fixed it! https://github.com/hashicorp/vagrant/issues/1941#issuecomment-42274573

[findthebug]

Just an extreme oddity of macOS Catalina: It seems that the ~/Documents directory cannot work with NFS, even though any other directory works (assuming homedir shared). It always shows as a stale filehandle. I have no idea why this would be.

Edit: The ~/Documents NFS problem
is solved by #10961 (comment) - You have to grant "Full Disk Access" to /sbin/nfsd

this solves my issue in Catalina 10.15.1

[alextsoi]

Just an extreme oddity of macOS Catalina: It seems that the ~/Documents directory cannot work with NFS, even though any other directory works (assuming homedir shared). It always shows as a stale filehandle. I have no idea why this would be.
Edit: The ~/Documents NFS problem
is solved by #10961 (comment) - You have to grant "Full Disk Access" to /sbin/nfsd

this solves my issue in Catalina 10.15.1

Yes I added full disk access on both Virtual box and nfsd

[mfn]

A co-worker tried thus but then ran into troubles with our setting for nfs, e.g. locking didn't work.

In the end we advised everyone _not_ to use those suddenly special folders for vagrant machines anymore 🤷‍♀️

[TVLester98]

The Full Disk Access for nfsd did the trick for me! My vagrant environment is now running.
I still have an issue with the Latte (template engine) cache. There is an filemtime() error, maybe because of the NFS paths?

[toabi]

If you get locking issues, try moving your vagrant project to some folder like ~/code and not have it in ~/Documents or ~/Desktop. That solved some locking issue for us.

[TVLester98]

If you get locking issues, try moving your vagrant project to some folder like ~/code and not have it in ~/Documents or ~/Desktop. That solved some locking issue for us.

Thanks!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.