Vagrant: NFS permission problem on MacOS Sierra

[crashev]

Vagrant version

Vagrant 1.9.0

Host operating system

MacOs Sierra 10.12.1

Guest operating system

Linux - Debian 8.5 - mokote/debian-8 (version 8.5)

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.synced_folder "~/SparkSoftware/applications", "/srv", type: "nfs", :linux__nfs_options => ["rw,no_subtree_check,no_root_squash"]
  config.vm.box = "spark"
  config.vm.network "private_network", ip: "192.168.33.10"
end

Debug output

vagrant@lb:/srv/octopus$ ls -ld README.md
-rw-r--r-- 1 501 dialout 221 Nov 30 15:59 README.md
vagrant@lb:/srv/octopus$ ls -ld app/config/parameters.yml
-rw-r--r-- 1 501 dialout 2905 Nov 28 14:30 app/config/parameters.yml
vagrant@lb:/srv/octopus$ rm README.md
vagrant@lb:/srv/octopus$ rm app/config/parameters.yml
rm: cannot remove ‘app/config/parameters.yml’: Permission denied
vagrant@lb:/srv/octopus$

And on MacOS, file /etc/exports contains:

# VAGRANT-BEGIN: 501 9f80a7df-67dc-4bee-b0c6-38a21aab06a2
"/Users/user/SparkSoftware/applications" 192.168.33.10 -alldirs -mapall=501:20
# VAGRANT-END: 501 9f80a7df-67dc-4bee-b0c6-38a21aab06a2

Expected behavior

I should be able to remove any file mounted in nfs share directory.

Actual behavior

I can only delete some files - seems like only in root directory, anything below can
not be deleted.

Steps to reproduce

Everything is in debug.

References

• https://github.com/mitchellh/vagrant/issues/6360 - last comments

[lightster]

I'm getting very similar behavior with CentOS 6 and 7 guest OS operating systems. My host operating system is also macOS Sierra. I've tried downgrading/upgrading Vagrant and Virtualbox.

The problem seems to be made worse by restarting my host/physical machine—everything on the shared partitions becomes undeletable as far as I can tell.

I have a few interesting ways to make the files deletable again:

• If I do an ls -lR from the host machine of the files/directories I'm trying to delete, I can then delete the files via the guest machine.
• If I edit the file from the guest machine, then I can delete the files from the guest machine! (Yes, that's right, the files are writeable from the guest machine even though they are not deletable)

I'm not sure this is really a Vagrant issue so much as a macOS Sierra nfsd issue, but hopefully the Vagrant community might be able to put information together to come up with a bug report and/or workaround.

[whizkid79]

i can confirm @lightster 's observations. same issues and solutions here with running ubuntu on parallels.

[joshuasmickus]

I have this issue too, CentOS 6.8 and Mac OS Sierra.

[kazysgurskas]

Confirmed with various Vagrant and Virtualbox versions and several guest OSes. Disabling NFS lookupcache seems to work, but the performance is unbearable. Recursive listing and then deletion seems to work as well. I found it the fastest to do find -delete twice to delete files.

[crashev]

I wonder when someone from Vagrant team will lookup into this, can't properly work on OSX with Vagrant

[dominikzogg]

@crashev it seems, that its a OSX Bug, not a vagrant ones. Until now there are no known working workarounds, with an acceptable performance

[araines]

@lightster mentioned a workaround - from your host machine (i.e. macOS) if you perform the following operation in the root of the shared folder:

ls -lR > /dev/null

Then that appears to refresh all the NFS links, meaning you are able to operate as normal inside Vagrant again. It isn't ideal, but I've been doing that pretty successfully for a while now.

[crashev]

@araines just chcked ls -lR but it does seem not to work for me, still a lot of permission denied when trying to delete folder like vendors/ in my php project, but found that's because ls -lR does not go into directories with dot like .git in this case it's better to use:

ls -alR > /dev/null

[joshuasmickus]

I tried quite a few things, and ended up removing vagrant (with instructions from the website) and installing 1.9.1 with virtualbox 5.1 and that worked for me - now my setup works as expected, not sure if that helps anyone.

[kazysgurskas]

@joshuasmickus could you please elaborate? As far as I know, vagrant 1.9.1 supports virtualbox up to 5.0, not 5.1

[arendjr]

After spending almost a day trying to debug this thing, it seems I have a "solution". The reason we kept on searching was that I was having this exact issue as described in this bug report, but a colleague of mine, also on Sierra, was having no problems: there had to be something more going on.

So, last thing we tried that seems to have fixed it: On my host machine the directories that were shared were owned by the group "staff" whereas on my colleague's machine the shared directories on the host were owned by group "wheel". So I've changed the group owning the shared directories on my machine to "wheel" and now it works.

I don't claim to fully understand why this matters, because in /etc/exports I see the NFS exports map the group to group ID 20, which is actually "staff" on my machine. But I'm done with this, it works and I'm not touching it any further... I hope someone here can confirm whether it works for them though!

Edit: Bummer, it doesn't appear to be entirely solved. Still getting the problem sometimes, even though somehow the situation does appear to have improved for some amount of time...

[jfbibeau]

@kazgurs - It definitely supports 5.1.x. From the docs page:

The VirtualBox provider is compatible with VirtualBox versions 4.0.x, 4.1.x, 4.2.x, 4.3.x, 5.0.x, and 5.1.x.

@joshuasmickus For what it's worth, I'm still seeing the issue on my setup with VirtualBox 5.1.2, Vagrant 1.9.1.

I think the problem is definitely intermittent so we have to be careful when seeing the problem go away for a while. Too bad the workaround with the group didn't work out either :/ Maybe the effect of doing a chown on the files were having similar effects to the ls -lR command.

[whizkid79]

@arendjr can you please investigate further what's different to your colleagues's machine?

Right now I'm running a ls every minute on a cron on my host, which hides the issue completely and I haven't noticed any performance side effects on this. But it is quite a dirty workaround.

[jfbibeau]

Been trying to debug this further into the NFS all morning. I can use rpcdebug on the client side (guest/linux), and I can see that the server seems to return a -13 error code (NFS3ERR_ACCES):

Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/676493), mask=0x81, res=-10
Dec 14 12:14:06 nsp-latest kernel: NFS call  access
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_update_inode(0:39/676493 fh_crc=0x29063737 ct=2 info=0x27e7f)
Dec 14 12:14:06 nsp-latest kernel: NFS reply access: 0
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/676493), mask=0x1, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_lookup_revalidate(/jf-sandbox) is valid
Dec 14 12:14:06 nsp-latest kernel: NFS call  access
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_update_inode(0:39/28952869 fh_crc=0xf6db0f64 ct=1 info=0x27e7f)
Dec 14 12:14:06 nsp-latest kernel: NFS reply access: 0
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/28952869), mask=0x1, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_lookup_revalidate(jf-sandbox/build) is valid
Dec 14 12:14:06 nsp-latest kernel: NFS call  access
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_update_inode(0:39/29802703 fh_crc=0x6720be52 ct=1 info=0x27e7f)
Dec 14 12:14:06 nsp-latest kernel: NFS reply access: 0
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/29802703), mask=0x1, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_lookup_revalidate(build/classes) is valid
Dec 14 12:14:06 nsp-latest kernel: NFS call  access
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_update_inode(0:39/29802705 fh_crc=0x42ffaad2 ct=1 info=0x27e7f)
Dec 14 12:14:06 nsp-latest kernel: NFS reply access: 0
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/29802705), mask=0x1, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_lookup_revalidate(classes/main) is valid
Dec 14 12:14:06 nsp-latest kernel: NFS call  access
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_update_inode(0:39/29802706 fh_crc=0x9c5681b8 ct=1 info=0x27e7f)
Dec 14 12:14:06 nsp-latest kernel: NFS reply access: 0
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/29802706), mask=0x1, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_lookup_revalidate(main/com) is valid
Dec 14 12:14:06 nsp-latest kernel: NFS: revalidating (0:39/29802710)
Dec 14 12:14:06 nsp-latest kernel: NFS call  getattr
Dec 14 12:14:06 nsp-latest kernel: NFS reply getattr: 0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_update_inode(0:39/29802710 fh_crc=0x6ebbe361 ct=1 info=0x27e7f)
Dec 14 12:14:06 nsp-latest kernel: NFS: (0:39/29802710) revalidation complete
Dec 14 12:14:06 nsp-latest kernel: NFS: dentry_delete(main/com, 10808cc)
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/676493), mask=0x81, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_lookup_revalidate(/jf-sandbox) is valid
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/28952869), mask=0x81, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_lookup_revalidate(jf-sandbox/build) is valid
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/29802703), mask=0x81, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_lookup_revalidate(build/classes) is valid
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/29802705), mask=0x81, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_lookup_revalidate(classes/main) is valid
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/29802706), mask=0x81, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: revalidating (0:39/29802710)
Dec 14 12:14:06 nsp-latest kernel: NFS call  getattr
Dec 14 12:14:06 nsp-latest kernel: NFS reply getattr: 0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_update_inode(0:39/29802710 fh_crc=0x6ebbe361 ct=1 info=0x27e7f)
Dec 14 12:14:06 nsp-latest kernel: NFS: (0:39/29802710) revalidation complete
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_lookup_revalidate(main/com) is valid
Dec 14 12:14:06 nsp-latest kernel: NFS call  access
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_update_inode(0:39/29802710 fh_crc=0x6ebbe361 ct=1 info=0x27e7f)
Dec 14 12:14:06 nsp-latest kernel: NFS reply access: 0
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/29802710), mask=0x24, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: open dir(main/com)
Dec 14 12:14:06 nsp-latest kernel: NFS: readdir(main/com) starting at cookie 0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_do_filldir() filling ended @ cookie 2; returning = 0
Dec 14 12:14:06 nsp-latest kernel: NFS: readdir(main/com) returns 0
Dec 14 12:14:06 nsp-latest kernel: NFS: readdir(main/com) starting at cookie 2
Dec 14 12:14:06 nsp-latest kernel: NFS: readdir(main/com) returns 0
Dec 14 12:14:06 nsp-latest kernel: NFS: dentry_delete(main/com, 10808cc)
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/676493), mask=0x81, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_lookup_revalidate(/jf-sandbox) is valid
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/28952869), mask=0x81, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_lookup_revalidate(jf-sandbox/build) is valid
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/29802703), mask=0x81, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_lookup_revalidate(build/classes) is valid
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/29802705), mask=0x81, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_lookup_revalidate(classes/main) is valid
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/29802706), mask=0x81, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_lookup_revalidate(main/com) is valid
Dec 14 12:14:06 nsp-latest kernel: NFS: permission(0:39/29802706), mask=0x3, res=0
Dec 14 12:14:06 nsp-latest kernel: NFS: rmdir(0:39/29802706), com
Dec 14 12:14:06 nsp-latest kernel: NFS call  rmdir com
Dec 14 12:14:06 nsp-latest kernel: NFS: nfs_update_inode(0:39/29802706 fh_crc=0x9c5681b8 ct=1 info=0x7feff)
Dec 14 12:14:06 nsp-latest kernel: NFS reply rmdir: -13
Dec 14 12:14:06 nsp-latest kernel: NFS: dentry_delete(main/com, 10808cc)

Unfortunately, I can't for the life of me figure out how to use the equivalent of rpcdebug on osx. I've tried upping the verbosity of nfsd on osx but it doesn't go anywhere as verbose as on linux using rpcdebug... No idea how I can look into this further on the server side since there's no logging available, but that looks like where the problem would lie...

[joshuasmickus]

Srana at my work suggested the following:

Right click on the folder which has the permissions error, and then click "Get info". At the bottom, give it "Read & write" access for "Everyone".

This should solve the permissions error

[arendjr]

@joshuasmickus I can confirm I'm having some success by doing that... but I'm holding my breath whether it'll last this time. Fingers crossed!

@whizkid79 I wish I could... My colleague and I are running the exact same virtual machine image now, and we cannot find any further changes on our hosts... Right now I'm sincerely out of ideas of what to try next. I can only hope Joshua's solution will last for now...

[arendjr]

One more update: My colleague has suddenly gotten the problem as well. I'm starting to believe no Sierra users are truly immune here...

And @joshuasmickus' workaround was also only a temporary relief...

[kazysgurskas]

@arendjr it was found in the original issue (#6360) the problem lies within NFS lookupcache. Changing permissions and listing files are just temporary workarounds, as it does refresh the cache. It would be nice if somebody with better understanding of NFS mechanism would dig deeper.

[panique]

Same here, i can reproduce the problem on my own local boxes but also on totally new ones (loading any vagrantfiles from any projects on the web, vagrantfiles that worked perfectly for years, and now they fail due to NFS issue.)

[matleh]

One workaround is to use unfs3d instead of the nfsd server included with MacOS.

brew install unfs3
sudo nfsd stop
sudo unfsd -e /absolute/path/to/some/exports

The exports file has another syntax then for the "normal" nfsd. I use something like:

/Users 192.168.0.0/16(rw,anonuid=501,anongid=20,all_squash)

[rozagh]

@matleh can you also put the vagrantfile command?

[toonvdn]

Spilled couple days on this, but I managed to get it working (for now at least):

• vagrant destroy
• remove the user data folder (~/.vagrant.d/)
• remove the vagrant project folder (.vagrant)
• vagrant up

EDIT: Nope... Temporary fix...

[matleh]

@rozagh actually, I don't use Vagrant - this problem also occurs with docker-machine-nfs.
I already moved away from my "solution" again, because it seemed like performance of the unfs3 setup was unbearable.

I am back to stock nfsd and running ls -lr > /dev/null on the host from time to time...

[oleyka]

This thread might be related: https://discussions.apple.com/thread/7760267?start=0&tstart=0

[pollosp]

For sure, this isn't going to work for all cases, but we are using this plugin https://github.com/Learnosity/vagrant-nfs_guest as work around.
Agree that it could be a MacOS Sierra problem.

[blissdev]

@pollosp are you using it on Sierra with success?

[pollosp]

@blissdev yes, with sierra

[rozagh]

I tried using nfs_guest but I still have file access issue. Did you just enabled the plugin and change the vagrant file to fix the issue or was there any additional steps?

[rozagh]

Also after I tried reloading my vm again, all my directory is removed! Seems like the nfs_guest is moving the files somewhere but if your mounted directory is the same as where you have VM defined all the data will be inaccessible.

[lantz]

I've seen similar problems deleting files from an Ubuntu client on VMware, mounting an NFS volume exported from macOS Sierra/10.12.1 NFS server. I see it frequently when trying to check out branches using git. I've noticed that if I create a file (e.g. touch some/dir/foo) in the offending directory, then I can remove the un-removable file.

I've only seen this with Sierra/10.12 and higher. I never ran into the problem on 10.11 and earlier. I submitted a bug report with Apple but they say they have received no other reports of this type and couldn't reproduce it, so I'd recommend submitting a bug if you haven't already on bugreport.apple.com.

[dominikzogg]

@lantz a coworker of mine already submitted a bug report about 3 weeks ago, so they received reports, but it seems they aren't be able to handle it

[lantz]

Submitted radar id 28927426 on bugreport.apple.com

Glad that I'm not the only one seeing NFS bugs on Sierra - hopefully Apple can fix it! They did ask me for additional information so hopefully someone is actually looking into it.

[honzabazant]

Just leaving my 2 cents here...
I also ran into the same nfs permission issues. I use Docker with macOS as host, some directories are mounted in the guest via nfs. In macOS 10.12 everything was fine for me but troubles began when I upgraded to 10.12.1. Current version 10.12.2 still has the problem. I also tried to reinstall macOS but it did not help.

I also used Wireshark on macOS to capture NFS communication - nfsd returns NFS3ERR_ACCES when I try to rm a file in the guest, but when I touch a temporary file in the same directory and then I use rm again it returns NFS3_OK (extactly the same behaviour as lantz wrote). I did not find any differences between those two NFS calls but I am not skilled in such low level debugs and I do not know much about NFS internals.

Although this bug is a pain, it's good to know that I am not alone. Of course I will submit a bug report to Apple.

[joshlopes]

I can confirm that on Sierra, setting "Everyone" to "Read and Write" fixed the problem permanently. Probably not the most correct, but it helps!

Edit:
Nope it doesn't fix permanently i need to change it back to read and then to read and write to apply the changes, annoying!!

[jamestyrrell]

+1 This bug sucks

[ascotan]

I have the exact same permissions problem, however, I will note that If I run this in VMWare with the VMware-Fusion plugin from vagrant, the NFS mount works correctly.

I have a feeling that this may in fact be a virtualbox problem related to the need for a private network. (Vagrant 1.9.1 with macOS Sierra)

1. switch the provider to vmware_fusion
2. get rid of the private newtwork IP
3. set folder line to "config.vm.synced_folder ".", "/vagrant", type: "nfs""
4. remove .vagrant directory
5. reboot
6. vagrant up

NFS now works in vmware.

[gsong]

@ascotan For me the problem is inconsistent, have you had constant success with starting/stopping the same VMWare box and have the nfs mount continue to work? With VirtualBox, sometimes it works, sometimes it doesn't. I'm trying out @joshuasmickus's suggestion to see if it's a medium term workaround, vs. having to do ls -aLR.

[jfbibeau]

FYI @joshlopes workaround does not work longterm, problem always comes back. Only thing that reliably works in this whole thread is the ls -aLr workaround.

[n3o77]

I'm also having the same problem with parallels, so probably not related to virtualbox.

[mttjohnson]

We are also seeing the problem at our office with several developers running into the issue spastically. We haven't narrowed down how to consistently reproduce it, and have used "ls -aLr" from the host as a temporary workaround.

I'm running MacOS Sierra 10.12.2 (16C67)
Vagrant 1.8.6
VirtualBox 5.0.28 r111378
VM OS - CentOS release 6.8 (Final)
VM NFS Client Flags: rw,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=10.10.10.1,mountvers=3,mountport=795,mountproto=udp,local_lock=none,addr=10.10.10.1

[mtarvainen]

I filed a bug report to Apple too about two weeks ago. Haven't still heard anything, it is received though.

You might find following one-liner as useful workaround:

$ (crontab -l 2>/dev/null; echo "*/5 * * * * find $HOME -type f -name Vagrantfile -not -path '*/\.*' -exec dirname {} \; | xargs ls -aLR > /dev/null 2>&1") | crontab -

That inserts cronjob that detects directories that have Vagrantfile and runs ls -aLR command in root of every found directory. Command is timed to run every 5minute.

[joshlopes]

I just let the window "get info" of the folder i want to refresh permissions open, so far no issues! :)

[peterdragon]

Another workaround is to use sshfs plugin for vagrant
https://github.com/dustymabe/vagrant-sshfs
Slower than NFS but it works

$ vagrant plugin install vagrant-sshfs
change the Vagrantfile
config.vm.synced_folder "/Users/someuser/shared", "/home/vagrant/shared", type: "sshfs"
$ vagrant reload

[jfbibeau]

Opened apple bug a few weeks ago (30347908), and they today closed-dup'd it to 28927426 from @lantz which is still Open, so hopefully they are still investigating.

[EvilHacker]

I am having the same problems. I noticed that the problems started after an OS X update. I don't remember exactly which update it was but it was before November 2016, so likely it was the 10.12.1 Update that introduced the problem.

I've tried reverting to an older NFS server executable but it did not help.
So I think the problem is caused by filesystem changes in OS X 10.12.

See the section titled Apple File System, it looks suspicious:
https://developer.apple.com/library/prerelease/content/releasenotes/MacOSX/WhatsNewInOSX/Articles/OSXv10.html

https://arstechnica.com/apple/2016/09/macos-10-12-sierra-the-ars-technica-review/7/
Says APFS changes are coming in 2017 but I think some changes were already made in an earlier update.

[EvilHacker]

Hey jfbibeau. I'd like to put some more pressure on Apple to fix this problem.
Where do I find the bugs you reference 28927426 and 30347908 ?

[jfbibeau]

https://bugreport.apple.com/ - but it looks like all bugs are private so you can't really see other people's reports, sadly.

[SAN1TAR1UM]

I ran into the same issues after upgrading to 10.12.3 today. I was able to get around the issue by not using nfs.
I changed:
config.vm.synced_folder "./magento", "/var/www/magento", id: "vagrant-root", type: 'nfs'
to:
config.vm.synced_folder "./magento", "/var/www/magento/", :mount_options => ["dmode=777", "fmode=777"]

[alibad]

@SAN1TAR1UM thanks. That finally did it for me.

[joshlopes]

anything other than nfs is slower :(

[btalbot]

The only workable solution for me so far is to keep using vagrant 1.8.7 which seems to work fine with NFS on OSX 10.12.3.

[jholloway7]

It works on 1.8.7? I'm using 1.7.4 and it's broken. Just curious, because I thought this was a lower-level problem in NFS/OSX that would affect all versions of Vagrant.

[bigboca]

My previously working 1.7.4 stopped working after a Sierra upgrade. It started prompting for an admin password 4 times regardless if the password was right. I have upgraded to 1.9.2 and now use the 'Get Info' fix suggested by @joshuasmickus and it seems to be working OK for now although it still asks for the password 4 times; it's almost un-usable for web development work without NFS enabled

[joshuasmickus]

Before commenting please read my comment above, we have had about 20+ engineers upgrade to Sierra and some of them have had this problem, my solution above has worked for all of them.

Posting here for brevity:

1) Right click on the folder which has the permissions error (or a parent folder)
2) Click "Get info"
3) At the bottom "Sharing & Permissions" click the "lock" symbol
4) Select "Read & write" access for "Everyone" and other users.
5) Click the cog
6) Select "Apply to enclosed items..."
7) Wait for permissions to apply...
8) Click the lock
9) ????
10) PROFIT!

It is worth mentioning this has happened to me a couple of times without me knowingly undoing it, although doing the above steps again rectified it :)

Cheers

[cloakedninjas]

Sorry @joshuasmickus this hasn't helped - problem came back a day later -will need to keep ls -la each day

[ed-at-work]

@joshuasmickus That's not a real fix, unfortunately. The problem is in the OS X kernel and won't be fixed unless Apple acknowledges the problem and fixes it. The only real solution otherwise is to stop using a VM for development and install something like MacPorts or Homebrew and install your tools in OS X.

[ed-at-work]

Any news on whether or not this is fixed in OS X 10.12.4 that was released today?

[tautrimas]

Any news on whether or not this is fixed in OS X 10.12.4 that was released today?

Same problem @ed-at-work.

[vmendoza]

I run into problems where I could not run "npm install" on vagrant, after the latest macOS update. removing .vagrant from the VM home and .vagrant.d from $HOME fixed it. Seems like vagrant remembers filesystem information and now with the OS update, it could not work with the old info.

[ev0rtex]

@vmendoza Is it still working for you since deleting that? I still have the problem after a completely clean install on a new 2016 MBP running the latest 10.12.4 macOS update.

[vmendoza]

It worked once. Then I had the problem again. Then I spent some hours tracking the issue down and it turn out that one of NPM packages was the root of the cause. xml2json created the issue. I replaced it with another package and the problem was solved for good.

[chrispelzer]

@vmendoza
The only "solution" is running ls -alR > /dev/null on your host within the root of the directory that is mounted to the VM or restarting the VM.

I'd bet the NPM package wasn't the issue, if you try to remove the node_modules directory within your VM there are other packages that will cause a permission denied error. The permission denied issue can happen to node_modules, .git, vendor or really any file/folder that is mounted with NFS.

[EvilHacker]

From the Mac host ls -aR > /dev/null works.
From inside the VM, this works as well:

find . -type d \
    -exec touch '{}'/.touch ';' \
    -exec rm -f '{}'/.touch ';' \
    2>/dev/null

[EvilHacker]

If caching is disabled in NFS then everything work fine but you don't get the performance improvement which is the whole point of using NFS.

I believe that one of the latest filesystem changes in Mac OS was the addition of higher resolution timestamps. Previously, file timestamps were to the nearest second now they have sub-second resolution.

I am suspicious that this is causing some problems with the caching in NFS. The higher resolution file times may be rounded to the nearest second.

Does anyone know details of the inner workings of NFS? Could this issue be fixed by a patch to NFS?

[rezonanc]

@EvilHacker could you possibly point to a source which states that the timestamp precision is indeed updated? I've tried to verify this by just using stat -r but failed to do so, maybe there are any other methods to verify this?

[chrisfromredfin]

I'm finding this about nanosecond precision, but that's with the new APFS. I'm having this issue and definitely using HFS+ (Journaled, Encrypted).

http://www.pcmag.com/article/345519/what-macos-sierras-new-apfs-file-system-means-to-you

[rezonanc]

Unfortunately according to Apple beta software agreement I can't share the results publicly. But I guess I'm allowed to say that I've reverted to El Capitan, problem solved.

[mihaimatei]

Filled bug report 33043401 on https://bugreport.apple.com/

8 months passed and this bug is still preventing developers to properly use the most common web development setup, on macOS Sierra which meanwhile had at least 3 versions released up to 10.12.5

[devon-pl]

Looks like 10.12.6 still with the same issue, can anyone confirm ?

[honzabazant]

Yes, the issue is still here, at least for me. :-(

[uberjay]

A little bird told me this has been fixed, but not in time for 10.12. So it'll be in 10.13. Which I think is not too far away. I can't risk trying out the latest public beta on my development machine, unfortunately.

[agentd00nut]

For what it's worth there's a series of steps that I've dropped into a custom remove function for all my bash scripts to work around this issue.

1. cd into the same directory as the file that is causing you trouble.
2. make a new directory. I usually use some vulgar term for kicks.
3. mv the offending file(s) into the new directory.
4. rm -rf the trash directory.

If you take a few moments to write some helper functions / scripts to use instead of "rm" that will automatically do those steps for you it becomes a fairly transparent problem... Obviously not ideal.

Interestingly if you are in a hurry between #1 and #2 you can attempt this list of steps.

1. try to rm the file now that you are in its directory, for some reason this seems to help sometimes.
2. try to touch the file, now try to rm
3. Potentially any of the previously mentioned variants of ls commands might work here as well, but i haven't tested that.

As a final note... Just a cd and a touch usually allow for removal. However I've encountered files that even this didn't work. Using the cd; mkdir d; mv f d; rm -rf d chain has always 100% worked for me.

[joshlopes]

Well i will put here, because is a lot simpler and is working for quite some time. Just open the get info of that folder on your mac and set everyone to read and write. Let this window open in one of your screen corners. You shouldn't have more issues with the folder, if you lock your pc or shutdown then the issue may appear you just go and click on the clog "Apply to enclosed items" and it's working again.

[agentd00nut]

Ah i should have mentioned, sorry, I'm in a weird situation where I can't open any of offending files through finder or on the host machine. In addition its scripted daemons that are trying to do periodic work on temporary files which may encounter the permission errors and so I needed an automated solution.

You're solution is certainly better and more established for a more sane work case.

[hmedney]

Ran into the same issue with Vagrant 1.9.7 and MacOS Sierra 10.12.6. None of the above worked for me, and I ultimately had to do a chmod -R o+w * on the files to make them writeable inside the VirtualBox VM by the vagrant user.

[emileber]

Here's a TL;DR in the form of a Stack Overflow question.

[pascaladriaansen]

Did anyone already try macOS 10.13 to see if it's fixed?

[anttoeknee]

I'm using 10.13, still not working for me, I am using the new APFS though, not tried it with the old HFS.

[robofearth]

Gave it a run on MacOS 10.13.0 (High Sierra) today with APFS, still exists for me. Not sure but didn't seem to be as frequent, with perhaps only 1/3 of the prior number of file removal issues.

Vagrant version

Vagrant 2.0.0

Host operating system

MacOs Sierra 10.13.0

Guest operating system

Linux - Ubuntu 14.04 (via VMware Fusion 8.5.8 + Vagrant VMware plugin)

[BrianGilbert]

I reported this to apple back in developer beta 2 or 3 High Sierra, See https://github.com/hashicorp/vagrant/issues/8957#issuecomment-332175947 for more info

[jfbibeau]

@BrianGilbert That's a different issue than this thread. The problem in this one is files/folders are listed accurately, but randomly users will get Permission Denied while removing a file. All other operations/listing works fine.

The video you posted from High Sierra seems like a much more severe bug! If this particular bug (deletion problem) is still present in High Sierra, and there's a new NFS bug where folders are being randomly duplicated or not listed, this makes NFS pretty much worthless on Mac OS :(

[BrianGilbert]

@jfbibeau ah.. sorry..

[jfbibeau]

@BrianGilbert No worries :) Just wanted to point out that now we have 2 different issues; one that seems High Sierra specific, and one that predates High Sierra.

[cuky23]

We run a development team and two of our staff upgraded to new High Sierra, we are seeing issues with new version of VB and vagrant https://releases.hashicorp.com/vagrant/1.9.8

keen to find out the root cause as back porting MBP OS with disk encrytion is a pain.
We suspect new version of VB. is at fault again.

we see duplicate file refs in the guest os and/or shares failing completly.

We have tested a build using earlier El cap and 1.9.1 this is fine. lucky for me !

[emileber]

@cuky23 See the issue #8788 for the High Sierra file syncing problem. This one is an unrelated issue with permissions that started on Sierra.

[jfbibeau]

For what it's worth, my Apple Bug Report for this specific issue in macOS Sierra had been previously closed as a duplicate. They just added a notification to let me know the original issue that it was dup'd against has just been closed.

Apple Developer Relations
October 18 2017, 4:33 PM
The original report on your issue has been closed recently. Please note that you will not be able to directly view the original report in order to keep its information confidential. 

If you have further questions about this issue, please update your report using the Apple Bug Reporter .

My Bug Report was "30347908". The original bug report was "28927426". @lantz this appears to be yours! Can you update us on what they said? Is a fix available?

[mihaimatei]

Just received notice from Apple that they closed the original bug report 28927426, which mine was a duplicate of. For said confidentiality reason, they leave us in the blinds, not mentioning what their resolution was. This is plain stupid Apple!
@lantz , as you are the original reporter, maybe you can enlighten us, please ?

[lantz]

@mihaimatei Here is the entire text added by Apple when they closed my report 28927426:
"Apple Developer Relations October 8 2017, 6:49 PM
The issue reported in this bug report is fixed but the High Sierra issue is specific to APFS and not the same issue. Please file a new nug for that issue. Thank you. We are now closing this bug report. If you have questions or comments about the resolution, please update your bug report with that information so we can respond."
Hopefully it will show up in a future macOS update? One can only hope.
They had previously asked me to retest in 10.13 High Sierra, but I wasn't able to test it easily since I'm waiting on upgrading until various application issues with APFS are sorted out.

[jfbibeau]

@lantz interesting. No mention or field anywhere indicating the "fixed in" load? Any chance you can update the bug report and ask them which macOS load the fix is in? Thanks for passing the information along!

[lantz]

@jfbibeau Done. Will update in the unlikely event that I receive a response.

[BrianGilbert]

My report was closed without being fixed, they even replied after I'd confirmed it still wanst fixed.

[BrianGilbert]

Actually I'm wrong.. mine is still open:
ID: 32866134
Title: nfs server gives incorrect and inconsistent directory listing
Created: June 20 2017, 3:17 PM for macOS + SDK

[emileber]

@BrianGilbert Looks like you're talking about #8957 which is a different problem than this one.

[BrianGilbert]

@emileber indeed I am.. my bad.

[scottsb]

Reports in #8788 are that that bug is fixed in High Sierra 10.13.2 Beta 2. Can anybody verify if this bug is fixed there as well?

[jfbibeau]

I sure hope so. Upgrading to that load this weekend, will very this one as well.

[jfbibeau]

Upgraded to High Sierra 10.13.2 Beta 2, removed my cronjob that would periodically do an ls -laR as a workaround, and will let it soak running a few clean + builds. It usually takes some random amount of time until this bug manifests, so I'm not calling it fixed yet, but will report back if I see it.

[emileber]

@jfbibeau Have you had the problem since?

[jfbibeau]

@emileber I'm happy to report I haven't seen this in almost 2 weeks of using High Sierra. It's completely fixed!

[uberjay]

I can confirm, too -- I've been using High Sierra (both beta2 and beta3) for a week with zero NFS issues. Woohoo!

[tristanbes]

Is this an OPEN beta ? I mean can we upgrade straight from Sierra to High Sierra beta 3 without a developper account ?

[chrisfromredfin]

I had to sign up for a developer account, but it was free.

[tristanbes]

thanks @chrisfromredfin

So just to be clear, with this beta, is it safe to upgrade or there are still pending issues with High Sierra ?

[phoenixgao]

I have upgraded to beta so far no issues. and it solved this issue...

2017年11月27日 22:47，"Tristan Bessoussa" [e-mail]写道：

thanks @chrisfromredfin https://github.com/chrisfromredfin

So just to be clear, with this beta, is it safe to upgrade or there are
still pending issues with High Sierra ?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/hashicorp/vagrant/issues/8061#issuecomment-347203807,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA0FWpCTmO-CaK4dkUW_tt2ndXsTJ_Zbks5s6st_gaJpZM4LAXYg
.

[sinisilm]

It would seem this issue has come to replace it: #8788

[jfbibeau]

@sinisilm That one is also fixed in High Sierra latest beta.

[chrisfromredfin]

Correct, since High Sierra Beta 2 I have not experience #8788 NOR this one, #8061.

[emileber]

Be careful with High Sierra as there is a major security breach right now: https://www.macrumors.com/2017/11/28/macos-high-sierra-bug-admin-access/

[briancain]

Also, I recommend installing the latest High Sierra patch from Apple, which fixes the security vulnerability that @emileber mentioned: https://support.apple.com/en-us/HT208315

[lantz]

Just got this from Apple - apparently it is fixed in 10.13.1 GM. (I can't test it atm since I'm deferring 10.13 due to APFS compatibility issues.)

This is a courtesy email regarding Bug ID# 28927426.  

Please verify this issue with the macOS High Sierra 10.13.1 GM and update your bug report at https://bugreport.apple.com/ with your results.

macOS High Sierra 10.13.1 GM (17B48)
https://developer.apple.com/download/
Posted Date: Oct 31st, 2017

If the issue persists, please attach a new sysdiagnose captured in the latest build and attach it to the bug report. Thank you.

[pnoeric]

Unclear if this is the same bug as what I am seeing, but it seems close. I am on High Sierra 10.13.3, using nfs (with bindfs extension) and Vagrant to develop.

The problem I have is, when I edit a file on the host machine that is in a folder to sync over nfs to the VM, it takes > 7 seconds for the change to go through, and I get an I/O error on the file while I am waiting.

BUT if I change the file, then immediately do an lson the folder from inside the VM, it updates immediately.

Oddly, if I just do an lson the file itself from inside the VM. That doesn't work. I have to just lsthe folder and then all is good.

Any ideas how I can fix this? I see a bunch of workaround above (like running a cron job to ls the directory every minute or whatever... eek)... but would love to just fix the problem.

[emileber]

@pnoeric it looks like a bug, but it's definitely not related to this issue. You should open a new one.

[chrisfromredfin]

I have been having somewhat similar issues - but mostly just a seemingly large lag between changing a file from the outside and it being detected on the inside - especially with IDEs using atomic writes. If you open a new issue, please post a reference back here.

[joshlopes]

@chrisfromredfin that's another issue well known to mac users - the lag / sync between mounted volumes.

This issue has been fixed on the new MacOS versions, no point in keeping it open. Should be closed.

[chrisroberts]

Closing this up as it _should_ be resolved upstream now. Cheers!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.