Vagrant: NFS File Syncing Slower in MacOS High Sierra 10.13.2

[fishfacemcgee]

Please note that the Vagrant issue tracker is reserved for bug reports and
enhancements. For general usage questions, please use the Vagrant mailing list:
https://groups.google.com/forum/#!forum/vagrant-up. Thank you!

This is building on top of the handful of comments that popped up at the end of GH-8788. If an issue was already created, I apologize, but I couldn't find one via Issue Search. Also, as with GH-8788, it's entirely possible this is something broken by Apple in MacOS.

Vagrant version

Vagrant 2.0.0

Host operating system

uname -a
17.3.0 Darwin Kernel Version 17.3.0: Thu Nov  9 18:09:22 PST 2017; root:xnu-4570.31.3~1/RELEASE_X86_64 x86_64 i386 MacBookPro11,5 Darwin

```
sw_vers -productVersion
10.13.2

### Guest operating system

cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)

### Vagrantfile
```ruby
# -*- mode: ruby -*-
# vi: set ft=ruby :

# Check vagrant version only on up
if ARGV[0] == "up"
    if Gem::Version.new(Vagrant::VERSION) < Gem::Version.new('2.0.0')
      puts "Warning related to running an old version of Vagrant"
      exit
    end
end

Vagrant.configure(2) do |config|
    config.vm.box = # Custom CentOS 7 based box
    config.vm.box_url = # Custom Box URL
    config.vm.box_version = ">= 1504287889"

    # forward agent for ansible access
    config.ssh.forward_agent = true

    config.vm.synced_folder ".", "/vagrant", type: "nfs"

    config.vm.provider "virtualbox" do |vb|
        # Customize the amount of memory on the VM:
        vb.memory = 3072
        vb.cpus   = 2
    end

    github_access_check(config)

    # Ability to override repository URL / branch / tags while provisioning
    provision_url = ENV['PROVISION_URL'] || # custom git repo
    provision_branch = ENV['PROVISION_BRANCH'] || 'master'
    provision_tags = ''
    if ENV['PROVISION_TAGS']
      provision_tags = '--tags ' + ENV['PROVISION_TAGS']
    end

    # Default vagrant machine configured from master
    config.vm.define "default", primary: true, autostart: true do |default|

        default.vm.network "private_network", ip: "192.168.59.103",
            nictype: "virtio"

        if Vagrant.has_plugin?("vagrant-persistent-storage")
            hasUpdatedStoragePlugin = Vagrant.has_plugin?("vagrant-persistent-storage", ">= 0.0.22")
            if hasUpdatedStoragePlugin
                PERSISTENT_STORAGE_LOCATION = File.join(ENV['HOME'], ".vagrant.d/default-web-mysql.vdi")
            else
                # Keep old file location
                PERSISTENT_STORAGE_LOCATION=File.join(File.dirname(__FILE__), "\~/default-web-mysql.vdi")
            end
            if ARGV[0] == "up"
                if hasUpdatedStoragePlugin
                    # Users who started with vagrant-storage-plugin < 0.0.22 and old Vagrantfile
                    oldPersistentStorageFile = File.join(File.dirname(__FILE__), "\~/default-web-mysql.vdi")
                    if File.exists? (oldPersistentStorageFile)
                      move_storage_file(oldPersistentStorageFile, PERSISTENT_STORAGE_LOCATION)
                    end
                    # Users who started with vagrant-storage-plugin = 0.0.22 and old Vagrantfile
                    previousPersistentStorageFile = File.join(ENV['HOME'], "default-web-mysql.vdi")
                    if File.exists? (previousPersistentStorageFile)
                      move_storage_file(previousPersistentStorageFile, PERSISTENT_STORAGE_LOCATION)
                    end
                end
            end

            # Set up persistent /var/lib/mysql that is not destroyed even with vagrant destroy
            default.persistent_storage.enabled = true
            default.persistent_storage.location = PERSISTENT_STORAGE_LOCATION
            default.persistent_storage.size = 5000
            default.persistent_storage.mountname = 'mysql'
            default.persistent_storage.filesystem = 'ext4'
            default.persistent_storage.mountpoint = '/var/lib/mysql'
            default.persistent_storage.volgroupname = 'myvolgroup'
        end

        # for LiveReload on the front-end
        if Vagrant.has_plugin?("vagrant-notify-forwarder")
            config.vm.network "forwarded_port", guest: 35729, host: 35729
        end

        default.vm.provision "ansiblervm", run: "always", type: "shell", inline: <<-SHELL
            rm -rf ~/.ansible/pull
            PYTHONUNBUFFERED=1 ansible-pull \
                --url=#{provision_url} \
                -C #{provision_branch} \
                #{provision_tags} \
                -i inventories/localhost \
                --vault-password-file=/dev/null \
                # Ansible Playbook
        SHELL

        # Enable provisioning with a shell script. Additional provisioners such as
        # Puppet, Chef, Ansible, Salt, and Docker are also available. Please see the
        # documentation for more information about their specific syntax and use.
        default.vm.provision "ansible", run: "always", type: "shell", inline: <<-SHELL
            rm -rf ~/.ansible/pull
            PYTHONUNBUFFERED=1 ansible-pull \
                --url=#{provision_url} \
                -C #{provision_branch} \
                #{provision_tags} \
                -i inventories/localhost \
                --vault-password-file=/dev/null \
                # Ansible Playbook
        SHELL
    end
end

def github_access_check(config)
    config.vm.provision "github_access_check", type: "shell", inline: <<-SHELL
        ssh -T [email protected] -o StrictHostKeyChecking=no || true
    SHELL
end

def move_storage_file(oldLocation, newLocation)
    if File.exists? (oldLocation)
      unless File.exists? (newLocation)
          FileUtils.mv(oldLocation, newLocation) || exit!
          puts "Moved persistent storage share from: ", oldLocation, "to: ", newLocation
      else
          puts "Persistent storage share exists in both old and new location. Please resolve manually"
          puts "Old file: ", oldLocation
          puts "New file: ", newLocation
      end
    end
end

Please note, if you are using Homestead or a different Vagrantfile format, we
may be unable to assist with your issue. Try to reproduce the issue using a
vanilla Vagrantfile first.

Debug output

https://gist.github.com/fishfacemcgee/9bc91d62e00a5f4cf7b1b030dbf37a03

Expected behavior

File changes made on the host machine are synced to the guest machine quickly. On Sierra/HFS+, single file changes are synced immediately (or fast enough to be perceived as immediate).

Actual behavior

File changes take over 20 seconds to sync. Attempting to run ls on the file that changed on the guest machine returns the following error:
ls: cannot access /path/to/file.extension: Input/output error

Steps to reproduce

1. Create a Vagrant machine with NFS sync on High Sierra
2. Modify a file on the Host Machine
3. run ls -l /path/to/file/on/guest/machine from within the guest machine

References

• GH-8788

[dnl-jst]

Having the same problem. Sometimes it helps to save the edited file again and again and hit refresh in the browser, but sometimes it's really dogged. Especially when working on .env files.

[krzysztofbukowski]

I'm having the same issue on my work MacBook Pro. After making a change I get "Forbidden" in the browser and several seconds later I can see the change.

Strangely it seems to be working fine on my iMac 🤔

Both are on macOS 10.13.2 and vagrant 2.0.0

[symmetriq]

For what it's worth, this doesn't seem to be an issue for me. File syncing doesn't seem any slower than before. I can make a change, reload a couple seconds later and the change is there.

Host: iMac 5K (Late 2015), macOS 10.13.2
VM: Vagrant 1.9.5, CentOS 7

I also tried it briefly on a MacBook Pro (Retina, late 2013) with the same VM, and didn't run into slow syncing there either. Maybe the guest OS is a factor?

[fishfacemcgee]

@symmetriq @krzysztofbukowski A few questions:

1. What VM provider (and what version of it) are you using? Is it (the provider and the version) the same across the machines you tested on?
2. Does your Vagrantfile have any custom settings for synced_folder besides using nfs? We've seen some better luck with additional config values (but have the problem fairly consistently without them):
   config.vm.synced_folder ".", "/vagrant", type: "nfs", :nfs => true, :mount_options => ['actimeo=2']
3. @symmetriq (since you mentioned you're using COS7) are you on the same version of CentOS7 as listed in the issue or are you on an earlier build?
4. Are the devices that are not having problems formatted using APFS? From my understanding, traditional platter-based HDDs do not get upgraded to APFS in the update to High Sierra, but SSDs should. However, the 2013 MBP working (I think Apple was all-in on SSDs at this point?) fine is curious. Perhaps they didn't update SSDs that old for some reason?

Edit: updated the config to match the smaller config that @krzysztofbukowski reported having success with after seeing success among my coworkers too.

[krzysztofbukowski]

@fishfacemcgee It seems that it does behave the same way on my iMac as well. Not sure why I didn't notice the problem before. It's very intermittent though. Sometimes it transfers changes immediately sometimes it takes time.

1. Guest Additions Version: 5.0.2
   VirtualBox Version: 5.1

2. I only had :nfs => true. Adding :mount_options => ['actimeo=2'], seems to fix the problem.

3. All my machines are formatted with APFS.

[fishfacemcgee]

@krzysztofbukowski Thanks for the information as well as finding a minimal config change to resolve the issue. Hopefully it'll help Hashicorp investigate further and will at least give us a decent workaround in the meantime.

[tisba]

While also trying to minimize the latency before changes pop up in my Vagrant box, I was trying to use NFS vers 4 which does not seem to work. Does anyone know if this is supported by macOS?

[symmetriq]

@fishfacemcgee

Sorry for the slow response…

1. VirtualBox 5.1.22. Same on both machines.
2. Nope…this is my config: config.vm.synced_folder ".", "/vagrant", type: "nfs"
3. Looks like I'm on an earlier version of CentOS: CentOS Linux release 7.3.1611 (Core)
4. Both devices are APFS (Encrypted).

I haven't used the MBP extensively since I upgraded to High Sierra; I waited for 10.13.2 specifically because of this issue, and just did a few changes/reloads to see if changes were delayed as described here.

I mostly work on the iMac, which I've now used for the past work week, and I haven't run into any issues with slow syncing or errors.

[edigu]

Same here.

• Vagrant version: 2.0.1
• Mac OS 10.13.2
• Virtualbox 5.2.4
• NFS config: config.vm.synced_folder '.', '/vagrant', type: 'nfs'

$ kextstat | grep -v com.apple 
Index Refs Address            Size       Wired      Name (Version) UUID <Linked Against>
  152    3 0xffffff7f87562000 0x63000    0x63000    org.virtualbox.kext.VBoxDrv (5.2.4) B81C8F96-8276-3708-938E-3BE8C8B48759 <7 5 4 3 1>
  157    0 0xffffff7f875c5000 0x8000     0x8000     org.virtualbox.kext.VBoxUSB (5.2.4) EAE7363A-28F0-3DDE-B6C0-7D1F2D50DC37 <156 152 58 7 5 4 3 1>
  158    0 0xffffff7f875cd000 0x5000     0x5000     org.virtualbox.kext.VBoxNetFlt (5.2.4) B375D15D-DD7E-363E-883A-8352873A1AD2 <152 7 5 4 3 1>
  159    0 0xffffff7f875d2000 0x6000     0x6000     org.virtualbox.kext.VBoxNetAdp (5.2.4) EB116E49-1A55-3176-B29F-D91EEDD15BC9 <152 5 4 1>

Update: I just tried to mount it with mount_options => ['actimeo=2'] option and it helped a bit. I am getting I/O errors less likely now.

[fishfacemcgee]

Im also seeing that mount_options => ['actimeo=2'] makes a noticeable improvement. For people on teams like mine, with people split between Sierra and High Sierra, I have not found any evidence of this change causing issues for Sierra users. So, there's at least a reasonable workaround you can commit to your Vagrantfile if you're in a situation similar to mine.

[ervinandsmith]

We've experienced a very discernible slowdown as well on our NFS share, running a WP site on a CentOS 6 VM. Site takes 15-30 seconds to fully load instead of 2-3s, but copying source files into the VM and toggling a home directory symlink between share and link narrows it down to NFS share performance.

Unfortunately we're less help, as it was an upgrade from 10.12.6/HFS to 10.13.3/APFS, so no one control variable in the upgrade. In the process of standing up a separate HFS volume for the time being.

Was previously using: config.vm.synced_folder “.”, “/vagrant”, :nfs => true, mount_options: [‘rw’, ‘vers=3’, ‘tcp’, ‘fsc’ ,‘actimeo=2’], nfs_version: 4, nfs_udp: false

Now using: config.vm.synced_folder “.”, “/vagrant”, :nfs => true, mount_options: [‘rw’, ‘vers=3’, ‘tcp’, ‘fsc’ ,‘actimeo=2’] but still experiencing substantial file I/O latency.

[daylerees]

Having the same issue here.

[devon-pl]

Does anyone know if Apple is working on fixing this issue? any ETA?

[daylerees]

This issue is now affecting my whole team across a variety of OSX minor versions (however only High Sierra).

Symptoms are:

• Slow sync times, resulting in:
• File read issues from within the guest. Stating missing files on change.
• Average update time of 10s (for me at least, somewhat variable).

Vagrant: 2.0.2
Virtualbox: 5.2.6 r120293
OSX: 10.13.3 (17D47)

Happy to help provide more debug info if it helps solve this problem, thank you! :)

[conradkleinespel]

We are experiencing the same issues as @daylerees , although weirdly enough, only some of our High Sierra users are experiencing issues. They are all on High Sierra 10.13.3. Happy to provide more information as well if needed.

[conradkleinespel]

We tried using a sshfs (ssh config via vagrant ssh-config and then sshfs -p <port> -i <keyfile> vagrant@localhost:/var/www/website website). We use PHPStorm which works fine with this setup. It gives us fast load times and almost instant file changes from Host to Guest and from Guest to Host. We'll try using this and see how it goes.

[dnl-jst]

vboxfs and sshfs are no reasonable way to replace nfs mounts. nfs is much faster when using frameworks like symfony or applications based on symfony like shopware. it's a major performance impact not using nfs in such cases. i hope there will be a fix. imho the problem was introduced by apfs, so it's apples part to fix it. has anyone reported the problem to apple?

[ervinandsmith]

Quick update, after some APFS resizing drama, finally got a HFS+ partition running, but simply moving everything over to HFS didn't solve it for us. Possibly a guest OS-specific variable at play as well?

I've returned to testing various mount option combinations, but that too isn't getting us back to pre-upgrade performance. For anyone that moved to HFS, and that alone resolved sync speed issues, what mount options are you using?

[chrisroberts]

As this not an issue with Vagrant itself, I'm going to close this now. Cheers!

[conradkleinespel]

@chrisroberts Would you happen to know where the issue comes from by any chance ?

[chrisroberts]

@conradkdotcom I do not. High Sierra introduced a bunch of issues with NFS and it's not clear if it's an issue with their nfsd, their file system, or something else altogether. Lacking access to the issue reports further muddies the water. This was one of the reasons I worked on getting SMB support working on macOS. Even though it doesn't provide the same features NFS does, it at least is another alternative.

[conradkleinespel]

@chrisroberts Good to know, thanks ! I thought SMB only worked on Windows, we'll give that a try on OSX though, thanks a lot for your help ! :smiley:

[pnoeric]

I am having almost exactly the same bug, though I am only seeing 4-5 seconds to update a file with the same config (MacOS HS 10.13.3, vagrant using NFS sync). Interestingly if I ls inside the box (Guest machine) the update is immediate. So my temporary workaround is to ssh into the machine and run watch -n0.5 ls (just does an ls command every half second). Which is a pretty blunt hack but it gets the job done. Hoping Apple will clean up the bugs in NFS soon...

[webvertisingstudios]

@chrisroberts I just tried it with SMB support, it works, but now all requests take ages! I am on Vagrant 1.8.1, can it be that upgrading to 2 will fix that issue?

It's a temporary solution, but I found the solution as suggested by @conradkdotcom to be the only one that is actually working.

[chrisroberts]

@webvertisingstudios SMB support for macOS as the host was not introduced until Vagrant 2.0.2. I would definitely suggest upgrading.

[webvertisingstudios]

@chrisroberts thanks! Have you, or anybody reading this, tried this and does it fix or reduce this issue?

[chrisfromredfin]

Running on Vagrant 2.0.2 and seems like smb is in general much slower (I
have huge directory trees with lots of small files). It also doesn't
support symlinks, so I didn't continue with it, so I can't speak to the
specific "laggy-writes" problem.

--
Christopher J. Wells
Redfin Solutions, LLC
office: 908-4-DRUPAL x601
fax: [phone]
[email protected]
www.redfinsolutions.com

On Wed, Apr 25, 2018 at 3:22 AM, webvertisingstudios <
[email protected]> wrote:

@chrisroberts https://github.com/chrisroberts thanks! Have you, or
anybody reading this, tried this and does it fix or reduce this issue?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/hashicorp/vagrant/issues/9267#issuecomment-384185851,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAQ_qm_Ai_p2PiilvH_CMPStQzZoJ-X1ks5tsCRKgaJpZM4RA2Ik
.

[iworker]

I had the same trouble (`input/output error) with my docker-machine (which is using virtual machine like Vagrant) and PhpStorm (or any other IDE from JetBrains).
The problem solved when I disabled "safe-write" option in PhpStorm:

Settings/Preferences | Appearance & Behavior | System Settings --> Use "safe write"...

[faridmovsumov]

@iworker it works for me thanks.

[webvertisingstudios]

@iworker Interestingly, it made things worse for me (at least in combination with Symfony2 assetic:watch for file changes tracking in assets)

[danemacmillan]

I experience close to a ten second delay when editing files in the host. I know this because I have to run some grunt watch tasks within the guest, and it only starts to process the changes after that amount of time. If I edit the file directly in the guest, the changes are processed immediately.

For reference, I'm using Magento2, which has a very large codebase with many files and directories.

Edit:

In addition to the actimeo option, I've managed to get the sync times down to a second or two using the following NFS options in my VagrantFile:

vagrantshell.vm.synced_folder ".", "/vagrant", type: "nfs", mount_options: ['rw', 'vers=3', 'tcp', 'actimeo=2'], linux__nfs_options: ['rw','no_subtree_check','all_squash','async']

[mpdude]

My impression is that one of the last OS X 10.13 updates (possibly since 10.13.2?) caused large NFS performance degradations: Running a Symfony web app in the VM, mounting with mount_options: ['fsc', 'actimeo=1'].

File changes made on the host show up in the VM almost immediately. But web app requests take 8-10s, which is about 15 times slower than checking out the code directly in the VM filesystem.

Possibly interesting observation: Running watch -n .5 nfsstat -v in OS X shows

Client Info:
RPC Counts:
     Getattr      Setattr       Lookup     Readlink         Read        Write
           0            0            0            0            0            0
      Create       Remove       Rename         Link      Symlink        Mkdir
           0            0            0            0            0            0
       Rmdir      Readdir     RdirPlus       Access        Mknod       Fsstat
           0            0            0            0            0            0
      Fsinfo     PathConf       Commit
           0            0            0
RPC Info:
    TimedOut      Invalid    X Replies      Retries     Requests
           0            0            0            0            0
Cache Info:
   Attr Hits       Misses    Lkup Hits       Misses    BioR Hits       Misses
           0            0            0            0            0            0
   BioW Hits       Misses    BioRLHits       Misses    BioD Hits       Misses
           0            0            0            0            0            0
   DirE Hits       Misses
           0            0

Server Info:
RPC Counts:
     Getattr      Setattr       Lookup     Readlink         Read        Write
    17998663       389571     11456675         1362       812610       310762
      Create       Remove       Rename         Link      Symlink        Mkdir
      115189       115738        15852         2872          252        20063
       Rmdir      Readdir     RdirPlus       Access        Mknod       Fsstat
       21980         1289      1398742      6289377            0      4471470
      Fsinfo     PathConf       Commit
          46           21        14942
Server Ret-Failed
          1728084
Server Faults
            0
Server Cache Stats:
      Inprog         Idem     Non-idem       Misses
          82         2111           10     43007378
Server Write Gathering:
    WriteOps     WriteRPC      Opsaved
      310762       310762            0

The Server Cache Stats – Misses number is constantly increasing, it almost seems that every NFS request is a "miss"? Does anyone know how to interpret this and/or if this is a problem? Maybe Apple had to short-circuit some internal caching mechanisms? IIRC there were some serious cache-related issues before 10.13.2?

[roelleor]

Hi @mpdude: I also use Symfony (2) with latest version of Mac and Vagrant and use these options for synced_folder which work quite well (perhaps for you as well):
sync_type: nfs
mount_options: actimeo 2
(or as puphpet config):

        synced_folder:
            folder1:
                sync_type: nfs
                nfs:
                    mount_options:
                        actimeo: 2

[sebastiaanluca]

Experiencing the same issue on macOS Mojave 10.14.1. Creating or deleting files takes a loooong time, serving files using nginx and PHP-FPM take about 20 seconds because of the slow read time.

[mpdude]

Currently testing Parallels. Their synced folder implementation seems to be reasonably fast, at least better than NFS.

[danemacmillan]

@mpdude To add to this, it just occurred to me to do the same.

I'd been using NFS since the days when I used VirtualBox, and never questioned it since. I've been using Parallels for at least three years, but never attempted using its folder sync. So far I'm seeing about the same performance as NFS, which is good, but--more importantly--I'm seeing zero filesystem errors, like random .nfs0000000000000000 files that cannot be deleted from the guest, or random utilities like git erroring out when it shouldn't be (on gigantic checkout operations of 30k+ files, and NFS seems incapable of keeping up).

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.