Vagrant: rsync-push and rsync-pull

[p0deje]

Until Vagrant 1.5 is released, I suggest to add one more feature to vagrant rsync. Currently, it's only possible to push local files to VM, but there is no way to pull them back. It'd be super useful for CI environments, when we need to retrieve test results and artifacts. It'd be pretty easy to implement (by changing position of local_path remote_path arguments).

It would also make sense to change commands to something like:

$ vagrant rsync-push
$ vagrant rsync-pull

I can work on this if you think it worths.

[mitchellh]

I think adding just rsync-pull would work! Vagrant 1.5 is pretty much wrapped up for features but I'd be happy to look at a pull to get it in in a 1.5.1 or something.

[p0deje]

Maybe then vagrant rsync --pull?

[mitchellh]

Perfect.

[smerrill]

If you'd like to try this functionality out, I made a gem that does (manual) pulls from the guest to the host. Check out https://github.com/smerrill/vagrant-rsync-back if you'd like to help test.

The gem itself is mostly implemented as a monkeypatch to RsyncHelper, so if people have good experiences with it, I can put together a pull request.

[sidneygijzen]

+1 for this feature.

I' ve also tested a little bit the vagrant-rsync-back plugin from @smerrill. Works fine so far on Arch Linux with Vagrant 1.5.4.

[rails-novice]

been working with vagrant-rsync-back by @smerrill (thanks) for a few weeks now, haven't noticed any issues so far with 1.5.2. Would be nice to have the watcher integrate this feature, it would improve the workflow a lot (stop watcher, rsync-back, start watcher).

[p0deje]

For now we're using smth like this in our CI run script:

vagrant ssh-config > vagrant-ssh-config
rsync --archive -z --exclude .vagrant/ --exclude Vagrantfile --exclude .git --exclude tmp --exclude .bundle -e 'ssh -F vagrant-ssh-config' default:/vagrant/ .
rm vagrant-ssh-config

[mrkeyboard]

Would it be possible to have pull included two ways using vagrant rsync-auto?

[starrychloe]

Consider Unison: http://www.cis.upenn.edu/~bcpierce/unison/

I don't recommend using rsync for bidirectional synch. It is not a trivial task. Here is background info (or pattern) for doing master-master synchronization, in the context of databases, but it applies to files as well: http://msdn.microsoft.com/en-us/library/ff650702.aspx

You have to keep track of deleted files and conflicts, and that is what Unison does with 'shadow tables', only it keeps databases of the files in ~/.unison.

I've gotten around the problem of one-way rsync for commands that need to be executed in the guest but write files that need to be available on the host with

$ (cd /vagrant/project; rake db:migrate)

This changes to the default vboxfs (which is dog slow), does the command, writes to guest disk, synchs to host disk, then rsync will synch back to the /rsync synched folder.

[backward-crazy-mage-puppy-36]

+1 I couldn't agree more with @laurentlemaire, have been watching this issue since March 14 because it's a killer feature for web developers, hope it makes it to the next release of vagrant :) ; BTW i know this might not be the most appropriate place, however, a sincere thanks to @mitchellh for vagrant

[matthew-dean]

Blarg, two-way sync worked so well (and automatically, without the need for rsync-auto) in 1.4.x. The current one-way sync just doesn't make any sense from a user perspective. I care little about performance when syncing, I just want to "vagrant up" and forget it.

Is there a way to revert to the previous shared-folder way of doing things?

If not, +1 to get back to some two-way syncing sanity.

[jdhiro]

@matthew-dean shared folder sync'ing wasn't removed, we are still using it.

This is a feature request for the people who can't/won't use shared folders because virtualbox has the worlds slowest implementation. Some of our devs see 1+ minute load times with vb shared folders.

(Personally, I'm using the Parallels provider, and shared folders are _blazing_ fast.)

[matthew-dean]

Hmm, ok. Maybe there's just something wrong with how I have it set up then, but syncing used to work before I upgraded Vagrant from 1.4.3 to 1.6.3. How do you use the parallels provider?

[jdhiro]

@matthew-dean https://github.com/Parallels/vagrant-parallels

If you happen to have a copy of Parallels, I highly recommend it. It blows away virtualbox.

[smerrill]

@khromov It has not been removed; you may still use NFS or VirtualBox/VMware shared folders.

[jenmo917]

I use the following solution as a workaround to get a two-way sync with acceptable performance. First, use the standard VirtualBox shared folders and sync to /vagrant. Then use Unison inside the VM, to sync between /vagrant and /var/www. I made a simple shell script that runs the Unison sync every 3 sec.

<-VirtualBox shared folder-> /vagrant <-Unison file sync-> /var/www

Maybe it's not optimal but I see no other solution right now.

[duro]

@jenmo917 That is an interesting solution, and it actually may help solve some of my problems. My biggest complaint with all current two-way syncing solutions is that they do not trigger OS file system events at the final destination in the VM (which I use to trigger build watchers like Grunt/Gulp). Rsync does a real write to disk, so the events are triggered, but it has been a real challenge to get it working two-way (and by challenge, I mean impossible). If your Unison solution works the same way as rsync at the final destination folder, then I may be able to use it.

Once question, is there a reason that you only run Unison every 3 seconds? Could it be run more frequently?

[kylemacfarlane]

I've found that for front end development (i.e. sass, js, etc) that Virtualbox shared folders are plenty fast. The only problem as @duro points out is that they don't fire inotify events (and neither does Samba). I use my own build scripts and make sure to use regular file system polling to detect changes (e.g. with Watchdog use watchdog.observers.polling.PollingObserver). For frontend stuff where you only have say 50 files this is super fast and well under a second. I don't know if it's possible to change the type of watcher in Grunt or Gulp but that's what would need to be done.

Now for backend development where I have tens of thousands of files, shared folders and file system polling really starts to fall apart and can take anywhere from 3 to 60 seconds to detect changes (not to mention high CPU usage). If I'm doing a lot of this I just reboot and switch to rsync-auto.

[kylemacfarlane]

I don't use Grunt but perhaps something like this alternate watcher is what people need: https://github.com/unbalanced/grunt-simple-watch

[jenmo917]

@duro I use 3 sec because it's enough for my purpose (PHP development in SugerCRM). When I save something in my IDE I have to wait approximately 3 sec until I can see the change in the web-browser. I tried to do the sync more frequently but I experienced issues with high CPU-load when I did that.

[WyseNynja]

I've been using unison instead of the built-in vagrant support and it's been working well. This also lets me easily sync with Vagrants launched by other people and even production machines all with pretty much the same sync commands

[duro]

@WyseNynja and @jenmo917, have you guys tried using the vagrant-unison plugin?

https://github.com/mrdavidlaing/vagrant-unison

I have not, I'm just wondering if you have tested it.

[jenmo917]

@duro, yes I have, and I had problems with it. I think it was because it didn't have auto-sync and you had to trigger the sync yourself each time. I took the "on change mechanism" from vagrant rsync-auto and implemented it into the vagrant unison plugin. The problem I faced was that Unison was triggered on changes on the host machine, but not on the guest machine. When you think about it is quite natural because Rsync, where I found the code, is a one-way sync. This was how far i went with the unison plugin.

[WyseNynja]

vagrant-unison can also only sync one directory. Additionally, I wanted to be able to easily sync to vagrants launched by other people and a vagrant plugin doesn't really make that easy

[btrepp]

I've been bitten using rsync assuming it would run like shared folders did :(. My fault entirely but yeah.
Was hacking on files inside the VM, this is what I usually did with shared folders, vagrant bootstraps the environment and then I work on the project. Was using rsync assuming it would be magical, broke some OS config playing around, easy enough I'll vagrant destroy and vagrant reload.
....Whoops there's all my changes, which were commited to git inside the VM, but never pushed out!, they were destroyed :(.

Also I feel this may be useful for my issues with building a windows machine. Windows shared folders are.... special at best. Atm I can't get vagrant to up a config using them, vbox shared folders worked perfectly here, but the requirement is visual studio in these machines.... which is also kinds of special, refuses to run projects of vbox shared folders.

A two-way rsync would let windows and its strange file system do whatever it wants, and just sync the changes around.

[kensykora]

@btrepp I think in general you should be following the rule that if you need to hack inside of the VM then you're probably doing it wrong. Both configuration and code changes should be made using either a configuration manager (puppet, chef, ansible) or code changes through the rsync tool in this case.

For doing .Net stuff, vagrant definitely isn't ready for prime time here unless you have a Windows host and can build using Visual Studio on your host. If you are using an OSX host I'd recommend running vagrant ssh -c msbuild myproj.sln and working that into your dev workflow.

[jdhiro]

@kensykora who's rule is that exactly? ;-) Some frameworks require (or are easiest to use) with guest-side commands. For example, we use Laravel's artisan command for easily generating migration files.

[btrepp]

@kensykora the situation you've described basically means 'never use vagrant for anything with an ide'.

If I'm required to keep a build environment outside of vagrant (what most ides sell themselves as). Then what is the purpose of vagrant?.

Also I'm typically a major vim user (windows is an experiment in order to get the windows guys drinking the automation coolade), but in regards to vim. Say I had a whole bunch of specific plugins to make it easier to code in the specific framework we are using. If there is vim inside a VM then all developers get those plugins easy.

Under your suggested scenario. I also have to tell all the other devs they need plugin x,y,z to be productive, which means they would have to hunt down how to install it?. What if one of these guys used windows and the other Linux as the host?.

I see the power of vagrant of automating the dev environment. Project from two months ago? Vagrant up and get those same tools ready to rock.

[kensykora]

Guys I apologize, I'm not trying to turn this rsync issue into a philosophical debate about dev workflows. Let's steer the conversation back to the issue at hand.

Definitely open to change on my opinion and I'd be happy to discuss this over twitter.

[wbkang]

Here's my attempt to summarize the issue:

• VirtualBox shared folder provides two-way synchronization but it does not fire inotify notifications so we can't run programs to watch changes and compile them (e.g., Less compiler).
  
  
  
  • The polling approach falls apart quickly because VirtualBox shared folder is slow.
  
  
• Setting up a shared folder via rsync is fast and fires inotify but it is only one-way.

My workaround is to set up the source directory as two shared folders, one via rsync and another via virtualbox shared folder. e.g.,

config.vm.synced_folder ".", "/vagrant", type: "rsync"
config.vm.synced_folder ".", "/vagrant_sf"

I set things up so that any program that watches file changes via inotify reads from /vagrant. Any changes that needs to be persisted goes out to /vagrant_sf (e.g., alembic migration files). This workflow seems to work reasonably well.

[brpaz]

I think an effective way of doing two-way sync is a must.
Lets stay I am manager of an IT company. I want that once a new developer arrives to the company, he just needs to do a git clone of the project following by vagrant up and it them has a complete environment ready to work. In the host system, he would only need to have the browser, the IDE, git and vagrant. All the tools needed for the project (bundler, composer, grunt, sass etc) are bundled in the VM and are executed inside the VM.
Then he moves to another project. Same procedure. git clone -> vagrant up and voila.
This would be an amazing way of work and a truly portable environment.
But for this to work, It has to be fast. I thought the rsync option would be the fastest compared with shared folders and nfs, until I read it only supports one way sync.

[khromov]

For those having issues with the performance of VirtualBox Shared Folder, I suggest trying VMWare (Workstation or Fusion). In our testing, we could see a 10x performance increase in load times on a Laravel web application when switching over from VirtualBox SF to VMWare Workstation SF. (From 600ms to 60ms generation time)

[patrickheeney]

@brpaz This is exactly my goal with this as well. Right now it is such a mess to keep the front-end devs in the right workflow. I can't even count the number of times I had to remote in and npm install or fix host machine bower, grunt, saas, node etc because I can't bundle it in the VM because of performance issues. Frustrating to say the least.

@khromov The speed issues I am struggling with is the build tools that monitor folders. 600ms to 60s is awesome but I wish we had a way to cut down 2 minute build times to the 2 seconds when running it from the host machine. I will check out that plugin though for faster laravel speeds!

[khromov]

@patrickheeney If you are monitoring on your host machine, the build times should improve with SF.

However, why not monitor (grunt watch, etc) on the VM directly? Then you'll get maximal performance.

[patrickheeney]

@khromov This issue and some others are trying to address that working on the vm directly just doesn't work.

The first problem is the build tools generate files which are then not available on the host machine. The developer rebuilds the box and then loses work because it never made it back (switching back to shared folders until this can be addressed). We lost several days worth of work when work because of missing files that never made it back.

Then when switching away from the one way rsync and back to shared folders is incredibly slow #3249 when working on the VM directly.

[khromov]

@patrickheeney

On my setup, I use Grunt on the VM together with VMWare shared folders. It takes 1.5-3 seconds for the build process (Compass to CSS) to complete after a file is changed, which seems normal.

[patrickheeney]

@khromov It's typically the watcher that is really slow, not specifically the build process itself. Lots of us are having problems over at #3249, but I wish I was seeing what you were seeing! I'll have to look at VMWare though as it seems to improve things.

[btrepp]

I'm using the rsync-back plugin for windows guests.

This is working pretty good. Pretty performant inside the VM. Rsyncing to
and from is okay, not great but okay. I do run the big risk of accidentally
nuking the guest and losing everything though.

Workflow is basically hack inside VM. Rysncback. Create a commit. Repeat.

Currently its the only way to use visual studio in a vagrant environment
(everyone I work with is windows biased). Visual studio wigs out with vbox
shared folders (and even windows shared folders), plus I couldn't get
windows shared folders working anyway.

Now that I think about it. Could something like sshfs be an option?. Host
could run a sshserver and expose the folder and the guest could mount it?
On 21 Aug 2014 22:36, "Patrick Heeney" [email protected] wrote:

@khromov https://github.com/khromov This issue and some others are
trying to address that working on the vm directly just doesn't work.

The first problem is the build tools generate files which are then not
available on the host machine. The developer rebuilds the box and then
loses work because it never made it back (switching back to shared folders
until this can be addressed). We lost several days worth of work when work
because of missing files that never made it back.

Then when switching away from the one way rsync and back to shared folders
is incredibly slow #3249
https://github.com/mitchellh/vagrant/issues/3249 when working on the VM
directly.

—
Reply to this email directly or view it on GitHub
https://github.com/mitchellh/vagrant/issues/3062#issuecomment-52928652.

[laurentlemaire]

What I do now is a local script that I built that watch the host project
folder and rsync it with the VM whenever there is a change.
As working with LESS, Symfony, Grunt many console command will generate
content. Those commands are only ran on the VM and then I use the rsync
back whenever one of those commands is used.
Also I make sure not to rsync back the entire project but only some
specific folder avoiding cache and logs to be copied over to the host.

On Thu, Aug 21, 2014 at 4:36 PM, Patrick Heeney [email protected]
wrote:

@khromov https://github.com/khromov This issue and some others are
trying to address that working on the vm directly just doesn't work.

The first problem is the build tools generate files which are then not
available on the host machine. The developer rebuilds the box and then
loses work because it never made it back (switching back to shared folders
until this can be addressed). We lost several days worth of work when work
because of missing files that never made it back.

Then when switching away from the one way rsync and back to shared folders
is incredibly slow #3249
https://github.com/mitchellh/vagrant/issues/3249 when working on the VM
directly.

—
Reply to this email directly or view it on GitHub
https://github.com/mitchellh/vagrant/issues/3062#issuecomment-52928652.

_Laurent Lemaire_
Elyotech LTD
Creative & IT Team
Tel: [phone]
Skype: lemstyle
[email protected]
http://www.travelyo.com

[btrepp]

While it would be useful. I'm not sure if rsync is the best tool for two way sync (googling implies rsync just isn't good for this). There's obviously a big demand here for something that isn't NFS/SMB/VboxSF and sort of just works.

Perhaps something like unison or sshfs could work?. My mind keeps on thinking that something like a filesytem over HTTP would be super useful, if it existed we could fire up a server on the host, and hopefully mount it as a fuse based fs in the guest, but i don't think something like that exists.

[marijnotte]

We now use the vagrant-winnfsd plugin (https://github.com/GM-Alex/vagrant-winnfsd) to use nfs on Windows. nfs is much than vboxsf and a does real sharing instead of syncing, so for us it's better than rsync.

[sethvargo]

Hello - this issue has been labeled and is "accepted". Please refrain from adding +1 or "me too" comments that cloud the conversation. If you are interested in receiving updates on this ticket, please click the "Subscribe" button in the sidebar.

[willmorgan]

@sethvargo Have you any news on a release date for such a feature?

Unfortunately I'm not a Ruby developer otherwise I'd give this a go myself.

[seeruk]

The main thing I want to see in any new syncing method is two-way sync, with customisable permissions. So, mounting files owned by me, as a specific user in the VM (for instance, mounting as www-data). Currently I'm using Vagrant's rsync implementation along with lsyncd running in the VM syncing files back to the host. It works... most of the time, but at least permissions are perfect - it seems no other synced folder provider manages to achieve that currently.

[dmatora]

consider using http://github.com/dmatora/vagrant-unison
it does realtime bidirectional folder sync

[duro]

@dmatora I have tried your plugin in the past and found it to be a little unreliable. I see you have picked back up with development in the new year. How has it improved?

[dmatora]

it's not my plugin, i just updated it few days ago, to make it functioning and reliable :)
but after few days of using it, i can say that's it's doing the job unbelievably well
i have 4 cores cpu and can afford every second checks though

[create-renegr]

@dmatora thanks for updating the plugin. It is working on my win8.1 host!

@jdhiro I would have tried to use parallel, but it seems, that it is no longer available for windows.

[kylemacfarlane]

@dmatora and anyone else using Unison.

I'm finding that Unison isn't such a good solution. When used with tools like sassc and UglifyJS it starts syncing files at the start of compilation and then abandons the sync because the "source file has been modified during synchronization". Is there any way to make Unison wait until the files have finished being written?

I'm using dmatora's vagrant-unison fork.

[akostadinov]

I think important use case is to allow a number of machines to share some files while provisioning. For example one VM generates some certificates, then the rest of the machines need to pick them up somehow. This is easy with shared folders but no with rsync. Generating stuff on host might not be so trivial especially if you want to showcase things to users and don;t expect user's host machine to have all necessary tools.

[purpleidea]

On Mon, May 11, 2015 at 5:47 PM, Aleksandar Kostadinov <
[email protected]> wrote:

I think important use case is to allow a number of machines to share some
files while provisioning. For example one VM generates some certificates,
then the rest of the machines need to pick them up somehow. This is easy
with shared folders but no with rsync. Generating stuff on host might not
be so trivial especially if you want to showcase things to users and don;t
expect user's host machine to have all necessary tools.

Interesting idea, but I don't think this should be a goal, because it won't
be something that's easily replicated in prod. If you want to do this, use
config management tools to do so, instead.

[akostadinov]

@purpleidea, if I'm using another config management system, why should I use vagrant? Also reading about ansible for example I don't see any big difference - the issue remains the same. I think vagrant is mostly useful for developing, testing and demos. For production you'd have pre-generated certificates and keys. While for the former use cases you need to generate things on the fly to avoid security issues.

[purpleidea]

On Tue, May 12, 2015 at 1:56 AM, Aleksandar Kostadinov <
[email protected]> wrote:

@purpleidea https://github.com/purpleidea, if I'm using another config
management system, why should I use vagrant? Also reading about ansible for
example I don't see any big difference - the issue remains the same. I
think vagrant is mostly useful for developing, testing and demos. For
production you'd have pre-generated certificates and keys. While for the
former use cases you need to generate things on the fly to avoid security
issues.

My point is that vagrant is for dev and not prod, and if you use the same
mechanism in both to distribute your files, then your dev env. will more
accurately resemble prod, and tests are more likely to accurately represent
real life.

[akostadinov]

@purpleidea, sure that might be possible sometimes. But you can't use your production keys and certificates in dev environment unless you want to get into trouble (assuming you even have access to the prod certificates). And I think more and more systems nowadays use some sort of certificates or keys to talk securely to each other. This use case - securely sharing data between the provisioned machines is not covered well in current vagrant when deploying non-locally.

[purpleidea]

On Tue, May 12, 2015 at 2:12 AM, Aleksandar Kostadinov <
[email protected]> wrote:

But you can't use your production keys and certificates in dev environment

Use different copies of course!

This use case - securely sharing data between the provisioned machines is
not covered well in current vagrant when deploying non-locally.

I don't think it's supposed to be, IMO.

I think I've offered my point clearly enough. It's up to you if you want to
take these suggestions or not. I don't think it's helpful for me to discuss
this any further. Cheers!

[akostadinov]

I don't think it's supposed to be, IMO.

Why?
I don't think it's helpful for me to discuss this any further.
Not helpful for me too unless you bring anything new to the table. Generating certificates and installing properly on all machines is often not trivial, also may depend on user preferences about number of machines, their role and certificate types. It's perfectly legitimate to allow that to happen automated within vagrant IMHO. For small projects it may not be an issue but for big projects it becomes an issue.

[rarkins]

We decided to go with unison instead of rsync, but using a manual installation instead of the plugin. I've documented the process here in case it helps anyone.

[nha]

@rarkins Thanks that looks useful. bidirectional sync is very important for certain workflows.

As a workaround, is there a way to mount several folders ?

[rarkins]

@nha I don't have a need for that, but would guess that the simple but inefficient way is to have multiple .prf files and run multiple simultaneous unison commands. An alternative might be symlinks, as described here?

Also, are you sure you can't just sync your main directory and _exclude_ paths within it you don't want sync'd?

[giovdk21]

+1 for two-way sync with rsync or at least an handy and reliable rsync-pull, but the former would be ideal.

[russjaguar]

We were able to get @rarkins solution working. I see there might be a way to automate the management a bit by having profiles name after the boxes... maybe... Would be nice if the plugin were to get maintained as the NFS module is hit-and-miss within the 20 or so boxes we end up working with. Can be _very_ annoying to make changes, not see them loading and thing you're doing something wrong only for them to suddenly load 1-2 minutes later.

[Perni1984]

@rarkins: Thanks, you saved my life. Unison is working great for bidirectional synching between windows and vms. Now I can develop and run symfony2 with full speed inside my vms on my windows host. For me, this works even better than vagrant-winnfsd.

[flynfish]

@dmatora:

1. does running vagrant plugin install vagrant-unison install your forked version? If not, how do we get yours installed?
2. Is there a way to change the ssh username to something besides 'vagrant' ?
3. I would have posted these questions on your fork, but issues are turned off

[dmatora]

@flynfish no, it does't.
Simple way to install it would be to download https://www.dropbox.com/s/1yu1s7pr3qlr8ag/vagrant-unison-0.0.15.gem and do vagrant plugin install vagrant-unison-0.0.15.gem
but it's currently outdated.

I'm sorry to say that guys, but i moved to using nfs_guest, i tried to use it before unison, and nfs silently failed for me. But later i discovered than on OSX you need to have localhost in your /etc/hosts file in order for nfs to work and that solved it. With nfs_guest i don't have fs_events, but performance is close to filesystem, and i don't have sync failures, so i don't fix those anymore. Yet CPU load is so much smaller.

[mikestaub]

I have been using the vagrant-unison2 plugin with success. I was also using NFS but ran into issues with windows and npm.

[seantcanavan]

I've seen a couple good suggestions here so far. I'm completely out of options because network-based shares don't work over VPN but our VPN client is not up for discussion which leaves openconnect out of discussion.

Rsync appears to be the only option as appallingly slow disk read and write speeds have ground our build and test processes down to a halt on the standard sync share.

But the issue with Rsync is that it only supports one way sync and of course requires cygwin to be installed as well. Bundling a two-way rsync into Vagrant would be a huge huge huge win.

It looks like there is hope though. I'm excited that this has been tagged as milestone 1.9.0 but 1.8.7 is only 33% complete as of this moment in time. An interim solution will absolutely have to be implemented.

[purpleidea]

Check out https://github.com/dustymabe/vagrant-sshfs/ /cc @dustymabe

[dustymabe]

@seantcanavan check out vagrant-sshfs - i've actually been thinking about proposing it for inclusion in vagrant itself. for now it lives pretty well on its own as a plugin.

[seantcanavan]

Thanks for the ideas @purpleidea and @dustymabe. Unfortunately the ruby gem bundler uses a custom user agent string which the corporate firewall immediately shoots down which makes plugin installation impossible so I was hoping for a baked-in solution.

Looks like I gotta get the network guys on the phone...

Cheers.

[dustymabe]

@seantcanavan can you download the gem and install them directly (i.e. install from file instead of from rubygems.org) I think for vagrant-sshfs you need win32-process and vagrant-sshfs. if you get those two gems and then install directly from file (i.e. vagrant plugin install ./win32-process-X-X.gem && vagrant plugin install ./vagrant-sshfs-X-X.gem) then it should work

[seantcanavan]

@dustymabe Finally got the gem files built for win32-process-0.8.3 and vagrant-sshfs-1.2.0. I created a gems directory at my project's root folder where the vagrant file is located and put both gem files in there. I executed vagrant plugin install ./gems/win32-process-0.8.3.gem from the root directory and received the error /usr/lib/ruby/2.3.0/rubygems/specification.rb:946:in all=': undefined method group_by' for nil:NilClass (NoMethodError)

EDIT01: Appears to be an issue with Ubuntu 16.04. Vagrant 1.8.1 and ruby 2.3.1.
EDIT02: Patched /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb successfully. Directions here
EDIT03: New error after trying to install: Could not find gem 'win32-process (= 0.8.3)' in any of the gem sources listed in your Gemfile or available on this machine.
EDIT04: Appears Ubunbut 16.04 is kicking me in the teeth again... Investigating: https://github.com/mitchellh/vagrant/issues/7493
EDIT05: Apparently need to upgrade Vagrant version. Command is "wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb && sudo dpkg -i vagrant_1.8.6_x86_64.deb". Restarted my box and plugin install SUCCEEDED!
EDIT06: vagrant destroy now produces There was an error while trying to load the gem 'win32-process'. Gem Load Error is: unable to resolve type 'uintptr_t' I feel like I'm on a roller coaster.
EDIT07: I've filed an issue with the ffi plugin, which is a dependency of the win32-process plugin, here: https://github.com/ffi/ffi/issues/530. I might have to open a different issue if the error message is a red herring and not really an ffi issue.
EDIT08: Installing win32-process on OSX and Linux totally breaks vagrant. Employing brain power to realize win32-process is a windows-only plugin solves the issue.
EDIT09: Getting a new error now on vagrant up: /opt/vagrant/embedded/gems/gems/vagrant-1.8.6/lib/vagrant/action/builtin/mixin_synced_folders.rb:137:inblock in synced_folders': Internal error. Report this as a bug. Invalid: sshfs (RuntimeError)` Double checking my syntax. Seems easy enough - like sshfs is an unknown synced_folder type.
EDIT10: Appears my locally built gem file is causing the error. When I manage to bypass the firewall and install remotely via the gem bundler it works fine. But more importantly success! Finally!

[dustymabe]

@seantcanavan wow - so you finally got something working?

[seantcanavan]

@dustymabe Yes!! I hope I'm not derailing and someone else will find this info useful. My last question is this: I've pulled in vagrant-sshfs from outside our firewall once and installed it locally and it works great. Now I need to package it in a gem file to add it to our source so people inside the firewall can reliably retrieve it. I found the plugin installed at /home/username/.vagrant.d/gems/gems/vagrant-sshfs-1.2.0 but when I run bundle install inside the install dir I get

[!] There was an error parsing Gemfile: You cannot specify the same gem twice coming from different sources.
You specified that vagrant-sshfs (>= 0) should come from source at . and source at .
. Bundler cannot continue.

# from /home/username/.vagrant.d/gems/gems/vagrant-sshfs-1.2.0/Gemfile:13
# -------------------------------------------
# group :plugins do"

gem "vagrant-sshfs" , path: "."
# # Add vagrant-libvirt plugin here, otherwise you won't be able to
# -------------------------------------------

Do you have a quick and easy way I can package your plugin to install locally via command line for future use?

[dustymabe]

@seantcanavan commenting out this line should work to fix that error: https://github.com/dustymabe/vagrant-sshfs/blob/master/Gemfile#L13

the other thing is - why not just download the gem from my releases page and use that?
https://github.com/dustymabe/vagrant-sshfs/releases/download/v1.2.0/vagrant-sshfs-1.2.0.gem

[chrisroberts]

Hi there,

We would really like this, but this issue has been open for over a year with no one working on it, Leaving it open is unfortunately making our issue count look higher than it is. I’m going to close this and if someone wants to work on it I’d still be open to a PR!

Cheers!

[mcandre]

Crap. I spent all this time and effort setting up Vagrant boxes in the hope of developing a suite of buildbots for cross-compiling systems language applications for different operating system kernels. And then I find out that rsync-based Vagrant synced folders--and there is no other option besides rsync-based Vagrant synced folders for the vast majority of guest operating systems--does not support bidirectional syncing. So there's little point in vagrant ssh -c 'clang -o /vagrant/hello hello.c' if the hello binary won't even be transferred to the host. :(

I have a few suggestions for implementing bidirectional rsync more quickly:

• https://github.com/deajan/osync , which claims to just depend on bash and rsync, and therefore could be easily installed automatically by Vagrant into rsync-enabled guests.
• Continue using /vagrant as a one-way source host->guest folder, and introduce a second folder like /vagrant-output that one-way syncs to the host as <host-cwd>/vagrant-output. When Vagrant rsyncs the /vagrant source folder, it should exclude vagrant-output.
• The user can also manually rsync files out of the guest to the host, but this represents a leaky API for Vagrant.

rsync configuration is something that Vagrant should really master, for more capable VM's. In the meantime, I might configure my buildbots to publish their artifacts to S3 instead of trying to copy the artifacts to the host :/

[dustymabe]

@mcandre would vagrant-sshfs work for you?

[cromulus]

sshfs looks like it has far more dependencies, particularly FUSE, than osync.

On Jan 3, 2018, at 21:05, Dusty Mabe [e-mail] wrote:

@mcandre would vagrant-sshfs work for you?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[daks]

@mcandre I use vagrant-scp to retrieve artifacts from Vagrant machines.

[mcandre]

@dustymabe vagrant-sshfs is interesting. Unfortunately, it is not clear that this plugin supports bidirectional sync in a single Vagrant box configuration--the documentation seems to indicate that either guest->host syncing can be configured, xor host->guest, but not both at once.

Also, this plugin does not support Windows, which for my purposes (cross-compilation) is a no-go.

Update

Good news, the vagrant-rsync-back plugin appears to work reasonably reliably, at least to-and-from UNIX-style operating systems like Debian, Illumos, and MINIX. Will be testing this with Windows as well. It stinks that vagrant rsync-back must be manually called in order to perform the other half of bidirectional sync, but at least it works! Would love to see this capability backported into the main Vagrant system so that rsync synced folders can work bidirectionally out of the box, so to speak.

[dustymabe]

@dustymabe vagrant-sshfs is interesting. Unfortunately, it is not clear that this plugin supports bidirectional sync in a single Vagrant box configuration--the documentation seems to indicate that either guest->host syncing can be configured, xor host->guest, but not both at once.

SSHFS is essentially a network filesystem. You can access files from the host or from inside the guest at the same time and they transparently update all locations. A longer description here.

Also, this plugin does not support Windows, which for my purposes (cross-compilation) is a no-go.

It does support windows, just not for reverse mounts (sharing guest files to a host folder), which I don't think you actually need. How about giving it a try so you'll know for sure 😃.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.