Docker-sync: [Unison] Unox watch doesn't ignore ignored files

[mickaelperrin]

Unox doesn't ignore any files while watching for file system changes, that's why Unison may reports in the log:

Looking for changes
Waiting for changes from server
Reconciling changes
Nothing to do: replicas have not changed since last sync.

Proposed solution

• Fork unox
• implement an additional ignore parameter
• implement a check on ignore parameters before launching a sync

NOTE: if you have a SHELL which retrieves some information about git branches on the prompt, each command you type in a shell may trigger a sync

[EugenMayer]

is that actually related to https://github.com/EugenMayer/docker-sync/issues/117 ?

[mickaelperrin]

I don't think so. But while inspecting unox code, I understood that...

[jzahedieh]

This is causing issues for me using Magento as it uses a lot of code generation which causes the sync to fire on these directories then silently fail because the file was removed during sync, not being able to ignored these directories is not ideal.

[mickaelperrin]

I don't think this issue is related to yours, this issue dealt only with false positive resulting in the Nothing to do: replicas have not changed since last sync message.

It looks like more that you have not or badly configured the sync_excludes unison option. Be sure to use the full directory path to the directories you want to exclude, or use another unison exclude option in sync_args.

[jzahedieh]

@mickaelperrin thanks, I had trailing slashes on the sync exclude directories, I see the issue you are raising now.

[EugenMayer]

@mickaelperrin does this mean, that excludes are not valid for the container part ( sync back ) also?

[stuyam]

Thanks @EugenMayer for moving my issue over here. It seems that my setup does show that there are "no files to sync" when I exclude the "tmp" folder, however things are still synced from the container back to my computer.

[mickaelperrin]

@EugenMayer I don't think so, this is an unox related issue. So the problem is only related to the host.

And don't forget, this is a watcher only problem not a sync problem. It will only trigger a false sync action but ignored files are not synced even with this issue.

It uses only some unneeded system resources to do an unneeded system sync.

[EugenMayer]

Ok even though i mixed up "where" we have the wrong-configured watcher i was aware that we actually trigger the event, but in the end we do not sync.

Do though not underestimate the impact of "more watched folders" under OSX, this is enormous. Being able to exclude some of those, can make a huge impact - so i would not degrade this issue to plain cosemtic.

Just to give you some metrics: using fswatch, i can more or less linear scale the CPU usage according to the files watch ( ovbv that is only watch, not sync), unox is not linear, but there is an impact, a important one. So not being able to exclude node_modules, thus watch 5k files in adition, can add 20% CPU in a blink of a moment.

What do you think?

[mickaelperrin]

@EugenMayer indeed, you got a point (as usual :) ). And the proposed solution is dumb because it won't decrease the amount of watched files, it would only prevent to launch a sync.

I quickly take a look at unox and fsevents for mac. Currently, it looks like there is no way to exclude some files / folders easily in the watching process.

The only solution I see is to alter unox in order to give to the fsevents Stream watcher not a base path but an array of paths excluding those we don't want to watch.

Note also, that if I understood well this issue, currently fsevents looks like to work natively at a directory level in order to reduce the number of events. So, unison rules that work at a file level won't be take into account.

I have to see also if unox is notified about the exclude rules of unison or if we have to implement a specific exclude rule in docker-sync.

[EugenMayer]

Sounds like a very, fairly complicated issue - it might not be worth the effort. I think we might be slowly start to think about switching to https://github.com/facebook/watchman - it does scale very well and AFAICs a lot better then unox

there is a lot more that we can do: https://facebook.github.io/watchman/docs/install.html

The implemented excludes https://github.com/facebook/watchman/commit/56fcdbac15bd6cda1790c13aa9b05bf9fa3cd237 and we probably have easier and better times with it

[EugenMayer]

closing in favour of #297

[james-relyea]

This was closed in favour of #297 but that was then closed because it's too much work.

What's the proposed solution then? Is there something we can work towards?

[EugenMayer]

No, watchman work has been stopped due to the availability of native_osx strategy. And that is the proposed solution here, actually.
Unison seems to not run under High Sierra and it seems it will never do, so beside alle the other reasons to go with native_osx, that's a very practical one I guess