Docksal: Allow overriding or hooking into "fin start" and "fin stop"

[mike-potter]

Feature request: I'd like a way to hook into the "fin project start" and "fin project stop" commands. I'm in the middle of adding support for Mutagen to replace Unison (I'll have a blog in a week or two I hope) because it's a game-changer for docker file system performance. However, similar to unison, it requires a "sync" to start when you start a project (and terminate the sync when stopping the project).

So yes, I know I can create my own "pstart" or "pstop" commands, but I'm trying to do something for our dozens of existing projects that doesn't change our normal workflow.

[achekulaev]

There is a ticket for the hooks system, that I have essentially abandoned. This case is one of a handful ones, when implementing hooks in addition to commands does really make sense. However since it is so rare I would rather bake that functionality into fin itself.

That is provided you succeed with the whole thing, of course. Because I tried mutagen before and it has design issues regarding the propagation of the x bit that rendered the tool non-useful for Docksal purposes.

[mike-potter]

I can post a separate topic on Mutagen, but it is working extremely well for us here. I saw your issue about the x bit but I don't see where that affects us at all. We are setting the default file permission to 644 and it's working just like git in how it handles the x bit. Drupal sites working fine, and our builds are working fine.

Also, we do not try running Mutagen within a container (like Docksal does with unison) since that seems to defeat the purpose of speeding up file system access. No need to another intermediate container that will cause the initial sync to take forever (unison in Docksal takes >10min for the initial file sync, whereas local Mutagen takes <2min).

This is a game changer for us because our project builds became 10x faster with Mutagen vs the "nfs" option in Docksal and doesn't have any of the crashing and unstable problems that "unison" has.

No Docksal changes are actually needed for Mutagen. We use the existing DOCKSAL_VOLUMES=none option and then just have a mutagen.yml file and do a mutagen project start right after we do a fin project start.

If you look into the hooks again for this, maybe you can better abstract what is already in fin regarding the unison-sync-wait command. Given how unreliable unison is for real-life projects (anybody with a /node_modules or /vendor folder) I really don't see ever using unison again in Docksal, and yet you have that support hard-coded in a way that doesn't allow alternatives to be hooked in.

[achekulaev]

I see. Could you share an example of mutagen.yml?

[mike-potter]

Sure:

sync:
  defaults:
    mode: "two-way-resolved"
    ignore:
      vcs: true
      paths:
        - "*.sql"
    permissions:
      defaultFileMode: 644
      defaultDirectoryMode: 755
  mySyncSession:
    alpha: "."
    # CHANGE THIS to the name of your project CLI container.
    beta: "docker://docker@myproject_cli_1/var/www"

[achekulaev]

Regarding hooks. I would need to take another look then. Back then the idea was to make hooks run inside cli, but here you would actually need it on the host, which means hooks need to run like regular commands instead, since you have mutagen installed locally.

(BTW I don't think mutagen in the container would be as slow as unison. Unison is just slow because it's old)

[mike-potter]

Anything running inside the container would have the same performance problem as nfs. In your unison container, for example, you mount the local file system into the unison container using the default volume mount. So that has the same performance problem as normal docker containers to get the files so unison itself can see them, which then syncs into the CLI container.

When we initially used Unison for Outrigger (before we tried Docksal) we had initial unison file syncs just as fast as Mutagen. So it isn't the fault of "unison being old" it's the specific way Docksal is doing the unison support.

(and yes, in this case the hooks need to run local). I think if I wanted to run hooks inside the container I would just extend the container Dockerfiles and add commands there (or to their Command or entry point, etc) . But there is no way to run local hooks with that method.

[achekulaev]

This is literally the first time I hear it! I have never tried unison locally myself, so I had no idea that it is much faster this way.

Is there a reason why not use Unison locally instead of Mutagen locally?

[mike-potter]

We used Unison with Docker a lot in the past, even before using Docksal. The problem with Unison is that its unstable when handling Mac OSX fsevent overflows. My stress-tests for any file syncing with Docker is to go into the container (cli) and do something like: rm -rf node_modules; npm install or rm -rf vendor; composer install on real projects that have thousands of small files in these directories. And then also do this locally. In both cases, measure the time lag between when the npm-install or composer-install finishes and the fully-synced filesystem being available.

On the MacOS, the fsevents buffer that tracks file system changes can overflow with the large number of file deletions/creations. The Unison sync would crash in the background with no notice. Then developers would waste a lot of time trying to figure out why their local file changes in their IDE were not having any affect. We had to set up scripts to monitor the Unison log to try and detect those cases.

Even with "bind" mounting I have seen situations where the files within the container stop syncing with the local files. I had thought this would be impossible with the osxfs mount, but I've seen it happen several times and the only fix is to restart Docker For Mac. So these are not "Docksal problems" but are larger problems with Docker4Mac and the OSX fsevent system.

In Docksal I haven't used Unison enough to see if those crashing problems still happen, but it definitely takes Docksal unison too long for the initial directory sync after doing a fin project reset or otherwise deleting the project_root docker volume. It was taking 15-20 minutes to perform the initial file sync. When using Unison with plain Docker before we switched to Docksal it was only taking 2-3min to perform the initial file sync. So something in the Docksal Unison image is causing the sync to be slower (but I've never had the time to investigate this as it was secondary to the Unison crash issues).

When Unison crashes, it often leaves the container file system in an indeterminate step and often the only solution is to remove the volume and let it resync from scratch. Thus our projects were wasting time doing that initial file sync more often than they should.

With Mutagen there seems to be specific recovery code to handle the OSX fsevents issue. During testing last week with the above commands to try and crash it, I could see Mutagen go into an "error" or "problem" state but then gracefully recover. Also, the initial file sync with a fresh volume was only 2 minutes instead of 15. The lag from the completion of the install to the sync being completed was only about 5-10 seconds.

So, the bottom line is that we previously had given up on Unison because its unreliability was a larger problem than its increase in performance. But now with Mutagen we can have both the improved performance, AND the reliability required (along with the ability for file changes to trigger inside the container, which both Unison and Mutagen have, but NFS did not).

(Caveat: We've only been using Mutagen for about a week, and while it looks extremely promising, it's possible there are side effects we have yet to discover)

[achekulaev]

@mike-potter thanks for the detailed answer.

[undersound]

@mike-potter thanks for your comprehensive answer, really helpful. I just heard about mutagen and I am curious if you are still having satisfying experiences using it. I have heard there are some issues still but I am wondering if you experience those too.

[lpeabody]

@undersound it's anecdotal but I saw Moshe make an enthusiastic post about it working with his mass.gov local setup a few days ago in Drupal Slack chat https://drupal.slack.com/archives/C4N6HELQL/p1600698685012900.

[lpeabody]

Here is the draft PR he opened to demonstrate his testing https://github.com/massgov/openmass/pull/470

[mike-potter]

I think this issue got sidetracked with the discussion of Mutagen and file system performance.

Just wanted to bump it and re-interate the need to be able to customize the fin start command. Some sort of "post-start" hook would be extremely help for both "start" and "stop". Seems like adding a .docksal/hooks folder and adding a couple of optional scripts that could be "sourced" from the fin script would be easy to implement.

[jajajaime]

@mike-potter did you ever end up writing a blog post about this?

[achekulaev]

1. Do we run them no matter what? Regardless of the results of fin start of fin stop?
2. What if some custom command runs fin start or fin stop, should the hooks be executed?

[mike-potter]

1. If fin start failed then it would be ok to not run the hook.
2. Yes, it shouldn't matter what triggered the fin start

Remember, in my specific use case I'm simply trying to replace the previously bundled Unison integration with my own Mutagen integration. In fin you would specifically call the unison "sync" when doing a fin-start so I'm just trying to hook in that same way.

Basically, look in fin and wherever you see unison_sync_wait you would call the hook. Obviously a more generic "hook mechanism" is a larger issue: do we need both "pre-" and "post-" hooks? Just for "fin start" or also for "fin stop"? What about "fin reset" etc?

But for this specific issue request, I'd just be happy with a hook that allowed me to use my own file syncing mechanism instead of the hard-coded Unison support.

[jacobsaw]

Echoing Mike's request.

I manage local development setups across a few teams and the developers have a wide range of understanding when it comes to the tools in the stack.

I'm reluctant to switch everyone over to Mutagen even though I've noticed significant improvements until I can customize fin start and fin stop to include Mutagen commands. Devs start/stop frequently enough that having them run additional commands would be a headache and I'd love to avoid a custom command to keep the workflow simple.

[nicoschi]

A hook system will be much appreciated for a better management, meantime for the Mutagen specific case I'm starting to use a custom Docksal command I created to initialize, start, stop and restart a Docksal project with a Mutagen project inside at the same time. @jacobsaw, if interested in you can find it in Mutagen for Docksal repository.