yarn install hangs after done inside docker container

[asgerhallas]

What is the current behavior?

When running "yarn install" it gets stuck after printing done, but eventually continues after about 5-10 minutes. It looks like this:

[4/4] Building fresh packages...
success Saved lockfile.
Done in 65.96s.

It does so both on version 1.10.1 and 1.12.1.

What is the expected behavior?
I expect it to run to completion and exit.

Please mention your node.js, yarn and operating system version.
Running in Docker with Windows containers using this image: https://hub.docker.com/r/stefanscherer/node-windows/.

Node version is 10.13.0. Windows is version 10.0.17134.345.

[arcanis]

It seems related to #6584 (even though #6584 mentions that it works fine on 1.10 ...).

I don't have Docker, so I'd appreciate some help to figure out what might cause it to behave so badly.

[asgerhallas]

Sure, though I'm not really sure where to start :)

But I just tried using npm install in lieu of yarn, and this has the same issue. So it might be something docker does (maybe when add a lot of files to an image?), or a specific package, I install.

[asgerhallas]

I'm pretty certain now, that this is a docker issue - and not yarn nor npm. I'll close this issue and return if further investigation proves me wrong :)

[such]

@arcanis I fixed #6584 to mention that it does not work fine in 1.10. I had to rollback all the way to 1.7 to make it work.

[arcanis]

Oh? Can you try disabling the part of the code that uses the CLONEFILE constant if available? I think in 1.8 or 1.9 we started using this flag to clone files of possible, but maybe it isn't well supported by Docker

[arcanis]

The exact constant to locate and disable is constants.COPYFILE_FICLONE

[such]

I checked inside the container and I have:

> require('fs').constants.COPYFILE_FICLONE
2

In cli.js i replaced:

var ficloneFlag = (_fs2 || _load_fs2()).constants.COPYFILE_FICLONE || 0;

by:

var ficloneFlag = 0;

But it was just as slow :(

Do you want me to try something else?

[oreporan]

I understand this was closed since it seems to be a docker issue - but does anyone have a solution? Is there an issue in docker for this?

[nathanchase]

We are seeing the same thing inside Docker Windows containers with yarn. It gets to the end of yarn install, and then hangs after "Done" for several minutes before moving on to the next step of the Dockerfile.

[nathanchase]

I'm pretty certain now, that this is a docker issue - and not yarn nor npm. I'll close this issue and return if further investigation proves me wrong :)

@asgerhallas Did you ever discover what it was about Docker using Windows containers that caused things to hang up at the end of a yarn installl / npm install?

[asgerhallas]

@nathanchase from what I remember, the most likely cause was that a lot of files are added in this step and thus the commit takes time, depending on the Storage Driver:

All filesystem changes are written to this thin writable container layer. Each time a container is committed (manually or as part of building a Dockerfile), the storage driver needs to provide a list of modified files and directories relative to the base image to create a new layer. Some drivers keep track of these changes at run time and can generate that list easily but for drivers with no native handling for calculating changes Docker provides NaiveDiffDriver. This driver produces a list of changes between current container filesystem and its parent layer by recursively traversing both directory trees and comparing file metadata. This operation is expensive for big images with many files and directories.
(from here https://dev.to/jarekprzygodzki/a-curious-case-of-slow-docker-image-builds-2o7k)

I don't know if this is actually the issue (I am not very knowledgeable about Docker, so please correct me if I'm misunderstanding how it works) and I ended up going another route (non-docker) because I ran out of time.

On one hand it seems unlikely that a diff of "one directory added with a lot files" should take that long, but on the other hand it's not impossible that the Windows implementation of the Storage Driver is lacking - I found that many things seemed a little rough at the edges for Docker on Windows.

Unfortunately the docs says you can't change the Storage Driver for Docker Desktop on Windows (see https://docs.docker.com/storage/storagedriver/select-storage-driver/). Could be interesting to run it on Docker Enterprise for Windows Server, to see if that makes a difference.

[asgerhallas]

Also this: https://github.com/yarnpkg/yarn/issues/749#issuecomment-253317395.

Might remedy some of the problem, if it's actually the number of files that matters.