Bazel: repository_ctx.symlink on a large directory take a lot of time

[alexandrvb]

Description of the problem / feature request:

repository_ctx.symlink on a 11G directory takes about 5 min between creating symlink and creating .marker file

Feature requests: what underlying problem are you trying to solve with this feature?

We are creating symlink of python anaconda directory to our workspace

What operating system are you running Bazel on?

Ubuntu 16.04

What's the output of bazel info release?

release 0.17.2

Any other information, logs, or outputs that you want to share?

We use repository_ctx.symlink(path, ""). Size of "path" directory is 11G. Loading last package takes about 5 min:

[17:03:27][Step 3/5] Starting local Bazel server and connecting to it...
[17:03:31][Step 3/5] Loading:
[17:03:31][Step 3/5] Loading: 0 packages loaded
[17:03:32][Step 3/5] Loading: 21 packages loaded
<...>
[17:06:59][Step 3/5] Analyzing: 865 targets (1010 packages loaded)
[17:08:01][Step 3/5] Analyzing: 865 targets (1010 packages loaded)
[17:08:22][Step 3/5] INFO: Analysed 865 targets (1011 packages loaded).
[17:08:22][Step 3/5] INFO: Found 630 targets and 235 test targets...

According to profile most time bazel spends in skyfunction: REPOSITORY_DIRECTORY.

On a bazel 0.15.2 issue isn't reproduced.

[solomatov]

Are there any update on this? Can we help somehow with fixing this issue?

[solomatov]

We investigated it further, and this problem was our fault. We created a symlink to a directory with a huge number of small files. It's no wonder that bazel tries to rescan it, and it's slow. I think, you can close it dependning on how important you think it is but it's not a problem for us anymore.

[irengrig]

I guess we can close it, as it works by contract - creates the symlinks, as it was asked.

Thank you for letting us know, @solomatov !