Bazel: Windows: rebuild and clean sometimes fail with DirectoryNotEmptyException

[meteorcloudy]

When I use Bazel 0.15.2 to rebuild Bazel I got this error sometimes:

ERROR: C:/tools/msys64/home/pcloudy/workspace/bazel/src/main/java/com/google/devtools/build/skyframe/BUILD:25:1: Building src/main/java/com/google/devtools/build/skyframe/libskyframe.jar (72 source files) and running annotation processors (AutoCodecProcessor, OptionProcessor) failed (Exit 1): java.exe failed: error executing command
  cd C:/users/pcloudy/_bazel_pcloudy/3uxo2r6v/execroot/io_bazel
  SET LC_CTYPE=en_US.UTF-8
  external/embedded_jdk/bin/java.exe -Xbootclasspath/p:external/bazel_tools/third_party/java/jdk/langtools/javac-9+181-r4173-1.jar -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/x64_windows-fastbuild/bin/src/main/java/com/google/devtools/build/skyframe/libskyframe.jar-2.params
java.io.IOException: Cannot clean 'bazel-out\x64_windows-fastbuild\bin\src\main\java\com\google\devtools\build\skyframe\_javac\skyframe\libskyframe_classes'
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.cleanupDirectory(SimpleJavaLibraryBuilder.java:91)
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.prepareSourceCompilation(SimpleJavaLibraryBuilder.java:53)
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.compileJavaLibrary(SimpleJavaLibraryBuilder.java:113)
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:132)
        at com.google.devtools.build.buildjar.BazelJavaBuilder.processRequest(BazelJavaBuilder.java:105)
        at com.google.devtools.build.buildjar.BazelJavaBuilder.runPersistentWorker(BazelJavaBuilder.java:67)
        at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:45)
Caused by: java.nio.file.DirectoryNotEmptyException: bazel-out\x64_windows-fastbuild\bin\src\main\java\com\google\devtools\build\skyframe\_javac\skyframe\libskyframe_classes\com\google\devtools\build
        at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:266)
        at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
        at java.nio.file.Files.delete(Files.java:1126)
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder$1.postVisitDirectory(SimpleJavaLibraryBuilder.java:85)
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder$1.postVisitDirectory(SimpleJavaLibraryBuilder.java:73)
        at java.nio.file.Files.walkFileTree(Files.java:2688)
        at java.nio.file.Files.walkFileTree(Files.java:2742)
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.cleanupDirectory(SimpleJavaLibraryBuilder.java:71)
        ... 6 more
Target //src:bazel failed to build
INFO: Elapsed time: 36.284s, Critical Path: 14.20s
INFO: 44 processes: 23 local, 21 worker.
FAILED: Build did NOT complete successfully

pcloudy@pcloudy0-w MSYS ~/workspace/bazel
$ bazel clean
Starting local Bazel server and connecting to it...
INFO: Starting clean.
ERROR: C:/users/pcloudy/_bazel_pcloudy/3uxo2r6v/execroot/io_bazel/bazel-out/x64_windows-fastbuild/bin/src/main/java/com/google/devtools/build/lib/remote/blobstore/_javac/blobstore/libblobstore_classes/com/google (Directory not empty)

pcloudy@pcloudy0-w MSYS ~/workspace/bazel
$ ls /c/users/pcloudy/_bazel_pcloudy/3uxo2r6v/execroot/io_bazel/bazel-out/x64_windows-fastbuild/bin/src/main/java/com/google/devtools/build/lib/remote/blobstore/_javac/blobstore/libblobstore_classes/com/google

pcloudy@pcloudy0-w MSYS ~/workspace/bazel
$ bazel clean
INFO: Starting clean.
ERROR: C:/users/pcloudy/_bazel_pcloudy/3uxo2r6v/execroot/io_bazel/bazel-out/x64_windows-fastbuild (Directory not empty)

pcloudy@pcloudy0-w MSYS ~/workspace/bazel
$ bazel clean
INFO: Starting clean.

FYI @laszlocsomor

[laszlocsomor]

Does bazel clean eventually succeed?

[meteorcloudy]

Yes, it took me three times although.

[laszlocsomor]

Can you repro reliably?

[meteorcloudy]

It happens constantly for me in the following situation:

# Clone bazel repo @ HEAD
$ bazel build //src:bazel
$ git reset 4c9149d --hard
$ bazel build //src:bazel

[meteorcloudy]

But I'm using 0.15.2, I'll try if it still happens with 0.16.1

[laszlocsomor]

I believe @meteorcloudy is working on this -- reassigning.

[laszlocsomor]

@meteorcloudy , have you seen this error with recent releases?

[meteorcloudy]

I haven't seen this after setting my output user root to C:/src/tmp, I guess it has something to do with the Windows Defender. I didn't receive any bug report from users, either. So I think it's fine to close issue.

[meteorcloudy]

Reopen since @laszlocsomor and I have both been seeing this issue for a while again, I'll look into it.

[meteorcloudy]

Related stackoverflow question: https://stackoverflow.com/questions/31606978/odd-behaviour-when-deleting-files-with-files-delete

[laszlocsomor]

Related stackoverflow question: https://stackoverflow.com/questions/31606978/odd-behaviour-when-deleting-files-with-files-delete

That was a great read, thanks!

[laszlocsomor]

@meteorcloudy : Can we close this bug?

[laszlocsomor]

ping

[meteorcloudy]

@laszlocsomor Sorry for the late response. Not yet, because in the JavaSimpleLibraryBuilder I sometimes still get DirectoryNotEmpty exception while rebuilding. It is not using our JNI code to delete file and directory, instead it uses MoreFiles.deleteRecursively from guava.

Did you ever have the same error?

[laszlocsomor]

Thanks for the update!

I don't remember clearly anymore if I've seen it or not.

[meteorcloudy]

Basically I can reproduce after about 50 commits change in Bazel repo,
eg.

$ bazel build //src:bazel
$ git reset HEAD~50 --hard
$ bazel build //src:bazel

[laszlocsomor]

@meteorcloudy : Any updates? Is P1 still adequate?

[meteorcloudy]

I am not working on this recently. I'm not getting any report from users, so I guess P2 should be more appropriate. But I can still reproduce this bug on my local machine, so I'll keep it open.

[laszlocsomor]

Hey @meteorcloudy , I'd like to take a look at this. Can you still repro this locally? Could you help me repro?

[meteorcloudy]

@laszlocsomor , thanks!
Yes, I can still repo this, basically following https://github.com/bazelbuild/bazel/issues/5907#issuecomment-413482068

[meteorcloudy]

Just reset to an old enough commit which will trigger a rebuild for most of Java code.

[meteorcloudy]

This issue will be fixed from guava side, there's internal work going on, so I'm closing this one.