Bazel: Resolve Windows long path issue during C++ compilation

[meteorcloudy]

https://stackoverflow.com/questions/47412512/bazel-build-nccl-archive-nccl-could-not-resolve-label-ws2-32-lib
When I tried to help a user build tensorflow_serving on Windows, I encountered an error Cannot open compiler generated file: '': Invalid argument:

  C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/amd64/cl.exe /c external/org_tensorflow/tensorflow/core/profiler/internal/advisor/internal_checker_runner_dummy.cc /Fobazel-out/msvc_x64-py3-opt/bin/external/org_tensorflow/tensorflow/core/profiler/internal/advisor/_objs/internal_checker_runner_dummy/external/org_tensorflow/tensorflow/core/profiler/internal/advisor/internal_checker_runner_dummy.o /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0600 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_SECURE_NO_WARNINGS /D_SILENCE_STDEXT_HASH_DEPRECATION_WARNINGS /bigobj /Zm500 /J /Gy /GF /EHsc /wd4351 /wd4291 /wd4250 /wd4996 /Iexternal/org_tensorflow /Ibazel-out/msvc_x64-py3-opt/genfiles/external/org_tensorflow /Iexternal/protobuf_archive /Ibazel-out/msvc_x64-py3-opt/genfiles/external/protobuf_archive /Iexternal/bazel_tools /Ibazel-out/msvc_x64-py3-opt/genfiles/external/bazel_tools /Iexternal/com_google_absl /Ibazel-out/msvc_x64-py3-opt/genfiles/external/com_google_absl /Iexternal/nsync /Ibazel-out/msvc_x64-py3-opt/genfiles/external/nsync /Iexternal/eigen_archive /Ibazel-out/msvc_x64-py3-opt/genfiles/external/eigen_archive /Iexternal/local_config_sycl /Ibazel-out/msvc_x64-py3-opt/genfiles/external/local_config_sycl /Iexternal/gif_archive /Ibazel-out/msvc_x64-py3-opt/genfiles/external/gif_archive /Iexternal/jpeg /Ibazel-out/msvc_x64-py3-opt/genfiles/external/jpeg /Iexternal/com_googlesource_code_re2 /Ibazel-out/msvc_x64-py3-opt/genfiles/external/com_googlesource_code_re2 /Iexternal/farmhash_archive /Ibazel-out/msvc_x64-py3-opt/genfiles/external/farmhash_archive /Iexternal/fft2d /Ibazel-out/msvc_x64-py3-opt/genfiles/external/fft2d /Iexternal/highwayhash /Ibazel-out/msvc_x64-py3-opt/genfiles/external/highwayhash /Iexternal/png_archive /Ibazel-out/msvc_x64-py3-opt/genfiles/external/png_archive /Iexternal/zlib_archive /Ibazel-out/msvc_x64-py3-opt/genfiles/external/zlib_archive /Iexternal/snappy /Ibazel-out/msvc_x64-py3-opt/genfiles/external/snappy /Iexternal/protobuf_archive/src /Ibazel-out/msvc_x64-py3-opt/genfiles/external/protobuf_archive/src /Iexternal/bazel_tools/tools/cpp/gcc3 /Iexternal/nsync/public /Ibazel-out/msvc_x64-py3-opt/genfiles/external/nsync/public /Iexternal/eigen_archive /Ibazel-out/msvc_x64-py3-opt/genfiles/external/eigen_archive /Iexternal/gif_archive/lib /Ibazel-out/msvc_x64-py3-opt/genfiles/external/gif_archive/lib /Iexternal/gif_archive/windows /Ibazel-out/msvc_x64-py3-opt/genfiles/external/gif_archive/windows /Iexternal/farmhash_archive/src /Ibazel-out/msvc_x64-py3-opt/genfiles/external/farmhash_archive/src /Iexternal/png_archive /Ibazel-out/msvc_x64-py3-opt/genfiles/external/png_archive /Iexternal/zlib_archive /Ibazel-out/msvc_x64-py3-opt/genfiles/external/zlib_archive /D__CLANG_SUPPORT_DYN_ANNOTATION__ /DEIGEN_MPL2_ONLY /DTENSORFLOW_USE_ABSL /DTF_USE_SNAPPY /showIncludes /MD /O2.
C:\tmp\_bazel_pcloudy\dpitg86y\execroot\tf_serving\external\org_tensorflow\tensorflow\core\profiler\internal\advisor\internal_checker_runner_dummy.cc : fatal error C1083: Cannot open compiler generated file: '': Invalid argument

It turns out that the object file the compiler trying to write C:/tmp/_bazel_pcloudy/dpitg86y/execroot/bazel-out/msvc_x64-py3-opt/bin/external/org_tensorflow/tensorflow/core/profiler/internal/advisor/_objs/internal_checker_runner_dummy/external/org_tensorflow/tensorflow/core/profiler/internal/advisor/internal_checker_runner_dummy.o is longer than 270 characters.

This happens when I already set the output directory to a short path C:/tmp.

[laszlocsomor]

I ran into the same limitation while working on #4148.

@ulfjack mentioned on 16/11/2017 on an internal mail thread that we are looking into shortening C++ output paths. Here, as well as with #4148, we are duplicating the package path in the output path. AFAICT this is only necessary when cc_*.srcs includes two files with the same basename.

I don't know Bazel's C++ compilation machinery enough to tell with certainty whether we could change these output paths, say, to bazel-out/configname/path/to/rule._objs/basename.o and only include the package name of the source file if there'd be a basename.o clash otherwise.

[laszlocsomor]

paging Mr. @mhlopko : what problems do you think we have to solve to get to the state I mentioned in my previous comment?

[meteorcloudy]

Can we use a hash string under _objs instead of the full package path?
Something like C:/tmp/_bazel_pcloudy/dpitg86y/execroot/bazel-out/msvc_x64-py3-opt/bin/external/org_tensorflow/tensorflow/core/profiler/internal/advisor/_objs/<short_and_unique_hash>/internal_checker_runner_dummy.o

[laszlocsomor]

We could as well include the object file's name in the hash, so it'd be c:/<output_root>/bazel-out/<configname>/bin/<cc_rule_package_path>/_objs/<hash_of_cc_rule_name_and_root_relative_path_of_source>.o

[hlopko]

Unfortunately I have no idea. I didn't even go that far to find out that we duplicate the path when there are multiple srcs with same basename. How sure are you this is the only case? I was also more interested in _solib contents, I'm not sure how much overlap there is.

In other words, I'm useless to you here :)

[laszlocsomor]

How sure are you this is the only case?

I'm not sure it's the only case, or that it's even the reason we compute paths the way we do. It's just a reason I could think of.

In other words, I'm useless to you here :)

I think you're wrong :P

[laszlocsomor]

I'm considering this for Q1-2018 because our users have reported that lack of good documentation and best practices is a real pain.

However, I'm leaving it as P2, because there are workaround:

• --output_user_root=x:/ or
• shortening target names, or
• making the package tree flatter.

[excitoon]

I've just encountered this issue by myself. I'd like to add that it can be solved in cl python wrapper substituting paths with shorter paths using symlinks.
Also I've created small repository to easily reproduce this issue:
https://github.com/excitoon/bazel-issues/tree/master/windows-long-file-paths-of-objects
Currently it blocks our company from moving our projects to bazel -_-.

[laszlocsomor]

@excitoon : Thanks for the repro case! We are actively looking for solutions, currently evaluating different approaches. This is one of our highest priority bugs.

[laszlocsomor]

Bumping to P1 because workarounds only get us so far, and it's only a matter of time before more people hit this bug. (Bazel itself started hitting it this week.)

[davido]

We are actively looking for solutions, currently evaluating different approaches.

Have you considered to consequently write response files and then calling cl with "@responsefile.rsp"?

[laszlocsomor]

Does that enable the compiler/linker to deal with long paths?

[rongjiecomputer]

I've just encountered this issue by myself. I'd like to add that it can be solved in cl python wrapper substituting paths with shorter paths using symlinks.

Invocating Python is pretty slow, and the time will quickly adds up with 1000+ compilation units. Wrapper script is not an ideal solution IMO.

Does that enable the compiler/linker to deal with long paths?

No.

In C:/tmp/_bazel_pcloudy/dpitg86y/execroot/bazel-out/msvc_x64-py3-opt/bin/external/org_tensorflow/tensorflow/core/profiler/internal/advisor/_objs/internal_checker_runner_dummy/external/org_tensorflow/tensorflow/core/profiler/internal/advisor/internal_checker_runner_dummy.o, we can clearly see that external/org_tensorflow/tensorflow/core/profiler/internal/advisor is repeated twice, this is as good as halving the already-very-short 270 limit.

In CMake, it will probably only something like C:/tensorflow/build/tensorflow/core/CMakeFiles/profiler/internal/advisor.dir/internal_checker_runner_dummy.o

[meteorcloudy]

Invocating Python is pretty slow, and the time will quickly adds up with 1000+ compilation units. Wrapper script is not an ideal solution IMO.

I agree, using a wrapper script is not a solution under discussion. We're focusing on how to short the object file path in Bazel.

[excitoon]

Invocating Python is pretty slow, and the time will quickly adds up with 1000+ compilation units. Wrapper script is not an ideal solution IMO.

That's right, but Python wrapper is already being invoked for each unit on MSVS (it does some flags substitutions and has other logic).

[laszlocsomor]

@excitoon : No, there's no Python wrapper for MSVC anymore. Bazel runs the compiler/linker directly.

[meteorcloudy]

That's right, but Python wrapper is already being invoked for each unit on MSVS (it does some flags substitution and has other logic)

No, the wrapper script has been removed since 0.5.3, see https://github.com/bazelbuild/bazel/commit/425f2496d65d1e190f633236575009e49e1252b3

[laszlocsomor]

@rongjiecomputer :

we can clearly see that external/org_tensorflow/tensorflow/core/profiler/internal/advisor is repeated twice, this is as good as halving the already-very-short 270 limit.

Yes, we made the same observation. @meteorcloudy is currently designing different ways to shorten the object file output paths.

Yes, CMake would probably just put the object file next to the source file, which is fine only as long as you don't compile the file with multiple compilers and/or different compiler options within the same build.

[rongjiecomputer]

which is fine only as long as you don't compile the file with multiple compilers and/or different compiler options within the same build.

In that I just use C:\tensorflow\build for compiler/configuration 1 (said pure MSVC + Debug) and C:\tensorflow\build2 for compiler/configuration 2 (said Clang on Windows + Release). This is a way that I think Bazel can't take anymore since by now a lot of people probably already depend on bazel-* folders instead of build folder from make/cmake time.

[davido]

Does that enable the compiler/linker to deal with long paths?

Starting from Windows 10, version 1607, MAX_PATH limitations have been removed from common Win32 file and directory functions. However, you must opt-in to the new behavior.

A registry key allows you to enable or disable the new long path behavior. To enable long path behavior set the registry key at HKLM\SYSTEM\CurrentControlSet\Control\FileSystem LongPathsEnabled (Type: REG_DWORD).

See: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx

[laszlocsomor]

@davido: Yes, this is great, but only available in Win10 + Anniversary Update and later. Win7 and 8 don't support this.

[davido]

@laszlocsomor So, two options: recover python wrapper from ancient Bazel version for older Os versions, while natively support long path starting from Win10 + Anniversary Update and later. All this without touching any bit in Bazel internals, well except restoring the Python wrapper behaviour.

[excitoon]

I guess, as C++ application, cl.exe is compiled with MAX_PATH=260 inside anyway and therefore that registry key won't affect it.

[laszlocsomor]

Yes, @excitoon makes a good point.
We can't convince cl.exe to accept long paths, not even with a wrapper, nor with the registry key you mentioned.

[meteorcloudy]

Hey, everyone, this issue has been resolved by https://github.com/bazelbuild/bazel/commit/bb9ae6a174b8cd255a62249f01919426f1d817f8.
You can now use --experimental_shortened_obj_file_path option to have a short object file path with Bazel@HEAD. This change will be released with 0.13.0 and the short object file will be default in the future. For more details of this change, please read the commit message.

[rongjiecomputer]

@meteorcloudy Just boostraped Bazel with this flag on and it works flawlessly! Will try to build Tensorflow with this flag when I am free (though it looks like Tensorflow's Bazel Windows CI is red again).

I notice that the change only affects C++. com\google\devtools\build\android repeats twice in C:\Users\me\GitHub\bazel\bazel-bin\src\tools\android\java\com\google\devtools\build\android\_javac\android_builder_lib\libandroid_builder_lib_classes\com\google\devtools\build\android. Since Java package name also can be very long, I think at some point we will see a Java project hitting this issue like C++ project.

[hlopko]

\o/

[meteorcloudy]

@rongjiecomputer Glad to hear it worked! I'm working on fixing the TF's Windows build and setting up a presubmit for it. Should be done next week.

Yes, this change only affected C++, because it's only cl.exe (and other MSVC build tools) that doesn't support long path. I tested with java.exe and javac.exe, they both accept long path. So we are saved here.

[RickVM]

@rongjiecomputer how did you bootstrap Bazel?
I tried downloading the current git repo and use msys2 to execute compile.sh but this returns:
ERROR: Must specify PROTOC if not bootstrapping from the distribution artifact
where it suggests to use bazel to build from source. Or use the developer checkout as output files are not included.

I tried to build from source with the latest bazel release (0.11) and after 700 seconds of downloading the very first package of the build list i get an error stating there is no such package as androidsdk.

I figure this is not the right path and I'm most likely missing something here as I'm new to bazel and just trying to install Tensorflow serving, for which I need the fix that will be implemented in 0.13.

[rongjiecomputer]

@RickVM I just run bazel build -c opt --experimental_shortened_obj_file_path=true //src:bazel.exe directly in the git repo.