Bazel: Include checking fails on windows for generated C header files (undeclared inclusion(s))

[ozio85]

Description of the problem / feature request:

Include checking fails on windows for generated C/CPP header files. (the compilation is OK, but when Bazel checks the files afterwards, the header file is not recognized.)

Feature requests: what underlying problem are you trying to solve with this feature?

The include checking fails in CppCompileAction.java, in validateInclusions.

The line which is supposed to say the header file is ok is (around line 807):

if (allowedIncludes.contains(input)) {
  continue;
}

The problem is that the artifact execroot differs on Windows and Linux for all generated files:
Edit: There seem to be a more intricate problem then just "all generated files". It might have something to do with circular dependencies? I am trying to get a minimal example rolling.

Linux:

• my/generated/header/path/myheader.h

Windows:

• bazel-out/x64_windows-fastbuild/bin/my/generated/header/path/myheader.h

which causes the include check to fail.

A quick fix would be to add yet another check:

// Problem on windows with generated headers.
for (Artifact artifact : allowedIncludes) {
  if (artifact.getRootRelativePath().toString().equals(input.getExecPath().toString()))
  {
     continue checking;
  }
}

But it is not the clean way forward.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

• Create a header-file with a Skylark rule (if gen-rule, set output_to_bindir =True).
• Include the file in a C file.
• Try to compile on windows.

Edit: The problem is more intricate than this, i am trying to create a minimal example.

What operating system are you running Bazel on?

Windows

What's the output of bazel info release?

release 0.21.0

Have you found anything relevant by searching the web?

Might be the same cause as to #3828 and #6337

Any other information, logs, or outputs that you want to share?

No

[excitoon]

Have you added $(BINDIR) to include directories list?

[ozio85]

Have you added $(BINDIR) to include directories list?

I tried, and it doesn't help, but will it really help the include checking? The compilation is fine, the object file is built, but when the includes are to be validated, it fails.

[laszlocsomor]

Please create a minimal repro.

From face value the bug sounds like it might be a duplicate of #5640. @ozio85, does https://github.com/bazelbuild/bazel/issues/5640#issuecomment-416957997 sound plausible in this case?

[ozio85]

An easy repro is to use buildfarm to build an extremely simple project, which passes on linux, but fails on windows, see this build farm issue (unclear if it is the same cause, but it triggers a similar behavior in a reproducible way):
Buildfarm issue: #195

It might be one way to trigger this bug, but I am running in to it every now and then.

[laszlocsomor]

I managed to repro this: ~https://github.com/laszlocsomor/projects/tree/3290d2d9e2af9067466e38576e10b8263ac5660a/bazel/gh-7030-buildfarm-includecheck-fails~
~https://github.com/laszlocsomor/projects/tree/aa29a64a7afc3b2e375cc7af2816a6ecf921b35d/bazel/gh-7030-buildfarm-includecheck-fails~

Third time is the charm:
https://github.com/laszlocsomor/projects/tree/44c9f3a601986ee8f01768f80deff1ce2835e88f/bazel/gh-7030-buildfarm-includecheck-fails

[laszlocsomor]

@meteorcloudy does this bug look familiar to you? Look at this first: https://github.com/laszlocsomor/projects/tree/44c9f3a601986ee8f01768f80deff1ce2835e88f/bazel/gh-7030-buildfarm-includecheck-fails/README.md -- it shows the bug and a repro.

[meteorcloudy]

@laszlocsomor Thanks for the repo, I am looking into this! Could be a similar bug to https://github.com/bazelbuild/bazel/issues/6847

[laszlocsomor]

Great! Thank you!

[meteorcloudy]

I debugged on this a bit, it indeed is a similar issue with #6847.

The MSVC compiler always output the absolute path of the header files it detected, but of course a file under the remote worker's execution directory is not declared as dependency, so it failed at header check.

https://github.com/bazelbuild/bazel/commit/a1aa5c14782b38d8edf86953230f76ba2662378d provided a work around for this problem. When Bazel sees an absolute header file path, it first tries to find execroot in the file path, then converts it to a relative path by ignoring everything before execroot. This relative path would be the correct path for the file actually declared in dependency.

But the execroot used for actions in buildfarm worker doesn't contain execroot
https://github.com/bazelbuild/bazel-buildfarm/blob/master/src/main/java/build/buildfarm/instance/AbstractServerInstance.java#L274

To solve this problem, we have to append execroot to the execution directory. Also it seems we don't have workspace name in path, that is also necessary.
Basically,

D:/gh-7030/worker/default_memory_instance/operations/af056149-3fd3-4c74-9c8e-3f8880e89041/foo.h

should become

D:/gh-7030/worker/default_memory_instance/operations/af056149-3fd3-4c74-9c8e-3f8880e89041/execroot/__main__/foo.h

The idea is the buildfarm worker's execution directory should mimic the actual Bazel execution root.

[meteorcloudy]

@werkt @buchgr Does buildfarm know the workspace name of the project it's building?

[buchgr]

@meteorcloudy no

[werkt]

@werkt @buchgr Does buildfarm know the workspace name of the project it's building?

Not explicitly. It would be capable of observing output paths and making a reasonable guess, but it would be predicated on knowing that a client is [a particular version of] bazel due to semantics. There is no explicit naming for an execution above the action, except in opaque IDs in metadata, which likely does not contain anything specific to a workspace or it's name.

[buchgr]

It would be capable of observing output paths and making a reasonable guess

please don't :-)

[ozio85]

I can confirm that #6931 fixes my include problem. However, i haven't been able to try it on the small build-farm example (seems like bazel and docker on windows does not work well together at the moment).

[laszlocsomor]

@ozio85 : Cool, glad to hear that! Can we close this bug or is there anything else to do?

[ozio85]

No lets close it. I open i new if the problem remains in bazel build-farm.