Bazel: Unpredictable "missing input file" error

[yinanfang]

Description of the problem / feature request:

Unpredictable "missing input file" error happens when we build java projects on Linux. Similar issue was reported here: https://github.com/bazelbuild/migration-tooling/issues/77. The workaround is to clean the cache with bazel clean --expunge but it'll make the build time much longer.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Unfortunately we don't have a good way to consistently reproduce it but it happens very often

What operating system are you running Bazel on?

Debian 8.5

What's the output of bazel info release?

release 0.14.1

Have you found anything relevant by searching the web?

Found a similar issue but can't find any solution
https://github.com/bazelbuild/migration-tooling/issues/77

Any other information, logs, or outputs that you want to share?

This is an example error message. The missing dependencies are different for every build.

[search] ERROR: missing input file '@com_google_guava_guava//jar:guava-24.0-jre.jar'
[search] ERROR: /home/jenkins/workspace/Postmerge_Pipeline/search/third_party/external/java/BUILD:5:1: //third_party/external/java:com_google_auth_google_auth_library_appengine: missing input file '@com_google_guava_guava//jar:guava-24.0-jre.jar'
[search] ERROR: /home/jenkins/workspace/Postmerge_Pipeline/search/third_party/external/java/BUILD:5:1 1 input file(s) do not exist

[sigaev]

I am having a similar issue.

[iirina]

It may be because Bazel upgraded to guava-24.1, so there is no guava-24.0-jre.jar. Do you use any version of Guava in your code?

[clusc]

It seems to happen randomly for a lot of java libraries, and we did verified the external resources exist.

[mmorearty]

We hit this error more and more often. It seems like someone on our team hits it every day. We have had no luck getting a repro scenario.

Some data points:

• Both Mac and Linux.
• Turning off our remote cache does not help.
• We are currently on Bazel 0.15.1 and are still hitting this.
• Our giant spike in occurrences happened when we upgraded from Bazel 0.12.0 to 0.15.0. (Yes I realize that's a big jump, not too helpful)
• bazel clean --expunge seems to fix it (until it happens again).
• When we spent a while trying to debug this on one person's machine, the error message — which jar file was "missing" — was different each time we ran Bazel. This was true even with --jobs=1.
• The "missing file" is always a jar file from an external workspace.
• Usually it matches this pattern: @WORKSPACE_NAME//jar:JARFILE_NAME
• For one user (below), it matched this pattern: @local_jdk//:jre/lib/JARFILE_NAME

Here are some of the possibly-relevant bazel arguments we use:

--experimental_guard_against_concurrent_changes
--experimental_strict_action_env
--spawn_strategy=sandboxed
--strategy=Scalac=worker
--worker_sandboxing=true

Here are some partial error logs, from different users:

missing input file '@com_google_code_gson_gson//jar:gson-2.8.0.jar'

ERROR: /private/var/tmp/_bazel_tylern/93dec4eefb112d8628e9fe3a29c8dbae/external/io_bazel_rules_scala/src/java/io/bazel/rulesscala/jar/BUILD:1:1: @io_bazel_rules_scala//src/java/io/bazel/rulesscala/jar:jar: missing input file '@local_jdk//:jre/lib/rt.jar'

ERROR: missing input file '@com_googlecode_json_simple_json_simple//jar:json-simple-1.1.1.jar'

ERROR: missing input file '@com_google_instrumentation_instrumentation_api//jar:instrumentation-api-0.4.3.jar'

ERROR: missing input file '@com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jackson-dataformat-yaml-2.6.7.jar'

ERROR: missing input file '@io_grpc_grpc_core//jar:grpc-core-1.10.1.jar'

missing input file '@com_google_protobuf_protobuf_java_util//jar:protobuf-java-util-3.5.1.jar'

missing input file '@io_grpc_grpc_core//jar:grpc-core-1.10.1.jar'

[mmorearty]

This seems just like https://github.com/bazelbuild/bazel/issues/5103 ... and yet I am still hitting this with Bazel 0.15.1

[mmorearty]

In the case of the one user who I know was still getting this after a bazel clean --expunge, it turns out the cause of the error was something else for him. On his machine, JAVA_HOME was not set; when he set it and did the build again, it succeeded.

A for the rest of my users who have hit this, it's possible that this problem went away for them after bazel clean --expunge. So in other words, I effectively withdraw my earlier, long comment :)

[yinanfang]

@mmorearty we upgraded bazel to 0.14.1. In the first week, we still keep getting that errors very often. We have to keep cleaning the cache to make it work. However, it has been working fine for the last 2 weeks for some reasons.
We didn't change the JAVA_HOME, so looks like that's not the cause.

[mariusgrigoriu]

This is also happening on bazel 0.23.2 when working with go repositories. In my case, I hit ctrl-c while fetching an external repository on a slow network so I could restart after connecting to a fast network.

$ bazel test //cmd/... //pkg/...
Starting local Bazel server and connecting to it...
Loading: 0 packages loaded
    Fetching @io_k8s_kubernetes; fetching 8s
^C
ERROR: build interrupted
INFO: Elapsed time: 16.766s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

After connecting to a better network:

$ bazel test //cmd/... //pkg/...
INFO: Analysed 38 targets (360 packages loaded, 10051 targets configured).
INFO: Found 30 targets and 8 test targets...
ERROR: missing input file '@io_k8s_kubernetes//vendor/github.com/onsi/ginkgo/internal/spec:specs.go'
ERROR: /private/var/tmp/_bazel_marius/c39204db2cb7e0a79ba2096a64d88d76/external/io_k8s_kubernetes/vendor/github.com/onsi/ginkgo/internal/spec/BUILD:3:1: @io_k8s_kubernetes//vendor/github.com/onsi/ginkgo/internal/spec:go_default_library: missing input file '@io_k8s_kubernetes//vendor/github.com/onsi/ginkgo/internal/spec:specs.go'
ERROR: /private/var/tmp/_bazel_marius/c39204db2cb7e0a79ba2096a64d88d76/external/io_k8s_kubernetes/vendor/github.com/onsi/ginkgo/internal/spec/BUILD:3:1 1 input file(s) do not exist
INFO: Elapsed time: 71.516s, Critical Path: 0.04s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

Immediately retrying the same command:

$ bazel test //cmd/... //pkg/...
ERROR: error loading package '': Unable to load package for '@io_k8s_kubernetes//build:workspace_mirror.bzl': BUILD file not found on package path
ERROR: error loading package '': Unable to load package for '@io_k8s_kubernetes//build:workspace_mirror.bzl': BUILD file not found on package path
INFO: Elapsed time: 0.160s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

No changes on subsequent retries. Only a bazel clean --expunge got me unstuck.

[mehrdadn]

For anyone else landing here from Google searches:

If you get missing input file even after clean/expunge (with all caching disabled):

It might be due to symbolic links in the file's path.
In particular, you might want to check your core.symlinks configuration in Git.

For example, on GitHub Actions, symlinks appear to be disabled by default.
Unless you enable symlinks and then checkout the repository, you'll get a different work tree than Bazel expects, and that can sometimes cause this error to occur.

[acecilia]

I also hit this issue with bazel 3.0.0. I have been experiencing it consistently when removing the bazel cache at /private/var/tmp/_bazel_<userName> (in macOS) and running a project. Subsequent runs after the first one did not hit the problem. The log is as follows:

ERROR: missing input file 'Carthage/build/iOS/SnapKit.framework/Headers/SnapKit-Swift.h', owner: '//:Carthage/build/iOS/SnapKit.framework/Headers/SnapKit-Swift.h'
ERROR: missing input file 'Carthage/build/iOS/SnapKit.framework/Headers/SnapKit.h', owner: '//:Carthage/build/iOS/SnapKit.framework/Headers/SnapKit.h'
ERROR: missing input file 'Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/arm.swiftmodule', owner: '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/arm.swiftmodule'
ERROR: missing input file 'Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/arm64-apple-ios.swiftmodule', owner: '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/arm64-apple-ios.swiftmodule'
ERROR: missing input file 'Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/arm64.swiftmodule', owner: '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/arm64.swiftmodule'
ERROR: missing input file 'Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/armv7-apple-ios.swiftmodule', owner: '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/armv7-apple-ios.swiftmodule'
ERROR: missing input file 'Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/armv7.swiftmodule', owner: '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/armv7.swiftmodule'
ERROR: missing input file 'Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/i386-apple-ios-simulator.swiftmodule', owner: '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/i386-apple-ios-simulator.swiftmodule'
ERROR: missing input file 'Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/i386.swiftmodule', owner: '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/i386.swiftmodule'
ERROR: missing input file 'Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/x86_64-apple-ios-simulator.swiftmodule', owner: '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/x86_64-apple-ios-simulator.swiftmodule'
ERROR: missing input file 'Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/x86_64.swiftmodule', owner: '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/x86_64.swiftmodule'
ERROR: missing input file 'Carthage/build/iOS/SnapKit.framework/Modules/module.modulemap', owner: '//:Carthage/build/iOS/SnapKit.framework/Modules/module.modulemap'
ERROR: /Users/andres/Git/BuildSystemGenerator/SwiftBuildSystemBazel/0/Libraries/SwiftModule1/BUILD.bazel:8:1: //Libraries/SwiftModule1:SwiftModule1: missing input file '//:Carthage/build/iOS/SnapKit.framework/Headers/SnapKit-Swift.h'
ERROR: /Users/andres/Git/BuildSystemGenerator/SwiftBuildSystemBazel/0/Libraries/SwiftModule1/BUILD.bazel:8:1: //Libraries/SwiftModule1:SwiftModule1: missing input file '//:Carthage/build/iOS/SnapKit.framework/Headers/SnapKit.h'
ERROR: /Users/andres/Git/BuildSystemGenerator/SwiftBuildSystemBazel/0/Libraries/SwiftModule1/BUILD.bazel:8:1: //Libraries/SwiftModule1:SwiftModule1: missing input file '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/arm.swiftmodule'
ERROR: /Users/andres/Git/BuildSystemGenerator/SwiftBuildSystemBazel/0/Libraries/SwiftModule1/BUILD.bazel:8:1: //Libraries/SwiftModule1:SwiftModule1: missing input file '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/arm64-apple-ios.swiftmodule'
ERROR: /Users/andres/Git/BuildSystemGenerator/SwiftBuildSystemBazel/0/Libraries/SwiftModule1/BUILD.bazel:8:1: //Libraries/SwiftModule1:SwiftModule1: missing input file '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/arm64.swiftmodule'
ERROR: /Users/andres/Git/BuildSystemGenerator/SwiftBuildSystemBazel/0/Libraries/SwiftModule1/BUILD.bazel:8:1: //Libraries/SwiftModule1:SwiftModule1: missing input file '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/armv7-apple-ios.swiftmodule'
ERROR: /Users/andres/Git/BuildSystemGenerator/SwiftBuildSystemBazel/0/Libraries/SwiftModule1/BUILD.bazel:8:1: //Libraries/SwiftModule1:SwiftModule1: missing input file '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/armv7.swiftmodule'
ERROR: /Users/andres/Git/BuildSystemGenerator/SwiftBuildSystemBazel/0/Libraries/SwiftModule1/BUILD.bazel:8:1: //Libraries/SwiftModule1:SwiftModule1: missing input file '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/i386-apple-ios-simulator.swiftmodule'
ERROR: /Users/andres/Git/BuildSystemGenerator/SwiftBuildSystemBazel/0/Libraries/SwiftModule1/BUILD.bazel:8:1: //Libraries/SwiftModule1:SwiftModule1: missing input file '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/i386.swiftmodule'
ERROR: /Users/andres/Git/BuildSystemGenerator/SwiftBuildSystemBazel/0/Libraries/SwiftModule1/BUILD.bazel:8:1: //Libraries/SwiftModule1:SwiftModule1: missing input file '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/x86_64-apple-ios-simulator.swiftmodule'
ERROR: /Users/andres/Git/BuildSystemGenerator/SwiftBuildSystemBazel/0/Libraries/SwiftModule1/BUILD.bazel:8:1: //Libraries/SwiftModule1:SwiftModule1: missing input file '//:Carthage/build/iOS/SnapKit.framework/Modules/SnapKit.swiftmodule/x86_64.swiftmodule'
ERROR: /Users/andres/Git/BuildSystemGenerator/SwiftBuildSystemBazel/0/Libraries/SwiftModule1/BUILD.bazel:8:1: //Libraries/SwiftModule1:SwiftModule1: missing input file '//:Carthage/build/iOS/SnapKit.framework/Modules/module.modulemap'
ERROR: /Users/andres/Git/BuildSystemGenerator/SwiftBuildSystemBazel/0/Libraries/SwiftModule1/BUILD.bazel:8:1 12 input file(s) do not exist

I managed to find what was causing it. I was running the following command:

bazel test --config ios-debug //...

The rule failing is on the root of the project (next to WORKSPACE) and is not a test rule (it is a apple_dynamic_framework_import rule). I changed the previous command to avoid including the mentioned rule, as follows:

bazel test --config ios-debug //Libraries/...

And that fixed the problem :)