Bazel: //src/test/shell/bazel/android:android_ndk_integration_test doesn't work with remote caching

Created on 20 Feb 2018 · 28Comments · Source: bazelbuild/bazel

INFO: From Executing genrule //src/test/java/com/google/devtools/build/android/desugar:desugar_testdata_from_directory_to_directory:
--
  | /tmp/tmp.aw3wZdZTdO /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/02e53aad341ca65ced1335c03162fe6d/bazel-sandbox/1716082235553702946/execroot/io_bazel
  | [2018-02-20T10:29:55Z] /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/02e53aad341ca65ced1335c03162fe6d/bazel-sandbox/1716082235553702946/execroot/io_bazel
  | ERROR: /var/lib/buildkite-agent/builds/buildkite-ubuntu1604-6z37/bazel/bazel/src/test/shell/bazel/android/BUILD:71:1: Couldn't build file src/test/shell/bazel/android/android_ndk_integration_test/shard_1_of_6/test.log:  failed: Unexpected IO error.: Input is a directory: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/02e53aad341ca65ced1335c03162fe6d/execroot/io_bazel/external/androidndk/ndk/build
  | ERROR: /var/lib/buildkite-agent/builds/buildkite-ubuntu1604-6z37/bazel/bazel/src/test/shell/bazel/android/BUILD:71:1: Couldn't build file src/test/shell/bazel/android/android_ndk_integration_test/shard_4_of_6/test.log:  failed: Unexpected IO error.: Input is a directory: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/02e53aad341ca65ced1335c03162fe6d/execroot/io_bazel/external/androidndk/ndk/build
  | ERROR: /var/lib/buildkite-agent/builds/buildkite-ubuntu1604-6z37/bazel/bazel/src/test/shell/bazel/android/BUILD:71:1: Couldn't build file src/test/shell/bazel/android/android_ndk_integration_test/shard_5_of_6/test.log:  failed: Unexpected IO error.: Input is a directory: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/02e53aad341ca65ced1335c03162fe6d/execroot/io_bazel/external/androidndk/ndk/build
  | ERROR: /var/lib/buildkite-agent/builds/buildkite-ubuntu1604-6z37/bazel/bazel/src/test/shell/bazel/android/BUILD:71:1: Couldn't build file src/test/shell/bazel/android/android_ndk_integration_test/shard_6_of_6/test.log:  failed: Unexpected IO error.: Input is a directory: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/02e53aad341ca65ced1335c03162fe6d/execroot/io_bazel/external/androidndk/ndk/build
  | ERROR: /var/lib/buildkite-agent/builds/buildkite-ubuntu1604-6z37/bazel/bazel/src/test/shell/bazel/android/BUILD:71:1: Couldn't build file src/test/shell/bazel/android/android_ndk_integration_test/shard_2_of_6/test.log:  failed: Unexpected IO error.: Input is a directory: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/02e53aad341ca65ced1335c03162fe6d/execroot/io_bazel/external/androidndk/ndk/build
  | ERROR: /var/lib/buildkite-agent/builds/buildkite-ubuntu1604-6z37/bazel/bazel/src/test/shell/bazel/android/BUILD:71:1: Couldn't build file src/test/shell/bazel/android/android_ndk_integration_test/shard_3_of_6/test.log:  failed: Unexpected IO error.: Input is a directory: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/02e53aad341ca65ced1335c03162fe6d/execroot/io_bazel/external/androidndk/ndk/build

P2 team-Android bug

Source

buchgr

All 28 comments

What is the error here? Now that we're using BuildKite, we should re-enable this as soon as possible.

jin on 23 Feb 2018

Ping, can we disable remote caching just for this test?

jin on 26 Feb 2018

The error is Unexpected IO error.: Input is a directory: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/02e53aad341ca65ced1335c03162fe6d/execroot/io_bazel/external/androidndk/ndk/build

We unfortunately can't disable remote caching just for one test, but we can completely disable remote caching until it's fixed? @philwo ?

buchgr on 27 Feb 2018

perhaps this is a problem with mac case sensitivity? the runfiles of this test should contain: androidndk/BUILD.bazel and well as androidndk/ndk/build/. Perhaps Bazel is trying to read the latter as if it were a BUILD file?

aj-michael on 27 Feb 2018

@aj-michael this looks like it's running on ubuntu buildkite-ubuntu1604-6z37

jin on 28 Feb 2018

@buchgr can you please provide some steps on how we can reproduce this with a remote caching setup? Is there an existing remote cache server that we can use to reproduce this?

jin on 28 Feb 2018

@buchgr I ran this test and it passes with --remote_http_cache=url-to-buildkite-cache on our Ubuntu CI test machine.

Managed to repro:

ERROR: /home/jingwen/bazel/src/test/shell/bazel/android/BUILD:71:1: failed: Unexpect
ed IO error.: Input is a directory: /home/jingwen/.cache/bazel/_bazel_jingwen/bb5fb95
2b6ed390892e90207ad3205d9/execroot/io_bazel/external/androidndk/ndk/build
Target //src/test/shell/bazel/android:android_ndk_integration_test up-to-date:
  bazel-bin/src/test/shell/bazel/android/android_ndk_integration_test
INFO: Elapsed time: 0.606s, Critical Path: 0.30s
FAILED: Build did NOT complete successfully
//src/test/shell/bazel/android:android_ndk_integration_test           NO STATUS

jin on 7 Mar 2018

I renamed the directory from build to _build and the issue persists:

ERROR: /home/jingwen/bazel/src/test/shell/bazel/android/BUILD:71:1:  failed: Unexpected IO error
.: Input is a directory: /home/jingwen/.cache/bazel/_bazel_jingwen/bb5fb952b6ed390892e90207ad320
5d9/execroot/io_bazel/external/androidndk/ndk/_build

Deleted the build directory and the error shows up with another directory:

ERROR: /home/jingwen/bazel/src/test/shell/bazel/android/BUILD:71:1:  failed: Unexp
ected IO error.: Input is a directory: /home/jingwen/.cache/bazel/_bazel_jingwen/b
b5fb952b6ed390892e90207ad3205d9/execroot/io_bazel/external/androidndk/ndk/meta

jin on 7 Mar 2018

@buchgr ping, is this a problem with external repositories? The NDK external repository is a directory of symlinks to a local path.

This test has been disabled for 3 weeks now and I'd like to help debug to get it back up again.

jin on 13 Mar 2018

Perhaps this is due to the fact that the test has an input that is a filegroup that contains a directory:

https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/bazel/rules/android/android_ndk_build_file_template.txt;l=22?q=android_ndk

aj-michael on 13 Mar 2018

@aj-michael Yes, I remember that directory inputs are very very problematic. They cannot be cached, the sandbox may fall over its own feet, Bazel has no clue when stuff inside that directory changes...

Is there any way we could get rid of that directory filegroup? Maybe replace it with a glob?

philwo on 13 Mar 2018

Thanks @aj-michael @philwo, let me try switching it to a glob.

jin on 13 Mar 2018

@philwo - my understanding is that directory inputs are something that is (unfortunately) supported in Bazel for historical reasons. Does remote caching not work with it at all? If so, I suspect there will be other users who are broken.

Also, I wonder why our SDK integration tests do not have this problem. They have a glob with exclude_directories = 0:

https://source.bazel.build/bazel/+/master:tools/android/android_sdk_repository_template.bzl;l=63?q=android_sdk_repository

There are two reasons that we do not use a glob here:

Performance - there are a lot of files, and globbing them can take 20-30seconds
Filenames contain characters that are invalid in the label syntax

Perhaps we can find a solution for 1, but I can't think of a solution for 2

aj-michael on 13 Mar 2018

Filenames contain characters that are invalid in the label syntax

For example, the NDK contains many paths with the string c++, and a few with !.

jin on 13 Mar 2018

@ulfjack for advice. Not sure how to proceed then :(

@buchgr Can we mark individual tests as uncachable?

philwo on 13 Mar 2018

There's the no-cache tag: https://docs.bazel.build/versions/master/remote-caching.html#exclude-specific-targets-from-using-the-remote-cache

However, I tried marking the integration test with this tag and it's still failing with the same error. Maybe it's not doing what I think it's doing?

jin on 13 Mar 2018

I have a partial change to properly support directory inputs, but haven't had time to finish it.

Unfortunately, due to a change by @ola-rozenfeld, the no-cache flag still uploads to the remote cache. @buchgr has a proposal for fixing that, but hasn't had time to work on it.

ulfjack on 14 Mar 2018

It's change https://bazel-review.googlesource.com/c/bazel/+/29872 - it's incomplete because it's not returning the correct metadata and it's not adding the metadata to the MetadataHandler.

That said, even with that change, I think it'll still require a patch to RemoteSpawnCache / RemoteSpawnRunner.

ulfjack on 14 Mar 2018

@buchgr has a proposal for fixing that, but hasn't had time to work on it.
Jakob told me yesterday he is actually working on this right now.

ola-rozenfeld on 14 Mar 2018

👍1

There's a workaround that will let us enable it.Instead of using --experimental_remote_spawn_strategy, we can use --spawn_strategy=remote --genrule_strategy=remote --strategy=Javac=remote --strategy=Closure=remote and then tag the test with no-remote. I just tested it. Works.

buchgr on 14 Mar 2018

@buchgr I think your change accidentally disabled sandboxing on all platforms, as the RemoteSpawnRunner uses the normal LocalSpawnRunner as its fallback.

philwo on 14 Mar 2018

So, what's the path forward here? This test is the only thing we have testing Bazel's Android NDK integration - which is very important to Tensorflow - and we haven't been running it for a few weeks now. Is someone working on supporting directory inputs to the remote cache? Or is someone working on a way to let us disable the remote cache for this test?

aj-michael on 29 Mar 2018

Ping. Is it possible to make no-remote or no-cache work with --experimental_remote_spawn_strategy?

jin on 12 Jun 2018

As long as you're okay with no sandboxing, it should work with the flags
that Jakob listed.

On Mon, Jun 11, 2018 at 7:17 PM Jingwen notifications@github.com wrote:

Ping. Is it possible to make no-remote work with
--experimental_remote_spawn_strategy?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/4663#issuecomment-396416283,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AYoKuJ45JgujujBcUNp7Xl2e_5v3HBLwks5t7vqPgaJpZM4SLwJn
.

ola-rozenfeld on 12 Jun 2018

@ola-rozenfeld according to @philwo, this combination wouldn't work on our CI because it disables sandboxing across all platforms (all tests?). My intention is to only exclude //src/test/shell/bazel/android:android_ndk_integration_test from being cached remotely, even if it requires sandboxing to be disabled.

jin on 12 Jun 2018

@jin yep I am working on that.

buchgr on 22 Jun 2018

update: it's been enabled for bazel postsubmit pipeline where we don't use remote caching.

buchgr on 22 Jun 2018

❤1

This seems to break in the downstream pipeline, e.g.
https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/774#966cc423-8bd8-40e3-b632-d94ac0d358b0