Bazel: Android tool causing Desugar action to fail when building Android tools from source

[nkoroste]

Description of the problem / feature request:

When building Android Tools from the tip of Bazel 3.3.0 branch an error is thrown in Desugar worker log saying: Error parsing command line: Unrecognized option: --persistent_worker

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Build Android tools from source and try to build Android binary target using them

1. Make sure you're on latest 3.3.0 release branch
2. bazel build //src:bazel-dev //tools/android/runtime_deps:android_tools
3. mkdir -p /tmp/bazel_android_tools
4. tar -xzvf bazel-bin/tools/android/runtime_deps/android_tools.tar.gz -C /tmp/bazel_android_tools
5. link the local build of Bazel built in step #2
6. build your app and supply a flag to use the locally built android tools ./bazelw build --override_repository=android_tools=/tmp/bazel_android_tools //apps/myApp:app

Also tried doing the same steps but with at SHA f3d1683fcb3c4a25c4c4b8251d00ddfa66f958ba from here:
https://github.com/bazelbuild/bazel/commit/f3d1683fcb3c4a25c4c4b8251d00ddfa66f958ba#diff-6f653881793ed1c14a8c9d92a49705b1
Also with no luck, I get the same error.

What operating system are you running Bazel on?

Linux/Mac

What's the output of bazel info release?

3.3.0

Have you found anything relevant by searching the web?

• https://github.com/bazelbuild/bazel/issues/11530
• https://github.com/bazelbuild/rules_android/issues/10#issuecomment-620754561

Any other information, logs, or outputs that you want to share?

ERROR: /Users/sachin/PersonalProjects/samples-bazel-android/app/src/main/java/com/xachin/playground/BUILD.bazel:3:19: Desugaring app/src/main/java/com/xachin/playground/playground-app_kt.jar for Android failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at /private/var/tmp/_bazel_sachin/6af8cdc15b57945be280298fb5d977c4/bazel-workers/worker-2-Desugar.log ---8<---8<---
Error parsing command line: Unrecognized option: --persistent_worker
Try --help.
---8<---8<--- End of log ---8<---8<---
Target //app:app failed to build

[nkoroste]

@jin fyi

[jin]

Thanks for the repro @nkoroste - mailed #11620.

[nkoroste]

Getting a different issue after this fix, see:

ERROR: /home/nk/Dev/android/foo/platform/BUILD.bazel:195:19: Desugaring platform/core.jar for Android failed (Exit 1): desugar_java8 failed: error executing command 
  (cd /home/nk/.cache/bazel/_bazel_nk/2007fbffc9ead87bb25ae587ecsdcc95/execroot/__main__ && \
  exec env - \
    PATH=/bin:/usr/bin:/usr/local/bin \
  bazel-out/host/bin/external/bazel_tools/tools/android/desugar_java8 @bazel-out/android-armeabi-v7a-fastbuild/bin/platform/core/_dx/core/core.jar_desugared.jar-0.params)
Execution platform: @local_config_platform//:host
java.lang.IllegalArgumentException: resource com/google/devtools/build/android/desugar/runtime/PrimitiveHashcode.class not found.
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:216)
    at com.google.common.io.Resources.getResource(Resources.java:195)
    at com.google.devtools.build.android.desugar.io.ResourceBasedClassFiles.lambda$getContent$0(ResourceBasedClassFiles.java:32)
    at com.google.devtools.build.android.desugar.io.FileContentProvider.get(FileContentProvider.java:102)
    at com.google.devtools.build.android.desugar.io.FileContentProvider.sink(FileContentProvider.java:143)
    at com.google.devtools.build.android.desugar.Desugar.lambda$copyRuntimeClasses$7(Desugar.java:449)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at com.google.devtools.build.android.desugar.Desugar.copyRuntimeClasses(Desugar.java:448)
    at com.google.devtools.build.android.desugar.Desugar.desugarOneInput(Desugar.java:366)
    at com.google.devtools.build.android.desugar.Desugar.desugar(Desugar.java:240)
    at com.google.devtools.build.android.desugar.Desugar.processRequest(Desugar.java:1008)
    at com.google.devtools.build.android.desugar.Desugar.runPersistentWorker(Desugar.java:968)
    at com.google.devtools.build.android.desugar.Desugar.main(Desugar.java:947)
Target //apps/myApp/foo:app failed to build
INFO: Elapsed time: 39.935s (0.67m), Critical Path: 38.49s (0.64m)
INFO: 230 processes: 3 linux-sandbox, 227 worker 

[Kernald]

I have the same issue when desugaring okhttp with Bazel 3.3 - no problem with Bazel 3.2.

[jin]

@Kernald could you try if https://github.com/bazelbuild/bazel/pull/11620 works for you and if you're running into the same new problem @nkoroste ran into described in that PR?

[Kernald]

I'm gonna try that - I never built Bazel myself so that might take a while.

[jin]

Thanks, you can follow the additional instructions in https://github.com/bazelbuild/bazel/issues/11618#issue-642086076 of this issue to build the desugar tool because it's unbundled from Bazel.

[Kernald]

I tried both with and without the repo override, and in both cases I got the same error that I have with Bazel 3.3.0 - the one just above.

[jin]

Gotcha, so there are really two bugs here.

FYI @dx404

[jin]

I see the issue. None of our tests cover the primitive type desugaring feature required by that missing desugar resource. Will submit a fix.

[nkoroste]

Tested out the latest patch but I now get yet another new error:

ERROR: /home/nk/foo/apk/BUILD.bazel:992:15: Sharding classes for dexing for //foo/apk:Foo failed (Exit 1) mapper failed: error executing command bazel-out/host/bin/external/bazel_tools/src/tools/android/java/com/google/devtools/build/android/ziputils/mapper @bazel-out/k8-fastbuild/bin/foo/apk/_dx/Foo/classes.jar-0.params
Use --sandbox_debug to see verbose messages from the sandbox
Error: Could not find or load main class com.google.devtools.build.android.ziputils.DexMapper
Caused by: java.lang.ClassNotFoundException: com.google.devtools.build.android.ziputils.DexMapper
Target //foo/apk:Foo failed to build

I'm not really sure who triggers DexMapper or how your change might of broke it. When compiling the DexMapper target directly with "_deploy.jar" I do see the class in the expected location.

[nkoroste]

@jin I see that this is now closed, does your fix account for the latest dexMapper issue I reported above?

[jin]

No, it doesn't. Can you file another bug for that with repro steps? The linked commit fixes the worker strategy desugar issue.

[nkoroste]

Were you able to build an Android App after your fix? The repro steps was to use your fix and attempt to build an app which failed with this DexMapper issue for me

[jin]

Yes, I was able to build the Android apps in rules_jvm_external, and the integration tests on CI are passing. I haven't seen that error while trying to fix the Desugar issue.

[katre]

Bazel 3.3.1 RC1 is now available: https://releases.bazel.build/3.3.1/rc1/index.html

Please test this against the errors in this issue.

[nkoroste]

@katre happy to test it out but I'm not really sure how to do it with the link you provided. I need to be able to rebase off of Bazel 3.3.1 RC1 branch/tag as we use a custom fork of Bazel.

[katre]

Every release candidate is a branch on Github, see here: https://github.com/bazelbuild/bazel/tree/release-3.3.1rc1

[nkoroste]

Great I can work with that, thanks!
I just checked https://github.com/bazelbuild/bazel/releases and didn't see it there so I was a bit confused.

[katre]

Right, that's only for finalized releases, this is a candidate (which is why it's a branch, not a tag). It'll end up there once it's fully released.

[nkoroste]

Still getting the DexMapper issue on 3.3.1 :(

[katre]

@jin can you take a look at DexMapper? At this point I think we should aim for a fix in #11687 (3.4.0), rather than another patch to 3.3.

[jin]

The DexMapper issue is not related to this issue. This issue has been fixed in 3.3.1.

[jin]

@nkoroste if you have a way to repro the DexMapper issue, please create a new issue for us to track it.

[nkoroste]

@jin I finally figured out what the issue was - It was related to sandboxing and the --override_repository=android_tools= flag. I was following your instructions from a different PR on how to build using locally built tools:

bazel build //src:bazel-dev //tools/android/runtime_deps:android_tools
mkdir -p /tmp/bazel_android_tools
tar -xzvf bazel-bin/tools/android/runtime_deps/android_tools.tar.gz -C /tmp/bazel_android_tools
./bazelw build --override_repository=android_tools=/tmp/bazel_android_tools //app

which resulted in this:

Error: Could not find or load main class com.google.devtools.build.android.ziputils.DexMapper

When I manually ran the dexmapper subcommand it passed just fine

I can file a seperate bug for this, unless this a user error and there is a more official way to use locally built android_tools?

[jin]

Yes, please file a separate bug as it looks like it's from a different failure mode.

[nkoroste]

done - https://github.com/bazelbuild/bazel/issues/11810