Bazel: //src/test/shell/bazel/android:{aar, android, android_sdk}_integration_test broken at HEAD

[jin]

Same error across all three tests:

ERROR: /usr/local/google/home/jingwen/.cache/bazel/_bazel_jingwen/d1719ddeb069b7c71d85ce79d1b4b9f4/bazel-sandbox/1420182360825110668/execroot/io_bazel/_tmp/5919110799a40c5df8736e66628ccce0/workspace.J7WiwpNX/BUILD:1:1: Extracting classes.jar and libs/*.jar from test.aar failed (Exit 1)
Traceback (most recent call last):
  File "/usr/local/google/home/jingwen/.cache/bazel/_bazel_jingwen/d1719ddeb069b7c71d85ce79d1b4b9f4/bazel-sandbox/1420182360825110668/execroot/io_bazel/_tmp/5919110799a40c5df8736e66628ccce0/root/bc97bb9455f8c5b5c7c7883be291d798/bazel-sandbox/6550599121653112025/execroot/__main__/bazel-out/host/bin/external/bazel_tools/tools/android/aar_embedded_jars_extractor.runfiles/__main__/../bazel_tools/tools/android/aar_embedded_jars_extractor.py", line 28, in <module>
    from third_party.py import gflags
  File "/usr/local/google/home/jingwen/.cache/bazel/_bazel_jingwen/d1719ddeb069b7c71d85ce79d1b4b9f4/bazel-sandbox/1420182360825110668/execroot/io_bazel/_tmp/5919110799a40c5df8736e66628ccce0/root/bc97bb9455f8c5b5c7c7883be291d798/bazel-sandbox/6550599121653112025/execroot/__main__/bazel-out/host/bin/external/bazel_tools/tools/android/aar_embedded_jars_extractor.runfiles/bazel_tools/third_party/py/gflags/__init__.py", line 1, in <module>
    from gflags import *
  File "/usr/local/google/home/jingwen/.cache/bazel/_bazel_jingwen/d1719ddeb069b7c71d85ce79d1b4b9f4/bazel-sandbox/1420182360825110668/execroot/io_bazel/_tmp/5919110799a40c5df8736e66628ccce0/root/bc97bb9455f8c5b5c7c7883be291d798/bazel-sandbox/6550599121653112025/execroot/__main__/bazel-out/host/bin/external/bazel_tools/tools/android/aar_embedded_jars_extractor.runfiles/bazel_tools/third_party/py/gflags/gflags/__init__.py", line 871, in <module>
    _helpers.SPECIAL_FLAGS)
  File "/usr/local/google/home/jingwen/.cache/bazel/_bazel_jingwen/d1719ddeb069b7c71d85ce79d1b4b9f4/bazel-sandbox/1420182360825110668/execroot/io_bazel/_tmp/5919110799a40c5df8736e66628ccce0/root/bc97bb9455f8c5b5c7c7883be291d798/bazel-sandbox/6550599121653112025/execroot/__main__/bazel-out/host/bin/external/bazel_tools/tools/android/aar_embedded_jars_extractor.runfiles/bazel_tools/third_party/py/gflags/gflags/__init__.py", line 560, in DEFINE_string
    DEFINE(parser, name, default, help, flag_values, serializer, **args)
  File "/usr/local/google/home/jingwen/.cache/bazel/_bazel_jingwen/d1719ddeb069b7c71d85ce79d1b4b9f4/bazel-sandbox/1420182360825110668/execroot/io_bazel/_tmp/5919110799a40c5df8736e66628ccce0/root/bc97bb9455f8c5b5c7c7883be291d798/bazel-sandbox/6550599121653112025/execroot/__main__/bazel-out/host/bin/external/bazel_tools/tools/android/aar_embedded_jars_extractor.runfiles/bazel_tools/third_party/py/gflags/gflags/__init__.py", line 392, in DEFINE
    flag_values, module_name)
  File "/usr/local/google/home/jingwen/.cache/bazel/_bazel_jingwen/d1719ddeb069b7c71d85ce79d1b4b9f4/bazel-sandbox/1420182360825110668/execroot/io_bazel/_tmp/5919110799a40c5df8736e66628ccce0/root/bc97bb9455f8c5b5c7c7883be291d798/bazel-sandbox/6550599121653112025/execroot/__main__/bazel-out/host/bin/external/bazel_tools/tools/android/aar_embedded_jars_extractor.runfiles/bazel_tools/third_party/py/gflags/gflags/__init__.py", line 413, in DEFINE_flag
    fv[flag.name] = flag
  File "/usr/local/google/home/jingwen/.cache/bazel/_bazel_jingwen/d1719ddeb069b7c71d85ce79d1b4b9f4/bazel-sandbox/1420182360825110668/execroot/io_bazel/_tmp/5919110799a40c5df8736e66628ccce0/root/bc97bb9455f8c5b5c7c7883be291d798/bazel-sandbox/6550599121653112025/execroot/__main__/bazel-out/host/bin/external/bazel_tools/tools/android/aar_embedded_jars_extractor.runfiles/bazel_tools/third_party/py/gflags/gflags/flagvalues.py", line 442, in __setitem__
    raise exceptions.DuplicateFlagError.from_flag(name, self)
gflags.exceptions.DuplicateFlagError: The flag 'flagfile' is defined twice. First from third_party.py.gflags.gflags, Second from third_party.py.gflags.  Description from first occurrence: Insert flag definitions from the given file into the command line.
Target //:test failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 9.401s, Critical Path: 0.40s
FAILED: Build did NOT complete successfully
-- Test log: -----------------------------------------------------------
------------------------------------------------------------------------
test_nonempty_aar_resources_tree_artifact FAILED: terminated because this command returned a non-zero status:
/usr/local/google/home/jingwen/.cache/bazel/_bazel_jingwen/d1719ddeb069b7c71d85ce79d1b4b9f4/bazel-sandbox/1420182360825110668/execroot/io_bazel/bazel-out/k8-fastbuild/bin/src/test/shell/bazel/android/aar_integration_test.runfiles/io_bazel/src/test/shell/bazel/android/aar_integration_test:78: in call to test_nonempty_aar_resources_tree_artifact
INFO[aar_integration_test 15:13:39.858557916 (+0000)] Cleaning up workspace
FAILED: test_nonempty_aar_resources_tree_artifact

** 0 / 2 tests passed. *********************************************************
** There were errors. **********************************************************

[jin]

cc @ahumesky @dkelmer @aj-michael

[aj-michael]

hmmm I wonder how we did not catch this internally

taking a look now

[aj-michael]

my guess is 8fd0f4a038ad49f77cc2ae0c1e789df7391be467 ?

[aj-michael]

Is this broken on CI?

[jin]

It's fb15f0f.

[aj-michael]

Yes, CI is red. And its not limited to just these tests.

https://ci.bazel.io/job/bazel-tests/1341/#showFailuresLink

.test_empty_tree_artifact_action_inputs_mount_empty_directories
.test_nonempty_aar_resources_tree_artifact
.test_empty_tree_artifact_action_inputs_mount_empty_directories
.test_nonempty_aar_resources_tree_artifact
.test_allow_custom_manifest_name
.test_allow_custom_manifest_name
.test_specifying_android_sdk_flag
.test_specifying_android_sdk_flag
tools/android/aar_embedded_jars_extractor_test.tools/android/aar_embedded_jars_extractor_test
tools/android/aar_embedded_jars_extractor_test.tools/android/aar_embedded_jars_extractor_test
tools/android/aar_native_libs_zip_creator_test.tools/android/aar_native_libs_zip_creator_test
tools/android/aar_native_libs_zip_creator_test.tools/android/aar_native_libs_zip_creator_test
tools/android/aar_resources_extractor_test.tools/android/aar_resources_extractor_test
tools/android/aar_resources_extractor_test.tools/android/aar_resources_extractor_test
tools/android/build_split_manifest_test.tools/android/build_split_manifest_test
tools/android/build_split_manifest_test.tools/android/build_split_manifest_test
tools/android/incremental_install_test.tools/android/incremental_install_test
tools/android/incremental_install_test.tools/android/incremental_install_test
tools/android/merge_manifests_test.tools/android/merge_manifests_test
tools/android/merge_manifests_test.tools/android/merge_manifests_test
tools/android/strip_resources_test.tools/android/strip_resources_test
tools/android/strip_resources_test.tools/android/strip_resources_test
tools/android/stubify_manifest_test.tools/android/stubify_manifest_test
tools/android/stubify_manifest_test.tools/android/stubify_manifest_test

[aj-michael]

The answer to how we did not catch this internally is that we don't automatically propagate external third_party changes internally.

[aj-michael]

Do we think this is just limited to tests?

[aj-michael]

It looks like in #3855, there is a series of commits. The tests pass on the PR as a whole, but we're currently merging the commits one-by-one. So hopefully this will resolve itself?

[akirabaruah]

Thanks for the analysis @aj-michael @jin

3855 ultimately deprecates and removes gflags in favor of abseil's flags package, so it may take longer for that PR to be fully merged (for backwards compatibility concerns).

4212 fixes this immediate bug on my local machine, but please let me know if you continue to see any problems.

[ulfjack]

I still see some failures with the patch, but I suppose they're not the same root cause? In any case, I merged the patch from @akira-baruah, so hopefully this is fixed now.

[aj-michael]

Reopening because Android builds now produce an unacceptable level of console spam of RuntimeWarnings:

INFO: From Filtering AAR native libs by architecture:
/tmp/devoutputbase/bazel-sandbox/8877899873393867950/execroot/io_bazel/bazel-out/host/bin/external/bazel_tools/tools/android/aar_native_libs_zip_creator.runfiles/bazel_tools/third_party/py/gflags/__init__.py:6: RuntimeWarning: Parent module 'third_party.py.gflags.gflags' not found while handling absolute import
  from gflags import *
INFO: From Filtering AAR native libs by architecture:
/tmp/devoutputbase/bazel-sandbox/8282687682378259805/execroot/io_bazel/bazel-out/host/bin/external/bazel_tools/tools/android/aar_native_libs_zip_creator.runfiles/bazel_tools/third_party/py/gflags/__init__.py:6: RuntimeWarning: Parent module 'third_party.py.gflags.gflags' not found while handling absolute import
  from gflags import *
INFO: From Filtering AAR native libs by architecture:
/tmp/devoutputbase/bazel-sandbox/2172885511557734566/execroot/io_bazel/bazel-out/host/bin/external/bazel_tools/tools/android/aar_native_libs_zip_creator.runfiles/bazel_tools/third_party/py/gflags/__init__.py:6: RuntimeWarning: Parent module 'third_party.py.gflags.gflags' not found while handling absolute import
  from gflags import *
INFO: From Filtering AAR native libs by architecture:
/tmp/devoutputbase/bazel-sandbox/8066523634325434932/execroot/io_bazel/bazel-out/host/bin/external/bazel_tools/tools/android/aar_native_libs_zip_creator.runfiles/bazel_tools/third_party/py/gflags/__init__.py:6: RuntimeWarning: Parent module 'third_party.py.gflags.gflags' not found while handling absolute import
  from gflags import *
INFO: From Filtering AAR native libs by architecture:
/tmp/devoutputbase/bazel-sandbox/3834343015066674051/execroot/io_bazel/bazel-out/host/bin/external/bazel_tools/tools/android/aar_native_libs_zip_creator.runfiles/bazel_tools/third_party/py/gflags/__init__.py:6: RuntimeWarning: Parent module 'third_party.py.gflags.gflags' not found while handling absolute import
  from gflags import *

[akirabaruah]

@aj-michael Thanks for pointing out the warnings issue. See #4224 for a proper solution.

[jin]

Warning spam during build/test is fixed on my local machine, closing this issue. Thanks @akira-baruah for the quick fix!