Bazel: Windows, testing: make all tests work on Windows

[laszlocsomor]

As of today, there are 282 tests that Bazel doesn't run on Windows.

EDIT(2017-12-13 15:52 +0100): we need to consider tests that check the current platform too, e.g. in Java do things like OS.getCurrent() == OS.WINDOWS.

C:\work\bazel>bazel --output_user_root=c:\tmp2 query "tests(//...)-tests(//src:all_windows_tests)"
//tools/test:LcovMergerTest
//tools/jdk:proguard_whitelister_test
//tools/build_defs/pkg:path_test
//tools/build_defs/pkg:make_rpm_test
//tools/build_defs/pkg:build_test
//tools/build_defs/pkg:archive_test
//tools/build_defs/docker:rewrite_json_test
//tools/build_defs/docker:create_image_config_test
//tools/android:stubify_manifest_test
//tools/android:strip_resources_test
//tools/android:resource_extractor_test
//tools/android:merge_manifests_test
//tools/android:junction_test
//tools/android:incremental_install_test
//tools/android:build_split_manifest_test
//tools/android:build_incremental_dexmanifest_test
//tools/android:aar_resources_extractor_test
//tools/android:aar_native_libs_zip_creator_test
//tools/android:aar_embedded_jars_extractor_test
//third_party/protobuf/3.4.0:win32_test
//third_party/protobuf/3.4.0:py_wire_format_test
//third_party/protobuf/3.4.0:py_unknown_fields_test
//third_party/protobuf/3.4.0:py_text_format_test
//third_party/protobuf/3.4.0:py_text_encoding_test
//third_party/protobuf/3.4.0:py_symbol_database_test
//third_party/protobuf/3.4.0:py_service_reflection_test
//third_party/protobuf/3.4.0:py_reflection_test
//third_party/protobuf/3.4.0:py_proto_builder_test
//third_party/protobuf/3.4.0:py_message_test
//third_party/protobuf/3.4.0:py_message_factory_test
//third_party/protobuf/3.4.0:py_json_format_test
//third_party/protobuf/3.4.0:py_generator_test
//third_party/protobuf/3.4.0:py_descriptor_test
//third_party/protobuf/3.4.0:py_descriptor_pool_test
//third_party/protobuf/3.4.0:py_descriptor_database_test
//third_party/protobuf/3.4.0:protobuf_test
//third_party/ijar/test:zip_test
//third_party/ijar/test:ijar_test
//third_party/ijar/test:IjarTests
//src/tools/skylark/javatests/com/google/devtools/skylark/skylint:SkylintTests
//src/tools/singlejar:zlib_interface_test
//src/tools/singlejar:zip_headers_test
//src/tools/singlejar:zip64_test
//src/tools/singlejar:transient_bytes_test
//src/tools/singlejar:token_stream_test
//src/tools/singlejar:output_jar_simple_test
//src/tools/singlejar:output_jar_bash_test
//src/tools/singlejar:options_test
//src/tools/singlejar:input_jar_scan_ziptool_test
//src/tools/singlejar:input_jar_scan_jartool_test
//src/tools/singlejar:input_jar_preambled_test
//src/tools/singlejar:input_jar_empty_jar_test
//src/tools/singlejar:input_jar_bad_jar_test
//src/tools/singlejar:combiners_test
//src/test/skylark/skylint:skylint_test
//src/test/skylark:skylark_test_testdata/string_test_characters_sky
//src/test/skylark:skylark_test_testdata/string_splitlines_sky
//src/test/skylark:skylark_test_testdata/string_split_sky
//src/test/skylark:skylark_test_testdata/string_partition_sky
//src/test/skylark:skylark_test_testdata/string_format_sky
//src/test/skylark:skylark_test_testdata/min_max_sky
//src/test/skylark:skylark_test_testdata/list_slices_sky
//src/test/skylark:skylark_test_testdata/int_sky
//src/test/skylark:skylark_test_testdata/int_constructor_sky
//src/test/skylark:skylark_test_testdata/equality_sky
//src/test/skylark:skylark_test_testdata/and_or_not_sky
//src/test/skylark:skylark_test_testdata/all_any_sky
//src/test/skylark:question_regexp_1
//src/test/skylark:question_regexp
//src/test/skylark:question_content
//src/test/skylark:assert_tone_below_1
//src/test/shell/integration:ui_test
//src/test/shell/integration:test_test
//src/test/shell/integration:stub_finds_runfiles_test
//src/test/shell/integration:startup_options_test
//src/test/shell/integration:stamping_test
//src/test/shell/integration:skylark_flag_test
//src/test/shell/integration:runfiles_test
//src/test/shell/integration:run_test
//src/test/shell/integration:rc_options_test
//src/test/shell/integration:python_test
//src/test/shell/integration:progress_reporting_test
//src/test/shell/integration:process_wrapper_test
//src/test/shell/integration:outputs_test
//src/test/shell/integration:output_filter_test
//src/test/shell/integration:logging_test
//src/test/shell/integration:loading_phase_tests
//src/test/shell/integration:linux_sandbox_test
//src/test/shell/integration:linux_sandbox_network_test
//src/test/shell/integration:java_integration_test
//src/test/shell/integration:incompatible_changes_conflict_test
//src/test/shell/integration:force_delete_output_test
//src/test/shell/integration:experimental_ui_test
//src/test/shell/integration:execution_phase_tests
//src/test/shell/integration:discard_graph_edges_test
//src/test/shell/integration:discard_analysis_cache_test
//src/test/shell/integration:cpp_test
//src/test/shell/integration:configured_query_test
//src/test/shell/integration:client_sigint_test
//src/test/shell/integration:build_event_stream_test
//src/test/shell/integration:bazel_worker_test
//src/test/shell/integration:bazel_testjobs_test
//src/test/shell/integration:bazel_sandboxed_worker_test
//src/test/shell/integration:bazel_query_test
//src/test/shell/integration:bazel_javabase_test
//src/test/shell/integration:bazel_command_log_test
//src/test/shell/integration:action_env_test
//src/test/shell/bazel/apple:bazel_objc_test
//src/test/shell/bazel/apple:bazel_apple_test
//src/test/shell/bazel/android:proguard_integration_test
//src/test/shell/bazel/android:desugarer_integration_test
//src/test/shell/bazel/android:android_sdk_integration_test
//src/test/shell/bazel/android:android_ndk_integration_test
//src/test/shell/bazel/android:android_integration_test
//src/test/shell/bazel/android:aidl_integration_test
//src/test/shell/bazel/android:aar_integration_test
//src/test/shell/bazel:workspace_test
//src/test/shell/bazel:toolchain_test
//src/test/shell/bazel:srcs_test
//src/test/shell/bazel:skylark_repository_test
//src/test/shell/bazel:skylark_git_repository_test
//src/test/shell/bazel:runfiles_test
//src/test/shell/bazel:rule_test_test
//src/test/shell/bazel:remote_execution_test
//src/test/shell/bazel:remote_execution_sandboxing_test
//src/test/shell/bazel:remote_execution_rest_test
//src/test/shell/bazel:persistent_test_runner_test
//src/test/shell/bazel:maven_test
//src/test/shell/bazel:maven_skylark_test
//src/test/shell/bazel:location_test
//src/test/shell/bazel:local_repository_test_jdk8
//src/test/shell/bazel:local_repository_test
//src/test/shell/bazel:local_action_cache_test
//src/test/shell/bazel:java_launcher_test
//src/test/shell/bazel:help_test
//src/test/shell/bazel:git_repository_test
//src/test/shell/bazel:external_skylark_load_test
//src/test/shell/bazel:external_integration_test
//src/test/shell/bazel:external_correctness_test
//src/test/shell/bazel:execroot_test
//src/test/shell/bazel:empty_package_test
//src/test/shell/bazel:embedded_tools_deps_test
//src/test/shell/bazel:cross_repository_test
//src/test/shell/bazel:cpp_darwin_integration_test
//src/test/shell/bazel:client_test
//src/test/shell/bazel:cc_static_binary_test
//src/test/shell/bazel:cc_integration_test
//src/test/shell/bazel:build_files_test
//src/test/shell/bazel:bound_targets_test
//src/test/shell/bazel:bazel_workspace_status_test
//src/test/shell/bazel:bazel_with_jdk_test
//src/test/shell/bazel:bazel_tools_test
//src/test/shell/bazel:bazel_toolchain_test
//src/test/shell/bazel:bazel_test_test
//src/test/shell/bazel:bazel_sandboxing_test
//src/test/shell/bazel:bazel_sandboxing_cpp_test
//src/test/shell/bazel:bazel_rules_test
//src/test/shell/bazel:bazel_repository_cache_test
//src/test/shell/bazel:bazel_random_characters_test
//src/test/shell/bazel:bazel_localtest_test
//src/test/shell/bazel:bazel_java_test
//src/test/shell/bazel:bazel_experimental_ui_test
//src/test/shell/bazel:bazel_execute_testlog
//src/test/shell/bazel:bazel_example_test
//src/test/shell/bazel:bazel_docgen_test
//src/test/shell/bazel:bazel_determinism_test
//src/test/shell/bazel:bazel_coverage_test
//src/test/shell/bazel:bazel_build_event_stream_test
//src/test/shell:shell_utils_test
//src/test/shell:bashunit_test
//src/test/java/com/google/devtools/build/lib/rules/objc:ObjcRulesTests
//src/test/java/com/google/devtools/build/lib/rules/cpp:cpp-rules-tests
//src/test/java/com/google/devtools/build/lib/rules/android:AndroidInstrumentationTestTest
//src/test/java/com/google/devtools/build/lib/profiler/memory:AllocationTrackerTest
//src/test/java/com/google/devtools/build/lib/profiler/callcounts:CallcountsTest
//src/test/java/com/google/devtools/build/lib/bazel/repository/downloader:DownloaderTestSuite
//src/test/java/com/google/devtools/build/lib/bazel/repository:RepositoryTests
//src/test/java/com/google/devtools/build/lib:standalone-tests
//src/test/java/com/google/devtools/build/lib:shell_test
//src/test/java/com/google/devtools/build/lib:server_test
//src/test/java/com/google/devtools/build/lib:sandbox-tests
//src/test/java/com/google/devtools/build/lib:foundations_test
//src/test/java/com/google/devtools/build/lib:analysis_test
//src/test/java/com/google/devtools/build/android/desugar/runtime:ThrowableExtensionTestWithReuseDesugaringStrategy
//src/test/java/com/google/devtools/build/android/desugar/runtime:ThrowableExtensionTestWithNullDesugaringStrategy
//src/test/java/com/google/devtools/build/android/desugar/runtime:ThrowableExtensionTestWithMimicDesugaringStrategy
//src/test/java/com/google/devtools/build/android/desugar:testdata_sanity_test
//src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_twice_jar_test
//src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_java8_jar_test
//src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_jar_test
//src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_for_try_with_resources_test_twice
//src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_for_try_with_resources_test
//src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_for_disabling_try_with_resources_with_large_minsdkversion_test
//src/test/java/com/google/devtools/build/android/desugar:testdata_desugared_core_library_jar_test
//src/test/java/com/google/devtools/build/android/desugar:stateless_lambda_has_no_factory_method_test
//src/test/java/com/google/devtools/build/android/desugar:simple_method_reference_has_no_factory_method_test
//src/test/java/com/google/devtools/build/android/desugar:no_stubs_for_lambdas_in_implemented_interface
//src/test/java/com/google/devtools/build/android/desugar:jacoco_legacy_default_method_regression_test
//src/test/java/com/google/devtools/build/android/desugar:interface_has_method_bodies_removed
//src/test/java/com/google/devtools/build/android/desugar:initializer_of_functional_interface_should_not_execute
//src/test/java/com/google/devtools/build/android/desugar:inherited_abstract_method_gets_no_default_method_stub
//src/test/java/com/google/devtools/build/android/desugar:desugar_testdata_by_disabling_lambda_desugaring_test
//src/test/java/com/google/devtools/build/android/desugar:desugar_idempotency_test
//src/test/java/com/google/devtools/build/android/desugar:capture_lambda_has_no_factory_method_test
//src/test/java/com/google/devtools/build/android/desugar:TryWithResourcesRewriterUnitTestWithReuseStrategy
//src/test/java/com/google/devtools/build/android/desugar:TryWithResourcesRewriterUnitTestWithNullStrategy
//src/test/java/com/google/devtools/build/android/desugar:TryWithResourcesRewriterUnitTestWithMimicStrategy
//src/test/java/com/google/devtools/build/android/desugar:StackMapBugTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarTryWithResourcesFunctionalTestWithReuseStrategyTwice
//src/test/java/com/google/devtools/build/android/desugar:DesugarTryWithResourcesFunctionalTestWithReuseStrategy
//src/test/java/com/google/devtools/build/android/desugar:DesugarTryWithResourcesFunctionalTestWithNullStrategyTwice
//src/test/java/com/google/devtools/build/android/desugar:DesugarTryWithResourcesFunctionalTestWithNullStrategy
//src/test/java/com/google/devtools/build/android/desugar:DesugarTryWithResourcesFunctionalTestWithMimicStrategyTwice
//src/test/java/com/google/devtools/build/android/desugar:DesugarTryWithResourcesFunctionalTestWithMimicStrategy
//src/test/java/com/google/devtools/build/android/desugar:DesugarObjectsRequireNonNullTestForAndroidLintMode
//src/test/java/com/google/devtools/build/android/desugar:DesugarObjectsRequireNonNullTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarMainClassTestLambdaDirectoryIncorrectlySet
//src/test/java/com/google/devtools/build/android/desugar:DesugarMainClassTestLambdaDirectoryCorrectlySet
//src/test/java/com/google/devtools/build/android/desugar:DesugarLongCompareTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarLikeAndroidStudioFunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarJava8LikeAndroidStudioFunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarJava8FunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarIdempotencyFunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarFunctionalTestWithMultipleInputs
//src/test/java/com/google/devtools/build/android/desugar:DesugarFunctionalTestWithClasspathDirectory
//src/test/java/com/google/devtools/build/android/desugar:DesugarFunctionalTestFromDirectoryToJar
//src/test/java/com/google/devtools/build/android/desugar:DesugarFunctionalTestFromDirectoryToDirectory
//src/test/java/com/google/devtools/build/android/desugar:DesugarFunctionalTestForSyntheticMethodsWithLambdaNames
//src/test/java/com/google/devtools/build/android/desugar:DesugarFunctionalTestForConstantArgumentsInLambdas
//src/test/java/com/google/devtools/build/android/desugar:DesugarFunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarDefaultMethodsIdempotencyFunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarDefaultMethodsFunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DesugarCoreLibraryFunctionalTest
//src/test/java/com/google/devtools/build/android/desugar:DefaultMethodClassFixerTest
//src/test/java/com/google/devtools/build/android/desugar:BytecodeTypeInferenceTest
//src/test/docker:test_cc_configure-ubuntu-16.04-opt
//src/test/docker:test_cc_configure-ubuntu-16.04-fastbuild
//src/test/docker:test_cc_configure-ubuntu-16.04-dbg
//src/test/docker:test_cc_configure-ubuntu-15.04-opt
//src/test/docker:test_cc_configure-ubuntu-15.04-fastbuild
//src/test/docker:test_cc_configure-ubuntu-15.04-dbg
//src/test/docker:test_cc_configure-fedora23-opt
//src/test/docker:test_cc_configure-fedora23-fastbuild
//src/test/docker:test_cc_configure-fedora23-dbg
//src/test/docker:test_cc_configure-debian-stretch-opt
//src/test/docker:test_cc_configure-debian-stretch-fastbuild
//src/test/docker:test_cc_configure-debian-stretch-dbg
//src/test/docker:test_cc_configure-centos6.7-opt
//src/test/docker:test_cc_configure-centos6.7-fastbuild
//src/test/docker:test_cc_configure-centos6.7-dbg
//src/java_tools/singlejar/javatests/com/google/devtools/build/zip:ZipTests
//src/java_tools/singlejar/javatests/com/google/devtools/build/singlejar:SingleJarTests
//src/java_tools/junitrunner/javatests/com/google/testing/junit/runner:utf8_test_log_test
//src/java_tools/junitrunner/javatests/com/google/testing/junit/runner:junit4_testbridge_integration_test
//src/java_tools/junitrunner/javatests/com/google/testing/junit/runner:deploy_jar_integration_test
//src/java_tools/junitrunner/javatests/com/google/testing/junit/runner:antxmlresultwriter_integration_test
//src/java_tools/junitrunner/javatests/com/google/testing/junit/runner:AllTests
//src/java_tools/junitrunner/javatests/com/google/testing/junit/junit4/runner:RegExTestCaseFilterTest
//src/java_tools/buildjar/javatests/com/google/devtools/build/java/turbine/javac:TreePrunerTest
//src/java_tools/buildjar/javatests/com/google/devtools/build/java/turbine/javac:ProcessorClasspathTest_bootclasspath
//src/java_tools/buildjar/javatests/com/google/devtools/build/java/turbine/javac:JavacTurbineTest
//src/java_tools/buildjar/javatests/com/google/devtools/build/java/bazel:BazelJavaCompilerTest
//src/java_tools/buildjar/javatests/com/google/devtools/build/buildjar:VanillaJavaBuilderTest
//scripts/release:relnotes_test
//scripts/release:release_test
//scripts:bash_completion_test
//examples/shell:test
//examples/py_native:test
//examples/py_native:fail
//examples/java-skylark/src/test/java/com/example/myproject:pass
//examples/java-skylark/src/test/java/com/example/myproject:fail
//examples/java-native/src/test/java/com/example/myproject:sleep3
//examples/java-native/src/test/java/com/example/myproject:sleep2
//examples/java-native/src/test/java/com/example/myproject:sleep1
//examples/java-native/src/test/java/com/example/myproject:sleep0
//examples/java-native/src/test/java/com/example/myproject:resource-fail
//examples/java-native/src/test/java/com/example/myproject:hello
//examples/java-native/src/test/java/com/example/myproject:fail
//examples/java-native/src/test/java/com/example/myproject:custom_with_test_class
//examples/java-native/src/test/java/com/example/myproject:custom
//examples/cpp:hello-success_test
//examples/cpp:hello-fail_test

We need to make them all run on Windows and turn them on on CI, in order to have full test coverage on Windows.

[laszlocsomor]

I'm tempted to make this bug P1.
Since we're about to plan the work for next 3 months, I'm marking this bug P1 and P2 at the same time, indicating that I'm not decided on priority yet, and making this bug easily discoverable.

[davido]

I'm, the most concerned about disabled buildjar tests:

//src/java_tools/buildjar/javatests/com/google/devtools/build/java/turbine/javac:TreePrunerTest
//src/java_tools/buildjar/javatests/com/google/devtools/build/java/turbine/javac:ProcessorClasspathTest_bootclasspath
//src/java_tools/buildjar/javatests/com/google/devtools/build/java/turbine/javac:JavacTurbineTest
//src/java_tools/buildjar/javatests/com/google/devtools/build/java/bazel:BazelJavaCompilerTest
//src/java_tools/buildjar/javatests/com/google/devtools/build/buildjar:VanillaJavaBuilderTest

TBH with you: I stopped working on gerrit build on Windows, untill these tests are up and running. I was not amused to find out, that transitive classpath calculation in buildjar was inherently broken on Windows, and that the existing test, that would probably catch this was disabled (and is still broken) on Windows.

[davido]

Strictly speaking, removing the warning from Bazel FAQ, that Windows support is bleeding edge, experimental and may produce unexpected results, was a mistake and this CL should be reverted. Nothing document this better as this issue.

[laszlocsomor]

David, I'm not sure what reaction you expect.

[pcj]

I'm seeing a problem that would likely be addressed by fixing these two tests:

//src/java_tools/singlejar/javatests/com/google/devtools/build/zip:ZipTests
//src/java_tools/singlejar/javatests/com/google/devtools/build/singlejar:SingleJarTests

ERROR: C:/src/jenova/test/baseline/BUILD.bazel:9:1: Building deploy jar test/baseline/ATE-369_xlsx_deploy.jar failed (Exit 1)
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 79
        at com.google.devtools.build.zip.ZipUtil.get16(ZipUtil.java:87)
        at com.google.devtools.build.zip.ZipUtil.getUnsignedShort(ZipUtil.java:119)
        at com.google.devtools.build.zip.ZipReader.findEndOfCentralDirectoryRecord(ZipReader.java:277)
        at com.google.devtools.build.zip.ZipReader.readCentralDirectory(ZipReader.java:194)
        at com.google.devtools.build.zip.ZipReader.<init>(ZipReader.java:94)
        at com.google.devtools.build.zip.ZipReader.<init>(ZipReader.java:74)
        at com.google.devtools.build.zip.ZipReader.<init>(ZipReader.java:62)
        at com.google.devtools.build.singlejar.ZipCombiner.addZip(ZipCombiner.java:431)
        at com.google.devtools.build.singlejar.SingleJar.run(SingleJar.java:240)
        at com.google.devtools.build.singlejar.SingleJar.singleRun(SingleJar.java:394)
        at com.google.devtools.build.singlejar.SingleJar.main(SingleJar.java:413)
INFO: Elapsed time: 2.851s, Critical Path: 2.17s

Seems to occur when building multiple java_test jars concurrently where there are shared deps or resources (not sure which one) between the multiple targets.

[laszlocsomor]

Update: I'm still (slowly) working on this issue and P1 is appropriate. Currently I have a PR (https://github.com/bazelbuild/bazel/pull/5141) that makes one shell test use the new Bash runfiles library, but I failed to merge it because, IIRC, some Google-internal tests broke and I haven't yet figured out how to fix them.

[laszlocsomor]

Current status (at https://github.com/bazelbuild/bazel/commit/20637b7870a09a35b716f69a6d8ce5b4c5fe3722):

• bazel query "tests(//src/...)" | wc -l --> 410
• bazel query "tests(//src/...)-tests(//src:all_windows_tests)" | wc -l --> 259

Currently only 63% of all tests under //src/... run on Windows.

In comparison, 3 months ago (at 47b2bf7e906e6b4b4dd02af7aa77a13aeaf081f6):

• bazel query "tests(//src/...)" | wc -l --> 385
• bazel query "tests(//src/...)-tests(//src:all_windows_tests)" | wc -l --> 245

That was also 63%.

Though we fixed 14 tests in 3 months but that didn't increase coverage by any percentage points.

[brandjon]

Laszlo, from a very quick glance it seems like converting shell tests to windows includes copy/pasting some header boilerplate on each .sh file. Can this boilerplate really not be factored into a common file? What's the plan for if the boilerplate needs to be changed in the future?

More to the point, I want to add a new shell test to src/test/shell/integration. Should I just cargo-cult a header from a converted file?

[laszlocsomor]

Jon, indeed there's a copy-pasted boilerplate.

The reason this can't be factored to a common file is, there's no common preamble to sh_* rules where we could source that file. When you "bazel build" a sh_binary, Bazel merely creates a symlink to whatever file is in the rule's srcs, be it a shell script or an arbitrary binary. Running the output from bazel-bin runs the file directly, without running it through a launcher like java_binary or py_binary would do. Users may already be depending on this, i.e. the ability to stick an arbitrary binary in sh_binary.srcs and run it without any wrappers, and so without knowing more we should not change this behavior.

[brandjon]

That part sounds reasonable, but why can't we factor the boilerplate preamble into a common utility file and explicitly source that file wherever it's needed?

[laszlocsomor]

Because we don't know the path of that file at runtime.

[laszlocsomor]

Remember we have no runfiles tree on Windows, so we must look up runfiles from a manifest file, but we must first find that manifest file (either from an envvar, or from next to the binary). This is what the boilerplate code actually does.

[brandjon]

I see. That's unfortunate...