Yes, I can handle it.

katre on 10 Sep 2018

Looks like last night's downstream projects run is fairly solid: the majority of the errors are build machine configuration related.

Using ac880418885061d1039ad6b3d8c28949782e02d6 as the baseline.

katre on 1 Oct 2018

Pushed release 0.19.0rc1 - baseline ac880418885061d1039ad6b3d8c28949782e02d6

katre on 1 Oct 2018

Release announcement sent, downstream tests starting.

katre on 1 Oct 2018

Declaring #6283 a release blocker for 0.19.0.

katre on 2 Oct 2018

6283 is fixed by 20bfdc67dc1fc32ffebbda7088ba49ee17e3e182, will cherrypick that into rc2 today.

katre on 3 Oct 2018

Created rc2:

scripts/release/release.sh create --force_rc=2 0.19.0 ac880418885061d1039ad6b3d8c28949782e02d6 20bfdc67dc1fc32ffebbda7088ba49ee17e3e182

katre on 3 Oct 2018

The "No JDK" platform is failing all tests for rc2, creating RC3 cherrypicking in 9bc3b20053a8b99bf2c4a31323a7f96fabb9f1ec to try and handle this.

katre on 3 Oct 2018

Command:

scripts/release/release.sh create --force_rc=3 0.19.0 ac880418885061d1039ad6b3d8c28949782e02d6 9bc3b20053a8b99bf2c4a31323a7f96fabb9f1ec 20bfdc67dc1fc32ffebbda7088ba49ee17e3e182

katre on 3 Oct 2018

No, everything on the "no JDK" host is still failing:

ERROR: /var/lib/buildkite-agent/builds/buildkite-worker-ubuntu1804-nojava-9ssq-1/bazel/bazel-bazel/src/main/java/com/google/devtools/build/lib/BUILD:1569:1: every rule of type java_library implicitly depends upon the target '@openjdk_linux_archive//:runtime', but this target could not be found because of: no such package '@openjdk_linux_archive//': The repository could not be resolved

katre on 3 Oct 2018

Also need to cherrypick 54c2572a8cabaf2b29e58abe9f04327314caa6a0 (this will fix https://github.com/bazelbuild/continuous-integration/issues/341)

katre on 4 Oct 2018

RC4 is out (and has been since Thursday, sorry): https://releases.bazel.build/0.19.0/rc4/index.html

katre on 8 Oct 2018

rc1-4 pushed to chocolatey

petemounce on 8 Oct 2018

Last RC has all green tests: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/491

katre on 9 Oct 2018

🎉2

Hi @katre, please cherry pick the fix of this issue: https://github.com/bazelbuild/bazel/issues/6292
Looks like it blocks the users.
Thank you!

irengrig on 11 Oct 2018

Is #6292 a regression since 0.18? If not, the fix should not be cherrypicked.

dslomov on 11 Oct 2018

Given that 0.18 isn't released, can anything be a regression since 0.18?

According to #6292, the cause is 4e42e1767d1752d6e3fde78a6b184c2231b12ebb, which went in on July 9.

The fix (914b4ce14624171a97ff8b41f9202058f10d15b2) looks low-risk.

katre on 11 Oct 2018

So it is in 0.17 and not cherrypicked into 0.18.
Our policy is that we do not cherry-pick non-regressions, so let's not do it.

dslomov on 11 Oct 2018

Ok, could you hint me what better comment on the issue in response to the user's request..

irengrig on 11 Oct 2018

I'm getting this suspicious warning when building gerrit:

INFO: From SkylarkAction external/bazel_tools/tools/jdk/platformclasspath.jar [for host]:
warning: could not find a JDK 8 bootclasspath in external/embedded_jdk, falling back to --release

I don't see this warning when building from Bazel@HEAD (72ef398c3b1a701316367a1790215a52d6babd29).

Used Bazel version:

$ bazel version
Build label: 0.19.0rc4

davido on 14 Oct 2018

I'm not sure what the process is, so just to be sure I'll say that I'll perform an incompatible change flip (#6380) in 0.19. I already pinged 0.18 about this. Adding here as well as FYI.

hlopko on 15 Oct 2018

Please cherry-pick 914b4ce to fix https://github.com/bazelbuild/bazel/issues/6292, it is a regression since 0.17.X, not in 0.16.0

meteorcloudy on 15 Oct 2018

So in my opinion, #6292 is a regression from 0.16.0 to 0.17.0, but only got discovered after releasing 0.17.0, I believe it is qualified to be cherry-picked into 0.19.0.

meteorcloudy on 15 Oct 2018

Who is blocked by #6292? Why we need it chery-picked into 0.19 (as opposed to waiting for 0.20?)

dslomov on 15 Oct 2018

Do we ask that with regressions?

hlopko on 15 Oct 2018

We do. But also, this is not a regression - it has been in two releases already.

dslomov on 15 Oct 2018

Ok, so tell me what is the state of #6292 @meteorcloudy ? If C++ builds are fully broken, we should cherry-pick that in all releases that has the issue (at least into 0.18 as well)

dslomov on 15 Oct 2018

@dslomov #6292 is blocking an internal team who's using Bazel on Windows.
See https://github.com/bazelbuild/bazel/issues/6292#issuecomment-429899643 (This I believe is a regression)

Also, it blocks enabling native singlejar on Windows and the envoy Windows build. (But those are not regressions).

meteorcloudy on 16 Oct 2018

"blocking an internal team" shouldn't be relevant.
Based on the comments, it seems like an important regression, so we should have a patch release.

laurentlb on 16 Oct 2018

Yes, I think we should have a patch release for 0.18, but also it means this has to be cherry-picked into 0.19.0

meteorcloudy on 16 Oct 2018

Please keep this thread updated on the final decision about the patch release to 0.18.0.

At this point I am not aware of further cherrypicks needed to 0.19.0.

katre on 16 Oct 2018

Please cherry-pick the fix of https://github.com/bazelbuild/bazel/issues/6292 in 0.19 (I requested a patch release for 0.18).

laurentlb on 16 Oct 2018

Created rc5 with:

$ scripts/release/release.sh create --force_rc=5 0.19.0 ac880418885061d1039ad6b3d8c28949782e02d6 9bc3b20053a8b99bf2c4a31323a7f96fabb9f1ec 54c2572a8cabaf2b29e58abe9f04327314caa6a0 20bfdc67dc1fc32ffebbda7088ba49ee17e3e182 914b4ce14624171a97ff8b41f9202058f10d15b2

katre on 16 Oct 2018

RC5 is available for testing.

Waiting for the results of https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/507

katre on 16 Oct 2018

Please cherry-pick https://github.com/bazelbuild/bazel/commit/83d406b7da32d1b1f6dd02eae2fe98582a4556fd into 0.19.
It fixes a regression introduced in 0.18 (by https://github.com/bazelbuild/bazel/commit/e6db6552716aa50a52a247ac8f9e5c7826d0f0fd).

laszlocsomor on 17 Oct 2018

Created rc6 with:

scripts/release/release.sh create --force_rc=6 0.19.0 ac880418885061d1039ad6b3d8c28949782e02d6 9bc3b20053a8b99bf2c4a31323a7f96fabb9f1ec 54c2572a8cabaf2b29e58abe9f04327314caa6a0 20bfdc67dc1fc32ffebbda7088ba49ee17e3e182 914b4ce14624171a97ff8b41f9202058f10d15b2 83d406b7da32d1b1f6dd02eae2fe98582a4556fd

katre on 17 Oct 2018

RC6 is now available for testing: https://releases.bazel.build/0.19.0/rc6/index.html

katre on 17 Oct 2018

RC6 passes all tests except rules_docker on GCE: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/509

katre on 17 Oct 2018

👍1

rules_docker has the same failure with last night's nightly

katre on 17 Oct 2018

I'm still seeing this warning:

warning: could not find a JDK 8 bootclasspath in external/embedded_jdk, falling back to --release

This seems to be a regression compared to 0.18.0. I filed this issue: [1].

[1] https://github.com/bazelbuild/bazel/issues/6433

davido on 18 Oct 2018

cc @iirina

hlopko on 18 Oct 2018

6433 is not a release blocker.

Given the length of time since RC6 with no new bugs, I am planning to release 0.19.0 on tomorrow (Oct 23) unless anything comes up.

katre on 22 Oct 2018

Please cherry-pick e025726006236520f7e91e196b9e7f139e0af5f4 for #6456.

cushon on 23 Oct 2018

Is that the same as #6467? Or is that a separate update, and does it also need to be in the release?
Also, if you can submit the fix for #6433 today, I'll go ahead and add that as well.

katre on 23 Oct 2018

yep #6467 and #6456 are the same.

(#6433 is distinct, I'm still trying to get the fix for that submitted)

cushon on 23 Oct 2018

Creating RC7 to add e025726006236520f7e91e196b9e7f139e0af5f4 and 5080238f16e8ba13d455622afbc79d2ce6ab9966 as cherrypicks:

scripts/release/release.sh create --force_rc=7 0.19.0 ac880418885061d1039ad6b3d8c28949782e02d6 9bc3b20053a8b99bf2c4a31323a7f96fabb9f1ec 54c2572a8cabaf2b29e58abe9f04327314caa6a0 20bfdc67dc1fc32ffebbda7088ba49ee17e3e182 914b4ce14624171a97ff8b41f9202058f10d15b2 83d406b7da32d1b1f6dd02eae2fe98582a4556fd e025726006236520f7e91e196b9e7f139e0af5f4 5080238f16e8ba13d455622afbc79d2ce6ab9966

katre on 23 Oct 2018

0.19.0rc7 is failing in bazel_determinism_test, looks to be due to the cherrypick for #6456.

katre on 23 Oct 2018

Actually, it seems to be 5080238. Once we've rolled that back and verified the fix, I'll create rc8 without that cherrypick.

katre on 23 Oct 2018

Creating rc8:

scripts/release/release.sh create --force_rc=8 0.19.0 ac880418885061d1039ad6b3d8c28949782e02d6 9bc3b20053a8b99bf2c4a31323a7f96fabb9f1ec 54c2572a8cabaf2b29e58abe9f04327314caa6a0 20bfdc67dc1fc32ffebbda7088ba49ee17e3e182 914b4ce14624171a97ff8b41f9202058f10d15b2 83d406b7da32d1b1f6dd02eae2fe98582a4556fd e025726006236520f7e91e196b9e7f139e0af5f4

katre on 24 Oct 2018

Test failures, from https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/529

Test Bazel (Ubuntu 18.04 (no JDK))
rules_rust (every variety)

katre on 24 Oct 2018

Test Bazel (Ubuntu 18.04 (no JDK))

The one failure is in //src/test/shell/bazel:bazel_coverage_test, and includes:

warning: could not find a JDK 8 bootclasspath in external/local_jdk, falling back to --release

The diagnostic isn't very good (related: #6475), but the problem it's reporting is that JAVA_HOME isn't set, and we're trying to build Java code. This is the 'no JDK' build, so that indicates //src/test/shell/bazel:bazel_coverage_test didn't use to depend on Java support, and now does.

Depending on whether or not that's a regression, the fix may just be to disable the test in that configuration. @iirina WDYT?

cushon on 24 Oct 2018

Can I request another cherrypick?

https://github.com/bazelbuild/bazel/commit/5f312dd1678878fb7563eae0cd184f2270346352

This bug prevents root causes during the execution phase from being reported in a way that users can correlate. The root causes are all reported, but the connection isn't made. Note that this is not a regression, but we've had some users ask for it, and it also affects our own CI as we're uploading CI results to the "Build Status" console using BEP.

ulfjack on 24 Oct 2018

Creating rc9:

scripts/release/release.sh create --force_rc=9 0.19.0 ac880418885061d1039ad6b3d8c28949782e02d6 9bc3b20053a8b99bf2c4a31323a7f96fabb9f1ec 54c2572a8cabaf2b29e58abe9f04327314caa6a0 20bfdc67dc1fc32ffebbda7088ba49ee17e3e182 914b4ce14624171a97ff8b41f9202058f10d15b2 83d406b7da32d1b1f6dd02eae2fe98582a4556fd e025726006236520f7e91e196b9e7f139e0af5f4 5f312dd1678878fb7563eae0cd184f2270346352

katre on 24 Oct 2018

Tests pass, except the same issue with bazel_coverage_test.

katre on 24 Oct 2018

Barring any further blockers, I plan to release 0.19.0 on Monday.

katre on 26 Oct 2018

Release 0.19.0 is live! See https://github.com/bazelbuild/bazel/releases/tag/0.19.0

@petemounce and @vbatts, can you update the packages you maintain?

katre on 29 Oct 2018

👍5 🎉1

Blog post is live: https://blog.bazel.build/2018/10/29/bazel-0.19.0.html

katre on 29 Oct 2018

https://copr.fedorainfracloud.org/coprs/vbatts/bazel/build/816673/

vbatts on 29 Oct 2018

failed compile on centos7

BUILDSTDERR: INFO: From Compiling src/main/cpp/blaze_util_posix.cc [for host]:
BUILDSTDERR: INFO: From Compiling src/main/cpp/blaze_util_posix.cc [for host]:
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc: In function 'void blaze::Daemonize(const char*, bool)':
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc: In function 'void blaze::Daemonize(const char*, bool)':
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc:270:28: warning: ignoring return value of 'int dup(int)', declared with attribute warn_unused_result [-Wunused-result]
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc:270:28: warning: ignoring return value of 'int dup(int)', declared with attribute warn_unused_result [-Wunused-result]
BUILDSTDERR:    (void) dup(STDOUT_FILENO);  // stderr (2>&1)
BUILDSTDERR:    (void) dup(STDOUT_FILENO);  // stderr (2>&1)
BUILDSTDERR:                             ^
BUILDSTDERR:                             ^
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc: In function 'void blaze::DieAfterFork(const char*)':
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc: In function 'void blaze::DieAfterFork(const char*)':
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc:317:49: warning: ignoring return value of 'ssize_t write(int, const void*, size_t)', declared with attribute warn_unused_result [-Wunused-result]
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc:317:49: warning: ignoring return value of 'ssize_t write(int, const void*, size_t)', declared with attribute warn_unused_result [-Wunused-result]
BUILDSTDERR:    write(STDERR_FILENO, message, strlen(message));  // strlen should be OK
BUILDSTDERR:    write(STDERR_FILENO, message, strlen(message));  // strlen should be OK
BUILDSTDERR:                                                  ^
BUILDSTDERR:                                                  ^
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc:318:32: warning: ignoring return value of 'ssize_t write(int, const void*, size_t)', declared with attribute warn_unused_result [-Wunused-result]
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc:318:32: warning: ignoring return value of 'ssize_t write(int, const void*, size_t)', declared with attribute warn_unused_result [-Wunused-result]
BUILDSTDERR:    write(STDERR_FILENO, ": ", 2);
BUILDSTDERR:    write(STDERR_FILENO, ": ", 2);
BUILDSTDERR:                                 ^
BUILDSTDERR:                                 ^
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc:319:59: warning: ignoring return value of 'ssize_t write(int, const void*, size_t)', declared with attribute warn_unused_result [-Wunused-result]
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc:319:59: warning: ignoring return value of 'ssize_t write(int, const void*, size_t)', declared with attribute warn_unused_result [-Wunused-result]
BUILDSTDERR:    write(STDERR_FILENO, error_string, strlen(error_string));
BUILDSTDERR:    write(STDERR_FILENO, error_string, strlen(error_string));
BUILDSTDERR:                                                            ^
BUILDSTDERR:                                                            ^
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc:320:32: warning: ignoring return value of 'ssize_t write(int, const void*, size_t)', declared with attribute warn_unused_result [-Wunused-result]
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc:320:32: warning: ignoring return value of 'ssize_t write(int, const void*, size_t)', declared with attribute warn_unused_result [-Wunused-result]
BUILDSTDERR:    write(STDERR_FILENO, "\n", 1);
BUILDSTDERR:    write(STDERR_FILENO, "\n", 1);
BUILDSTDERR:                                 ^
BUILDSTDERR:                                 ^
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc: In function 'uint64_t blaze::AcquireLock(const string&, bool, bool, blaze::BlazeLock*)':
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc: In function 'uint64_t blaze::AcquireLock(const string&, bool, bool, blaze::BlazeLock*)':
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc:678:30: warning: ignoring return value of 'int ftruncate(int, __off_t)', declared with attribute warn_unused_result [-Wunused-result]
BUILDSTDERR: src/main/cpp/blaze_util_posix.cc:678:30: warning: ignoring return value of 'int ftruncate(int, __off_t)', declared with attribute warn_unused_result [-Wunused-result]
BUILDSTDERR:    (void) ftruncate(lockfd, 0);
BUILDSTDERR:    (void) ftruncate(lockfd, 0);
BUILDSTDERR:                               ^
BUILDSTDERR:                               ^

vbatts on 29 Oct 2018

err
wrong spot

BUILDSTDERR: [382 / 394] Compiling third_party/grpc/src/cpp/server/server_builder.cc [for host]; 0s linux-sandbox
BUILDSTDERR: [382 / 394] Compiling third_party/grpc/src/cpp/server/server_builder.cc [for host]; 0s linux-sandbox
BUILDSTDERR: [438 / 447] Compiling external/com_google_protobuf/src/google/protobuf/reflection_ops.cc [for host]; 0s linux-sandbox
BUILDSTDERR: [438 / 447] Compiling external/com_google_protobuf/src/google/protobuf/reflection_ops.cc [for host]; 0s linux-sandbox
BUILDSTDERR: [478 / 489] Compiling external/com_google_protobuf/src/google/protobuf/compiler/csharp/csharp_field_base.cc [for host]; 1s linux-sandbox ... (2 actions running)
BUILDSTDERR: [478 / 489] Compiling external/com_google_protobuf/src/google/protobuf/compiler/csharp/csharp_field_base.cc [for host]; 1s linux-sandbox ... (2 actions running)
BUILDSTDERR: [520 / 531] Compiling external/com_google_protobuf/src/google/protobuf/compiler/java/java_file.cc [for host]; 1s linux-sandbox ... (2 actions running)
BUILDSTDERR: [520 / 531] Compiling external/com_google_protobuf/src/google/protobuf/compiler/java/java_file.cc [for host]; 1s linux-sandbox ... (2 actions running)
BUILDSTDERR: [562 / 574] Compiling external/com_google_protobuf/src/google/protobuf/text_format.cc [for host]; 2s linux-sandbox ... (2 actions running)
BUILDSTDERR: [562 / 574] Compiling external/com_google_protobuf/src/google/protobuf/text_format.cc [for host]; 2s linux-sandbox ... (2 actions running)
BUILDSTDERR: ERROR: /builddir/build/BUILD/bazel-0.19.0/src/java_tools/junitrunner/java/com/google/testing/junit/junit4/BUILD:13:1: Compiling Java headers src/java_tools/junitrunner/java/com/google/testing/junit/junit4/librunner-hjar.jar (6 files) failed (Exit 1) java failed: error executing command external/embedded_jdk/bin/java -Xverify:none -XX:+UseParallelOldGC -XX:-CompactStrings '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' ... (remaining 44 argument(s) skipped)
BUILDSTDERR: ERROR: /builddir/build/BUILD/bazel-0.19.0/src/java_tools/junitrunner/java/com/google/testing/junit/junit4/BUILD:13:1: Compiling Java headers src/java_tools/junitrunner/java/com/google/testing/junit/junit4/librunner-hjar.jar (6 files) failed (Exit 1) java failed: error executing command external/embedded_jdk/bin/java -Xverify:none -XX:+UseParallelOldGC -XX:-CompactStrings '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' ... (remaining 44 argument(s) skipped)
BUILDSTDERR: Use --sandbox_debug to see verbose messages from the sandbox
BUILDSTDERR: Use --sandbox_debug to see verbose messages from the sandbox
BUILDSTDERR: Unrecognized VM option 'CompactStrings'
BUILDSTDERR: Unrecognized VM option 'CompactStrings'
BUILDSTDERR: Error: Could not create the Java Virtual Machine.
BUILDSTDERR: Error: Could not create the Java Virtual Machine.
BUILDSTDERR: Error: A fatal exception has occurred. Program will exit.
BUILDSTDERR: Error: A fatal exception has occurred. Program will exit.
BUILDSTDERR: Target //scripts:bazel-complete.bash failed to build
BUILDSTDERR: Target //scripts:bazel-complete.bash failed to build
BUILDSTDERR: Use --verbose_failures to see the command lines of failed build steps.
BUILDSTDERR: Use --verbose_failures to see the command lines of failed build steps.
BUILDSTDERR: INFO: Elapsed time: 296.200s, Critical Path: 13.34s, Remote (0.00% of the time): [queue: 0.00%, setup: 0.00%, process: 0.00%]
BUILDSTDERR: INFO: Elapsed time: 296.200s, Critical Path: 13.34s, Remote (0.00% of the time): [queue: 0.00%, setup: 0.00%, process: 0.00%]
BUILDSTDERR: INFO: 595 processes: 595 linux-sandbox.
BUILDSTDERR: INFO: 595 processes: 595 linux-sandbox.
BUILDSTDERR: FAILED: Build did NOT complete successfully
BUILDSTDERR: FAILED: Build did NOT complete successfully
BUILDSTDERR: FAILED: Build did NOT complete successfully
BUILDSTDERR: FAILED: Build did NOT complete successfully

vbatts on 29 Oct 2018

Do you know what JDK centos7 is using?

katre on 29 Oct 2018

@cushon Do you recognize the "CompactStrings" error?

katre on 29 Oct 2018

update: all fedora releases and centos failed.
java-1.8.0-openjdk-1.8.0.181.b15-6.fc28

vbatts on 29 Oct 2018

I recognize the flag. It's only supported by JDK 9 and newer, not JDK 8. That use might be coming from here:

https://github.com/bazelbuild/bazel/blob/41a1e837edd6d98c1e8ebdc5b6e0ec94b413f82c/tools/jdk/default_java_toolchain.bzl#L25

Is that a regression from 0.18? Can you provide more detail about your environment? i.e. do you have anything in your .bazelrc, or any custom java_toolchain definitions?

cushon on 29 Oct 2018

0.18.0 compiled just fine

vbatts on 29 Oct 2018

interesting, can you start a new bug and include more detail about the configuration that build is using?

cushon on 29 Oct 2018

Hi,
I am currently building docker containers that also involves compiling tensorflow using bazel. While doing this I got the new bazel version and since then no build works and I get ERROR: Config value cuda is not defined in any .rc file is there any way to easily fix that?
Edit: For now I managed to downgrade to 0.18 (using this https://github.com/bazelbuild/continuous-integration/issues/128#issuecomment-356224187 ) and can continue my work.

aboettcher on 29 Oct 2018

0.19.0 is published to chocolatey.

petemounce on 29 Oct 2018

@vbatts I filed #6550

cushon on 30 Oct 2018

@aboettcher From 0.19.0, bazel stops read bazelrc file at tools/bazel.rc, you should manually import it in your .bazelrc file under the source root. Basically, do
echo import %workspace%/tools/bazel.rc >> .bazelrc

meteorcloudy on 30 Oct 2018

@aboettcher From 0.19.0, bazel stops read bazelrc file at tools/bazel.rc, you should manually import it in your .bazelrc file under the source root. Basically, do
echo import %workspace%/tools/bazel.rc >> .bazelrc

Hmm. That would be a behavioral change which is not mentioned on the 0.19.0 release notes page, and also not found in git commit messages (e.g. git log --grep=bazel.rc - only shows the 0.18 release commit actually mentioning this bazel.rc file location in the workspace as a feature with a bugfix via #6321 / #6322).

gertvdijk on 30 Oct 2018

Thank you @meteorcloudy! I can confirm the comment from @meteorcloudy: adding the line fixes the problem using bazel 19. But as shown below in the output of the 0.18 version this error is expected (WARNING: Processed legacy workspace file /tmp/tensorflow/tools/bazel.rc. This file will not be processed in the next release of Bazel. Please read https://github.com/bazelbuild/bazel/issues/6319 for further information, including how to upgrade.). Still would be nice to mention it in the release notes I guess.

Details:
Bazel 0.19 without the fix from @meteorcloudy:

WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files:                                   
/tmp/tensorflow/tools/bazel.rc
Extracting Bazel installation...
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
You have bazel 0.19.0 installed.
Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See tools/bazel.rc for more details.
        --config=mkl            # Build with MKL support.
        --config=monolithic     # Config for mostly static monolithic build.
Configuration finished
WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files:
/tmp/tensorflow/tools/bazel.rc
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'build' from /tmp/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/opt/conda/bin/python --action_env PYTHON_LIB_PATH=/opt/conda/lib/python3.6/site-packages --python_path=/opt/conda/bin/python --de$ine with_jemalloc=true --define with_gcp_support=true --define with_hdfs_support=true --define with_xla_support=true --action_env TF_NEED_OPENCL_SYCL=0 --action_env TF_NEED_CUDA=$ --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda-10.0 --action_env TF_CUDA_VERSION=10.0 --action_env CUDNN_INSTALL_PATH=/usr/lib/x86_64-linux-gnu --action_env TF_CUDNN_VERSION=7 -$action_env TF_NCCL_VERSION=1 --action_env TF_CUDA_COMPUTE_CAPABILITIES=3.7,5.2,5.3,6.0,6.1,7.0 --action_env LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib$4 --action_env TF_CUDA_CLANG=0 --action_env GCC_HOST_COMPILER_PATH=/usr/bin/gcc --config=cuda --define grpc_no_ares=true
ERROR: Config value cuda is not defined in any .rc file
/tmp/install_tensorflow.sh: line 48: bazel-bin/tensorflow/tools/pip_package/build_pip_package: No such file or directory

Bazel 0.19 with the fix:

WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files:                                   
/tmp/tensorflow/tools/bazel.rc                                                                                                                                                     
Extracting Bazel installation...                                                                                                                                                   
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".                                                     
You have bazel 0.19.0 installed.                                                                                                                                                   
Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See tools/bazel.rc for more details.
        --config=mkl            # Build with MKL support.
        --config=monolithic     # Config for mostly static monolithic build.
Configuration finished
Starting local Bazel server and connecting to it...
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
    currently loading: tensorflow/tools/pip_package
Loading: 0 packages loaded
    currently loading: tensorflow/tools/pip_package
Loading: 0 packages loaded
    currently loading: tensorflow/tools/pip_package
DEBUG: /home/jovyan/.cache/bazel/_bazel_root/e5cce820cc082410b4fcc604db349066/external/bazel_tools/tools/cpp/lib_cc_configure.bzl:115:5: 
Auto-Configuration Warning: 'TMP' environment variable is not set, using 'C:\Windows\Temp' as default
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (1 packages loaded)
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (2 packages loaded, 0 targets configured)
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (318 packages loaded, 9588 targets configured)
WARNING: /tmp/tensorflow/tensorflow/core/BUILD:2463:1: in includes attribute of cc_library rule //tensorflow/core:framework_internal_headers_lib: '../../external/com_google_absl' resolves to 'external/com_google_absl' not below the relative path of its package 'tensorflow/core'. This will be an error in the future. Since this rule was created by the macro 'cc_header_only_library', the error might have been caused by the macro implementation in /tmp/tensorflow/tensorflow/tensorflow.bzl:1373:20
WARNING: /tmp/tensorflow/tensorflow/core/BUILD:2562:1: in includes attribute of cc_library rule //tensorflow/core:stream_executor_headers_lib: '../../external/com_google_absl' re$olves to 'external/com_google_absl' not below the relative path of its package 'tensorflow/core'. This will be an error in the future. Since this rule was created by the macro 'c$_header_only_library', the error might have been caused by the macro implementation in /tmp/tensorflow/tensorflow/tensorflow.bzl:1373:20
WARNING: /tmp/tensorflow/tensorflow/core/BUILD:2548:1: in includes attribute of cc_library rule //tensorflow/core:framework_headers_lib: '../../external/com_google_absl' resolves to 'external/com_google_absl' not below the relative path of its package 'tensorflow/core'. This will be an error in the future. Since this rule was created by the macro 'cc_head$r_only_library', the error might have been caused by the macro implementation in /tmp/tensorflow/tensorflow/tensorflow.bzl:1373:20
WARNING: /home/jovyan/.cache/bazel/_bazel_root/e5cce820cc082410b4fcc604db349066/external/grpc/BUILD:1992:1: in srcs attribute of cc_library rule @grpc//:grpc_nanopb: please do no$ import '@grpc//third_party/nanopb:pb_common.c' directly. You should either move the file to this package or depend on an appropriate rule there. Since this rule was created by t$e macro 'grpc_generate_one_off_targets', the error might have been caused by the macro implementation in /home/jovyan/.cache/bazel/_bazel_root/e5cce820cc082410b4fcc604db349066/ex$ernal/grpc/bazel/grpc_build_system.bzl:172:12
WARNING: /home/jovyan/.cache/bazel/_bazel_root/e5cce820cc082410b4fcc604db349066/external/grpc/BUILD:1992:1: in srcs attribute of cc_library rule @grpc//:grpc_nanopb: please do no$ import '@grpc//third_party/nanopb:pb_decode.c' directly. You should either move the file to this package or depend on an appropriate rule there. Since this rule was created by t$e macro 'grpc_generate_one_off_targets', the error might have been caused by the macro implementation in /home/jovyan/.cache/bazel/_bazel_root/e5cce820cc082410b4fcc604db349066/ex$ernal/grpc/bazel/grpc_build_system.bzl:172:12

Despite all the warnings the compile continues.

As a reference the output using bazel 18 without the fix (what I usually did before all the problems started):

WARNING: Processed legacy workspace file /tmp/tensorflow/tools/bazel.rc. This file will not be processed in the next release of Bazel. Please read https://github.com/bazelbuild/bazel/issues/6319 for further information, including how to upgrade.
Extracting Bazel installation...
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.protobuf.UnsafeUtil (file:/home/jovyan/.cache/bazel/_bazel_root/install/f1e11885a5cc7ba9947679cffb18bf94/_embedded_binaries/A-server.jar) to field java.lang.String.value
WARNING: Please consider reporting this to the maintainers of com.google.protobuf.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
You have bazel 0.18.0 installed.
Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See tools/bazel.rc for more details.
        --config=mkl            # Build with MKL support.
        --config=monolithic     # Config for mostly static monolithic build.
Configuration finished
WARNING: Processed legacy workspace file /tmp/tensorflow/tools/bazel.rc. This file will not be processed in the next release of Bazel. Please read https://github.com/bazelbuild/bazel/issues/6319 for further information, including how to upgrade.
Starting local Bazel server and connecting to it...
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
    currently loading: tensorflow/tools/pip_package
Loading: 0 packages loaded
    currently loading: tensorflow/tools/pip_package
Loading: 0 packages loaded
    currently loading: tensorflow/tools/pip_package
DEBUG: /home/jovyan/.cache/bazel/_bazel_root/e5cce820cc082410b4fcc604db349066/external/bazel_tools/tools/cpp/lib_cc_configure.bzl:115:5: 
Auto-Configuration Warning: 'TMP' environment variable is not set, using 'C:\Windows\Temp' as default
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (1 packages loaded)
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (270 packages loaded)
WARNING: /tmp/tensorflow/tensorflow/core/BUILD:2463:1: in includes attribute of cc_library rule //tensorflow/core:framework_internal_headers_lib: '../../external/com_google_absl' resolves to 'external/com_google_absl' not below the relative path of its package 'tensorflow/core'. This will be an error in the future. Since this rule was created by the macro 'cc_header_only_library', the error might have been caused by the macro implementation in /tmp/tensorflow/tensorflow/tensorflow.bzl:1373:20
WARNING: /tmp/tensorflow/tensorflow/core/BUILD:2562:1: in includes attribute of cc_library rule //tensorflow/core:stream_executor_headers_lib: '../../external/com_google_absl' resolves to 'external/com_google_absl' not below the relative path of its package 'tensorflow/core'. This will be an error in the future. Since this rule was created by the macro 'cc_header_only_library', the error might have been caused by the macro implementation in /tmp/tensorflow/tensorflow/tensorflow.bzl:1373:20
WARNING: /tmp/tensorflow/tensorflow/core/BUILD:2548:1: in includes attribute of cc_library rule //tensorflow/core:framework_headers_lib: '../../external/com_google_absl' resolves to 'external/com_google_absl' not below the relative path of its package 'tensorflow/core'. This will be an error in the future. Since this rule was created by the macro 'cc_header_only_library', the error might have been caused by the macro implementation in /tmp/tensorflow/tensorflow/tensorflow.bzl:1373:20

aboettcher on 30 Oct 2018

The changed location of the rc files was discussed extensively in the release process for 0.18. I agree that it should have been mentioned in the release notes. It wasn't because there was no actual change enabling it: the location change was present originally in 0.18 (and mentioned in those release notes) before being removed.

katre on 30 Oct 2018

@vbatts and @cushon Thanks for the details.

@vbatts Would it work to update the required jdk in bazel.spec? Right now it specifies "java-1.8.0-openjdk-headless", I don't know if fedora or centos have a jdk9 available.

katre on 30 Oct 2018

@katre centos7 does not have openjdk 9, it looks like fedora does.

vbatts on 30 Oct 2018

RPM-based builds are fixed downstream, see #6550 for details.

katre on 30 Oct 2018

With RPM packages in place, I am closing the tracking issue for this release.

katre on 30 Oct 2018

An error has been flagged with 0.19.0's compatibility with the mingw compiler (on Windows, obviously). See #6610 for details.

Looks like cherrypicking c3fb1db9e4e817e8a911f5b347b30f2674a82f7c will fix the issue. I may also cherrypick 8e280838e8896a6b5eb5421fda435b96b6f8de60 to add tests for this case and to be sure it is addressed.

I'll start the process of creating a RC today, it will need to bake for at least two days by the release policy.

katre on 7 Nov 2018

👍1

Release creation command:

./scripts/release/release.sh create --force_rc=1 0.19.1 0.19.0 8e280838e8896a6b5eb5421fda435b96b6f8de60

(Only cherrypicking the tests, first, so I can make sure they fail and catch the underlying issue).

katre on 7 Nov 2018

The rbe_ubuntu error (https://buildkite.com/bazel/bazel-bazel/builds/5383#55a74208-1ed3-4ea0-a43f-5a3c2abe7719) seems to be that the worker was updated to use a later crosstool_top, but the bazel_toolchains repository in the release is too old. Ignoring this error.

katre on 7 Nov 2018

As expected, 0.19.1rc1 failed on Windows, creating RC2 with the actual fix.:

./scripts/release/release.sh create --force_rc=2 0.19.1 0.19.0 c3fb1db9e4e817e8a911f5b347b30f2674a82f7c 8e280838e8896a6b5eb5421fda435b96b6f8de60

katre on 7 Nov 2018

❤1 👍1

Need an rc3, to update bazel-toolchains, so that CI can progress:

./scripts/release/release.sh create --force_rc=3 0.19.1 0.19.0 c3fb1db9e4e817e8a911f5b347b30f2674a82f7c 8e280838e8896a6b5eb5421fda435b96b6f8de60 fd52341505e725487c6bc6dfbe6b5e081aa037da

(CI error: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/571#be600c55-5dee-457c-b5ee-8f4f3297da59)

katre on 7 Nov 2018

Noone has reported any issues with the RC, but I'm going to wait until Monday to release it so I can be sure nothing goes wrong when I can't try and fix it.

katre on 9 Nov 2018

No blockers reported, going ahead with the release.

katre on 12 Nov 2018

Release 0.19.1 is ready. @petemounce and @vbatts, can you also update?

katre on 12 Nov 2018

On 12/11/18 11:18 -0500, katre wrote:

Release 0.19.1 is ready. [1]@petemounce and [2]@vbatts, can you also
update?

https://copr.fedorainfracloud.org/coprs/vbatts/bazel/build/823004/

vbatts on 12 Nov 2018

Published to chocolatey. I noticed #6665 / #6662 were opened however :(

petemounce on 12 Nov 2018

Yes, this is unfortunate, I think we'll need a 0.19.2 as soon as I can figure out the solutions.

katre on 12 Nov 2018

also mentioned in the issues, I think the problem is 0.19.1 did not include https://github.com/bazelbuild/bazel/commit/6bc452874ddff69cbf7f66186238032283f1195f

nlopezgi on 12 Nov 2018

I am going to create 0.19.2 with f7e5aef145c33968f658eb2260e25630dc41cc67 and 6bc452874ddff69cbf7f66186238032283f1195f. @nlopezgi please advise on how I can test for issue #6665 before I do the actual release.

katre on 13 Nov 2018

Release 0.19.2rc1:

./scripts/release/release.sh create create --force_rc=1 0.19.2 0.19.1 6bc452874ddff69cbf7f66186238032283f1195f f7e5aef145c33968f658eb2260e25630dc41cc67

katre on 13 Nov 2018

RC1 testing: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/586#_
The test failures in singlejar have two causes:

src/tools/singlejar:output_jar_simple_test references the target @local_jdk//:jdk-default, which may have existed when that commit was submitted but does not exist now; and
Tests for singlejar were not run until d5713548b636264f68bfc5f601d83693e58a6079, last week.

Given the massive changes in src/tools/singlejar/BUILD since the version in the release-0.19.2 branch, I don't think a fix is feasible.

The other error, rules_apple, fails in //test:ios_imessage_test.device. I am investigating.

Further, a user has reported the following error building their project:

ERROR: /usr/local/google/home/minors/.cache/bazel/_bazel_minors/86021668b1f881516e569ba9a9189b20/external/local_config_cc/BUILD:57:1: in cc_toolchain rule @local_config_cc//:cc-compiler-k8: Error while selecting cc_toolchain: Toolchain identifier 'local' was not found, valid identifiers are [stub_armeabi-v7a, local_linux, local_darwin, local_freebsd, local_windows_mingw, local_windows_msys64_mingw64, local_windows_clang, local_windows_msys64, vc_14_0_x64]
ERROR: Analysis of target '//REDACTED:REDACTED' failed; build aborted: Analysis of target '@local_config_cc//:cc-compiler-k8' failed; build aborted
INFO: Elapsed time: 0.246s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)

Given these, I am not publishing this RC, and will track down the errors to try and find a fix.

katre on 13 Nov 2018

Filed https://github.com/bazelbuild/rules_apple/issues/245 for the rules_apple issue.

katre on 13 Nov 2018

It's a known flake.

katre on 13 Nov 2018

Considering cherrypicking 683c302129b66a8999f986be5ae7e642707e978c to fix #6662 and #6665, waiting to see if it addresses the reported problems.

katre on 14 Nov 2018

Testing looks good, I am going to go ahead and create 0.19.2rc2 with this additional cherrypick so it can be tested.

katre on 14 Nov 2018

Created rc2:

./scripts/release/release.sh create --force_rc=2 0.19.2 0.19.1 6bc452874ddff69cbf7f66186238032283f1195f f7e5aef145c33968f658eb2260e25630dc41cc67 683c302129b66a8999f986be5ae7e642707e978c

katre on 14 Nov 2018

RC2 shows serious errors in cc-rules-tests (https://storage.googleapis.com/bazel-buildkite-artifacts/e813c7da-880e-4b59-b805-c14f30289cbd/src/test/java/com/google/devtools/build/lib/rules/cpp/cpp-rules-tests/attempt_1.log)

katre on 14 Nov 2018

@scentini has found the relevant commits that should fix CcToolchainTests. One applies many changes (b91d8308e1b8cf165d6b8a32f593d63684ad185c), the second rolls most of them back (00ac609c20c49fbf6433810155c10822ea94be41), but should leave the tests fixed. Trying that today, hopefully can get RC3 out. (Might also need fc7e7405ef8db508ed3c92161bf3913e0f77c079, which comes between the two.)

katre on 15 Nov 2018

Yes, it does need fc7e740.

katre on 15 Nov 2018

No, that's not enough. This is turning into a lot of cherrypicks for one test, looking into alternatives

katre on 15 Nov 2018

Since the error is known and is the result of a mis-configured test, I'm ignoring it and releasing RC2 anyway.

katre on 15 Nov 2018

Sent announcement about 0.19.2RC2.

katre on 15 Nov 2018

will 0.19.1 hit homebrew?

sergiocampama on 16 Nov 2018

It is supposed to, looks like the PR (https://github.com/bazelbuild/homebrew-tap/pull/17) failed in CI due to xcode versions, so I filed an issue (https://github.com/bazelbuild/homebrew-tap/issues/18) which hasn't been addressed yet, I'll poke people, but 0.19.2 should be released on Mondayish.

katre on 16 Nov 2018

Planning to release 0.19.2 today, since I haven't seen any further blockers reported.

katre on 19 Nov 2018

Bazel release 0.19.2 is now available.

@petemounce and @vbatts, can you please update your packages?

I will update Homebrew now.

katre on 19 Nov 2018

Bazel's Homebrew-tap project updated: https://github.com/bazelbuild/homebrew-tap/commit/8e0ab3dd0a2570492277c0f91552da4e82f57073

katre on 19 Nov 2018

On 19/11/18 11:37 -0500, katre wrote:

Bazel release 0.19.2 is now available.
[1]@petemounce and [2]@vbatts, can you please update your packages?

https://copr.fedorainfracloud.org/coprs/vbatts/bazel/build/826621/

vbatts on 19 Nov 2018

0.19.2 is available on chocolatey.

petemounce on 19 Nov 2018

Thanks! Closing issue now that release is complete.

katre on 19 Nov 2018

Bazel: Release 0.19 - October 2018

Most helpful comment

All 111 comments

6283 is fixed by 20bfdc67dc1fc32ffebbda7088ba49ee17e3e182, will cherrypick that into rc2 today.

6433 is not a release blocker.

Related issues