Nixpkgs: Chromium dead on release 17.09?

[copumpkin]

If I look at https://hydra.nixos.org/build/69805372, it seems like the chromium build is broken for a nontrivial reason:

[12876/30173] ACTION //third_party/WebKit/public:android_mojo_bindings__generator(//build/toolchain/linux:x64)
[12877/30173] ACTION //third_party/WebKit/public:android_mojo_bindings__generator(//build/toolchain/linux:x64)
[12878/30173] STAMP obj/third_party/WebKit/common/mojo_bindings.inputdeps.stamp
[12879/30173] ACTION //third_party/WebKit/public:android_mojo_bindings__generator(//build/toolchain/linux:x64)
[12880/30173] STAMP obj/third_party/WebKit/public/android_mojo_bindings__generator.stamp
[12881/30173] STAMP obj/url/mojo/url_mojom_gurl__generator.inputdeps.stamp
[12882/30173] CXX obj/third_party/WebKit/common/mojo_bindings/message_port.mojom.o
FAILED: obj/third_party/WebKit/common/mojo_bindings/message_port.mojom.o 
g++ -MMD -MF obj/third_party/WebKit/common/mojo_bindings/message_port.mojom.o.d -DBLINK_COMMON_IMPLEMENTATION=1 -DBLINK_CORE_IMPLEMENTATION=1 -DV8_DEPRECATION_WARNINGS -DUSE_UDEV -DUSE_AURA=1 -DUSE_GLIB=1 -DUSE_NSS_CERTS=1 -DUSE_X11=1 -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCXXABI_DISABLE_VISIBILITY_ANNOTATIONS -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DGLIB_VERSION_MAX_ALLOWED=GLIB_VERSION_2_32 -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_26 -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DU_STATIC_IMPLEMENTATION -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -DUCHAR_TYPE=uint16_t -I../.. -Igen -I/nix/store/00wprg4rdsay4a73snv4qkjnhx7ls48l-glib-2.52.3-dev/include/glib-2.0 -I/nix/store/vmyh7mpa2nw3d2fs1lmj523z897gcj6i-glib-2.52.3/lib/glib-2.0/include -I../../third_party/ced/src -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -funwind-tables -fPIC -pipe -pthread -m64 -march=x86-64 -Wall -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-deprecated-declarations -fno-delete-null-pointer-checks -Wno-missing-field-initializers -Wno-unused-parameter -O2 -fno-ident -fdata-sections -ffunction-sections -fno-omit-frame-pointer -g0 -fno-builtin-abs -fvisibility=hidden -std=gnu++14 -Wno-narrowing -fno-exceptions -fno-rtti -nostdinc++ -isystem../../buildtools/third_party/libc++/trunk/include -isystem../../buildtools/third_party/libc++abi/trunk/include -fvisibility-inlines-hidden -c gen/third_party/WebKit/common/message_port/message_port.mojom.cc -o obj/third_party/WebKit/common/mojo_bindings/message_port.mojom.o
In file included from gen/third_party/WebKit/common/message_port/message_port.mojom.cc:15:0:
gen/third_party/WebKit/common/message_port/message_port.mojom.h:38:55: fatal error: third_party/WebKit/common/blob/blob.mojom.h: No such file or directory
 #include "third_party/WebKit/common/blob/blob.mojom.h"
                                                       ^
compilation terminated.
[12883/30173] ACTION //third_party/WebKit/common:mojo_platform_bindings__generator(//build/toolchain/linux:x64)
[12884/30173] ACTION //url/mojo:url_mojom_gurl__generator(//build/toolchain/linux:x64)
[12885/30173] ACTION //third_party/WebKit/common:mojo_platform_bindings__generator(//build/toolchain/linux:x64)
ninja: build stopped: subcommand failed.
builder for ‘/nix/store/pjw5zfwb359iylgj6r6i3760dq85wxkn-chromium-64.0.3282.167.drv’ failed with exit code 1

Which might be nondeterminism, because that same version seems to have built in the past for us.

cc @bendlas who might know more. I can trigger a Hydra rebuild but I think that kills existing logs, right?

[copumpkin]

Here's the preserved build log, since I want to trigger a rebuild on Hydra: https://gist.github.com/copumpkin/981fb9aab48779a807ae0b30acb91f1a

[vcunat]

The same again. It looks like a parallel-~make~ ninja problem.

[bendlas]

Of course, this is bound to happen, when I come home in the middle of the night, drunk. Ok, let's see.

Mojom generation seems to break in semi-random ways.
In the AUR thread here and here, people posted patches. I'll try to use those as a reference, to make message_port.mojom.h depend on blob.mojom.h.

cc @aszlig @YorikSar do you think, I'm on the right track with this?

[vcunat]

Sounds a good track to me. BTW, in some packages it's easier to explicitly make the problematic target before building the rest, instead of trying to add dependencies.

[bendlas]

Great thinking! @vcunat

Following the money, I found the target mojo_platform_bindings to build base.mojom, So I tried inserting it before the chrome target.
Strange thing is: The target building message_port.mojom, mojo_bindings, already depends on mojo_platform_bindings ... :-/
Perhaps this is an actual parallel build bug in ninja ... let's see if it helps.
Could somebody attempt the fix on hydra? I'm drunk, I shouldn't commit, right now. Also, I need to go to sleep. Good night.

[copumpkin]

I don't know the build so don't feel confident trying to fix it. I've tried triggering the Hydra build again but it seems to be failing somewhat consistently so far (or at least failed three times in a row now).

[copumpkin]

My most recent attempt at a rebuild worked, so 17.09 is happy again, but if any input to chromium changes again, it'll probably fail.

[bendlas]

I've pushed the fix to master, along with a non-security-critical stable update. If it builds, I'll pick it to 17.09.

[bendlas]

Hm, seems like the build is failing again: https://hydra.nixos.org/build/69979799
While the master build hasn't started yet: https://hydra.nixos.org/build/70051726

I've picked it right now: https://github.com/NixOS/nixpkgs/commit/2ac349a65d0e288314b72846d4adb79ad1c885c6
Can hardly get any worse.

[YorikSar]

It looks like we need this patch: https://chromium.googlesource.com/chromium/src/+/b1e3cfd4f9bfe43a1e08c5670b51c8c80d3e6372 - without it dependency between mojo_platform_bindings and mojo_bindings was being cut (I don't know the original reason behind this).

@bendlas Are you still up? Can you try to apply that patch and start a build? I'm burned out for today...

[bendlas]

Good find! I'll apply this instead of my workaround.

Thanks! @YorikSar
Out of interest: was that indexed by google, or did you grep commit messages?

[bendlas]

meta

Is Review the right label to signify, that a ticket is due to be closed as soon as the fix is verified?

[YorikSar]

Thanks for adding it @bendlas! BTW, good job abstracting out fetching patch from GitHub. I would've just added git-review format-patch'ed version here.

To shame I don't know where to check status of Hydra build for this commit. Can someone guide me there?

was that indexed by google, or did you grep commit messages?

I was looking at what's different about mojo_bindings target first in 64.x, then in master. I've noticed that these two lines are gone in master, went to mojo/public/tools/bindings/mojom.gni to see what they meant. It says there that these lines just cut out mojo_platform_bindings dependency, so I went to see if they have been dropped on purpose. Then I've just spent some time looking through commits to third_party/WebKit/common/BUILD.gn and found this one :-)

[YorikSar]

I've found only https://hydra.nixos.org/build/70142746 and https://hydra.nixos.org/build/70051726 but they have been cancelled... Are there any job running to verify if this patch works?..

[vcunat]

Here: https://hydra.nixos.org/build/70212592#tabs-summary but with parallel-make problems you never have certainty you weren't just lucky this time.

[vcunat]

Let's close until the problem re-appears.

[YorikSar]

@vcunat Thanks. One of these days I'll figure out how Hydra works with GitHub and stuff :)

[bendlas]

unstable-channel is built here: https://hydra.nixos.org/jobset/nixos/trunk-combined
release here: https://hydra.nixos.org/jobset/nixos/release-17.09

Click on the latest build and use search to check for specific package