Nixpkgs: chromium build cycling on Hydra
The chromium builds are now repeatedly "aborted" on Hydra.nixos.org, and it isn't even clear why. The result is huge waste of resources, and blocked channel due to never finishing.
Examples:
- https://hydra.nixos.org/build/83226757#tabs-buildsteps
- https://hydra.nixos.org/build/83250657#tabs-buildsteps
- https://hydra.nixos.org/build/83250711#tabs-buildsteps
I see it happening on two different machines. @edolstra: any idea how to get some logs from the abortions? Otherwise I can't see how to deal with it. Also /cc @grahamc (Packet machines).
My first guess would be that it exhausts some resource, e.g. /tmp/ space.
vcunat
All 14 comments
Looking at job lists (e.g. trunk), it always started exactly on 69 -> 70 bump, both on master and 18.09, so we could most likely avoid the problem temporarily by reverting that bump.
vcunat
on 30 Oct 2018
I now at least manually cancelled most of the jobs, so there aren't too many versions tried at once.
vcunat
on 30 Oct 2018
This might be a hint:
https://github.com/NixOS/nixpkgs/blob/ed91407784dce5a5ebe7518fcc51581207f8027f/pkgs/applications/networking/browsers/chromium/common.nix#L274-L281
volth
on 30 Oct 2018
Synthetica9
on 30 Oct 2018
That post seems windows-specific. It's over a year old, and at least at that time I had no such issues when building chromium via nix(os) in background (I haven't tried now yet).
vcunat
on 30 Oct 2018
Hm, so it started failing before switching to clang, which is simultaneously good and bad news ;-)
Let's hope, it can be resolved as part of https://github.com/NixOS/nix/issues/2493 as Eelco suggested.
Sorry, that I didn't catch this failure. I have now starred
https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.chromium.x86_64-linux and enabled notifications on hydra. That should mean, I get notified of such conditions, right?
Unfortunately, the aborted jobs don't seem to have log files available. Do you have any hint as to what caused it to abort? @vcunat
We don't happen to build chromium on a tmpfs, do we?
I'll keep an eye on https://hydra.nixos.org/build/83226757/nixlog/140/tail
If it goes through, can I re-attempt the version bump right away, or should I wait until the channel finishes?
bendlas
on 30 Oct 2018
No, I found no relevant logs.
vcunat
on 31 Oct 2018
/tmp(fs): a superficial search in configs suggested that only ike uses it, and that one doesn't have the "big-parallel" feature (so shouldn't get chromium).
vcunat
on 31 Oct 2018
OK, that build you linked succeeded :tada: I hope that's it, but we'll see.
vcunat
on 31 Oct 2018
The build succeeded but the tests are now failing due to changes in the sandbox outputs.
@bendlas Could you have a look at them?
Here is a screenshot of the VM test that fails:
The logs from 18.09 before the recent systemd bump: https://gist.github.com/andir/2b64a68bb0bf953ebe02102aca358b44 & hydra log for comparison https://nix-cache.s3.amazonaws.com/log/adaq71i6yc400pm392d4hc04xwl9ick1-vm-test-run-chromium-stable.drv
hydra build: https://hydra.nixos.org/build/83250711
andir
on 1 Nov 2018
Related problem I see now: https://github.com/NixOS/hydra/issues/591#issuecomment-435125621
vcunat
on 1 Nov 2018
I've pushed https://github.com/NixOS/nixpkgs/commit/89ede978a0614fd7215e0aa2c483df142cc79af9 in a desperate attempt to fix timeout issues.
domenkozar
on 1 Nov 2018
Here's an out-of-space error on packet-t2-4, but those aren't that often.
vcunat
on 2 Nov 2018
We now had multiple successes of the test and chromium itself, on both branches. Thanks everyone!
vcunat
on 3 Nov 2018
Related issues
timokau
路
66Comments
nh2
路
76Comments
grahamc
路
88Comments
grahamc
路
77Comments
worldofpeace
路
103Comments
Most helpful comment
We now had multiple successes of the test and chromium itself, on both branches. Thanks everyone!