Nixpkgs: Channels not advancing

[FRidh]

Issue description

The nixpkgs-unstable channel does not seem to be advancing despite being in good state.

https://howoldis.herokuapp.com/

[vcunat]

I see a couple of stuck jobs (queued) up to 5 days old evals, so I'm cancelling them. I assume that was the blocker.

[danbst]

@vcunat is "cancelling queued jobs" happening often enough to introduce some automation for that? Is there an issue open for Hydra?

[vcunat]

Sometimes it happens that I see a job "executing" for several days, which is clearly some kind of stuck state, and not even cancelling will stop the state (but it removes the job from "queued" status). Normally Hydra does apply a timeout and cancels stuff – actually the limit is now often too short for us on some aarch64 builds (ghc, llvm, fortran).

[danbst]

@vcunat I see there that machine d3bcab1f is executing lots of jobs for 1d 6h already. Is that stuck as well?

Also, in case of timeout, is build retried on another, more powerful machine? Is build timeout same for all machines? (yeah, those aarch GHC builds)

[vcunat]

IIRC we can increase timeouts per-job from the nix expressions, so I suppose we'll do that for some of the expensive builds, regardless of platform. _We might also mark them as big-parallel, but that won't make much difference on our current Hydra setup._

[grahamc]

I found several jobs hanging on the hydra server, and I've since terminated them.

[grahamc]

I just verified that the channel advancement tooling is not having problems, so presumably it is just the hydra jobs at the moment.

[vcunat]

Yes, the channel did update quite fast after I cancelled the laggards, so that part is probably OK. Some automatic avoidance of these occasional problems (stuck builds) would sure be nice, but it doesn't seem to cause too much trouble so far.

[grahamc]

One reason we don't automatically do that is usually it represents bugs in Nix, so I try to capture some stack traces to investigate.

[vcunat]

Yay, with nix you can dare to debug in production... I guess? (sometimes?)

[globin]

It has advanced, nixos-unstable hasn't yet but it looks to be running fine.

[mmahut]

It's published.