Nixpkgs: Hydra.nixos.org problems with database (probably)

[vcunat]

Jobsets now fail to evaluate with errors like this extended example from trunk-combined:

Errors occurred at 2020-05-22 19:58:57.

DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR:  insert or update on table "builds" violates foreign key constraint "fk3"
DETAIL:  Key (project, jobset, job)=(nixos, trunk-combined, nixpkgs.xfce.xfce4_eyes_plugin.x86_64-linux) is not present in table "jobs". [for Statement "UPDATE builds SET iscurrent = ? WHERE ( id IN ( SELECT build.id FROM jobsetevalmembers me  JOIN builds build ON build.id = me.build WHERE ( me.eval = ? ) ) )" with ParamValues: 1='1', 2='1588817'] at /nix/store/zwhrmhwqkwwvg6454z320993g6947z0c-hydra-0.1.20200512.15a45f1/bin/.hydra-eval-jobset-wrapped line 750

It seems very unlikely to have been caused by nixpkgs changes, as I see it on multiple branches. (I'm opening here anyway, for lack of knowledge of a better place.)

[vcunat]

@grahamc: I hope you know who might look into this.

[grahamc]

Can you paste some more info with like links, logs, times, etc. for future reference? Looking in to it.

[FRidh]

Just now after I wanted staging-next to be evaluated (about half an hour ago, so May 23rd 10:30 CEST).
https://hydra.nixos.org/jobset/nixpkgs/staging-next#tabs-errors
8a77c900ddf7477e5e2ffbdb41008f37f4ef0188

Errors occurred at 2020-05-23 10:58:22.

DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR:  insert or update on table "builds" violates foreign key constraint "fk3"
DETAIL:  Key (project, jobset, job)=(nixpkgs, staging-next, xfce.xfce4_cpugraph_plugin.x86_64-linux) is not present in table "jobs". [for Statement "UPDATE builds SET iscurrent = ? WHERE ( id IN ( SELECT build.id FROM jobsetevalmembers me  JOIN builds build ON build.id = me.build WHERE ( me.eval = ? ) ) )" with ParamValues: 1='1', 2='1588935'] at /nix/store/zwhrmhwqkwwvg6454z320993g6947z0c-hydra-0.1.20200512.15a45f1/bin/.hydra-eval-jobset-wrapped line 750

[vcunat]

Not all jobsets suffer from this, e.g. the -small ones still appear to work fine.

_I don't know... say, transaction size might be relevant or something. I'm not a database person._

[grahamc]

Just saying this is a bit out of my league so far. I've pinged Eelco to try and take a look.

[FRidh]

The python-unstable job also presented the same error, however, eventually did proceed and jobs are building now.

[edolstra]

Fixed, see https://github.com/NixOS/hydra/issues/766#issuecomment-634655430.