Nixpkgs: easier detection of mass rebuilds

[vcunat]

A wishlist coming from IRC discussion:

• [x] have a tool that outputs (numerically) how much a change would rebuild (e.g. a PR). nox review-pr (link) does something a bit different but it might be easy to modify it.
• [x] a GitHub bot that determines whether a PR is mass-rebuild.
  
  
  
  • [x] auto-tag the PR, and/or
  
  
  • [ ] give alert if a mass-rebuild PR is open against a non-staging branch.
  
  
• [ ] a rough time estimate could be done by summing average build time reported by Hydra.

[copumpkin]

Another thing that would be interesting is separating mass-darwin-rebuild from mass-rebuild, since the different stdenvs often include different components. So changing python doesn't cause a mass linux rebuild but it does cause a mass darwin rebuild, because the llvm build process depends on python and our toolchain is llvm-based.

Either way, eager 👍 from me.

[vcunat]

I imagined counting exactly the package sets as Hydra does (over all platforms built there).

[copumpkin]

Okay cool! Just wanted to make sure it didn't assume platform was same as
host platform :)
On Wed, Sep 28, 2016 at 13:44 Vladimír Čunát [email protected]
wrote:

I imagined counting exactly the package sets as Hydra does (over all
platforms built there).

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/issues/19045#issuecomment-250242494,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAKP6JDztn6TBPeU29ZgRCrubjXVjSNks5quqeTgaJpZM4KI_dv
.

[vcunat]

ATM nox does assume that, I think.

[domenkozar]

I'll practice my Haskell skills and write this, it's a huge step forward for the community. But only in October :)

[vcunat]

@domenkozar: nox was written in Python, not Haskell ;-)

[domenkozar]

I know, but nox takes a different path than we should've. It needs to be rewritten from scratch.

[vcunat]

Ah. I assumed small tweaks would be enough, but if you want to implement it I certainly won't restrict you (and generally I do prefer Haskell over Python).

[LnL7]

it looks like creating a diff with --out-path could be used to do part of this, but there are a bunch of issues with that approach.

nix-env -f https://github.com/NixOS/nixpkgs/archive/master.tar.gz -qaP --out-path --argstr system x86_64-linux

[domenkozar]

That's what nox currently does :)

[vcunat]

A simple tool for now: https://gist.github.com/vcunat/b2eb3e8e8a916f1656ba4d1c7e5e4e32 Sure, feel free to come with something better.

[domenkozar]

This suffers from https://github.com/NixOS/nixpkgs/issues/7304

[vcunat]

Yes, I know, and from at least one another bug noted under the gist. To do this properly, one might have to work with the C++ API, as the commands don't expose everything I'd like for it.

[vcunat]

I would also like it to print the "roots of change", i.e. those derivations in the DAG that have changed their outPath but none of their dependencies did change their outPath. Even in mass rebuilds, such list of "roots" should be typically very small. (EDIT: this might include the dependency graph of a few levels above those points.)

[Ericson2314]

Comming from https://github.com/NixOS/nixpkgs/issues/20393, let me give this a +1, and try @vcunat's tool.

[LnL7]

I anybody working on an alternative for vuncat's script at the moment?
If not I would suggest we add this to the maintainer scripts so other people can use and improve it.

[vcunat]

I'm not aware of anyone.

[vcunat]

OK, I have a better WIP implementation now (reference above). It's simple bash+nix, and it should fix problems like #7304. Feedback welcome. (and ideas on how to plug it into our Travis, etc.)

Example usage:

$ maintainers/scripts/rebuild-amount.sh master HEAD
Estimating rebuild amount by counting changed Hydra jobs.
   8366 x86_64-darwin
  14456 x86_64-linux

[copumpkin]

@vcunat where is that? I don't see a revision on the gist

Edit: never mind, I missed the commit reference!

[copumpkin]

Anyone have experience with GitHub bots? It would be super cool to plug this into all PRs and maybe even individual commits on master and have a bot just comment on each commit saying how much gets rebuilt for each platform (or mentioning that there's an evaluation error being introduced)

[vcunat]

I planned integrating this into our travis. In particular, I'd like to avoid travis attempting mass rebuilds by making nox-review (or similar) conditional on this check. I'd probably also replace or amend nox-review to use the same package list (so it will e.g. work with haskell stuff as well).

I suppose the bot could just read the travis result and post a comment and/or add a label. (But no experience with these on my side.) It could also remain without a bot, just as another way of failing the travis checks.

[Ericson2314]

@vcunat that sounds great and is just what I was thinking.

Just yesterday, somebody landed a mass rebuild to master by mistake (for all I know, maybe it was me!) and the ensuing destruction of CI was quite annoying.

[vcunat]

I've been waiting for staging to finish building the single evaluation for about a week now. I'll probably have to cancel, merge master and redo yet again, but there's been almost no staging jobs run recently anyway, so I'm waiting for an optimal time to reevaluate. Staging is a low-priority jobset on purpose, but our current practice just doesn't work, as master seems to have finished more builds than staging during the past week.

I understand it's often not obvious that a change causes a mass rebuild, and that's why I consider it important to make it easier, via pushing detection into CI.

[vcunat]

Implemented ~by~ as @GrahamcOfBorg ❤️ by @grahamc.

[danbst]

I've tried to sum up required build time in rebuild-amount.sh, but stumbled on this - https://github.com/NixOS/hydra/issues/642

but even then, it is not clear how much built-time is too much. This depends on number of cores in build farm, right?