Nixpkgs: Future of package updaters in Nixpkgs

[matthewbauer]

Issue description

For the last few months, we've had @ryantm's nixpkgs-update tooling. IMO it has been very successful and has lead to many more packages being up to date than in the past. Can we make it the official package updater of Nixpkgs? That is, will this be the one true automated updater going forward. Likewise, is it reasonable to start removing some of the older ways of doing this still in Nixpkgs, like:

• common-updater
• upstream-updater
• maintainers/scripts/update.nix

I tend to think this tooling is better kept and maintained in a separate repo, but perhaps not all do. Some of the above may still have legitimate use cases that nixpkgs-update doesn't handle. Feel free to strongly disagree with this! It might also be worth integrating these into nixpkgs-update .

Pinging current contributors to these updaters to see if they are still useful:

/cc @7c6f434c @jtojnar @garbas @dezgeg @taku0

[CMCDragonkai]

I think the @ryantm package updater is awesome. I was wondering if there should be some documentation on how it works, and what are the constraints of the system, how much update intelligence does it have?

[jtojnar]

GNOME, Elementary and Deepin (@romildo) use maintainers/scripts/update.nix, since their package collection often needs to be updated in lockstep (packages need new glib, gtk, libdazzle, etc.) and simple upgrade is usually not possible outside of patch versions (new features require new libraries or backported patches need to be removed). I guess we could switch to nixpkgs-updater but we would need support for:

• updating only packages under a certain attribute path
• updating the file even if the build fails (the point of update.nix is to automate the tedious part of updating: finding out the latest version, changing the hashes, printing the changelog committing)
• obtaining changelogs (I have a hack that dumps gnome changelogs in update.nix, something cleaner would be nice)
• updating multiple lockstep packages in a single branch
• obtaining the list of versions from the vendor (we do not want to wait for Arch or someone to update)
• having a more convenient way to run it (e.g. nix run nixpkgs-update -c nixpkgs-update --update gnome3)

[7c6f434c]

Do I understand correctly that nixpkgs-update tracks Repology, i.e. never allows us to notice and update before some other repo does?

In that case update.nix seems to be a useful additional part of a complete solution…

[jtojnar]

Do I understand correctly that nixpkgs-update tracks Repology, i.e. never allows us to notice and update before some other repo does?

Yes, but adding more backends is something we need to do anyway, if we want changelog support. Repology does not track changes.

[ryantm]

Thanks for the support.

I agree with everything @jtojnar says. Hopefully we can support all those features someday in nixpkgs-update.

Do I understand correctly that nixpkgs-update tracks Repology, i.e. never allows us to notice and update before some other repo does?

Yes. Technically, the only "backend" is an input file that has lines with the format package-name old-version new-version. When I run nixpkgs-update with r-ryantm, I always use an input file that I generate using information from Repology's API, and various PR/commit messages assume the input info came from Repology.

As for anything concrete we can do about this issue raised by @matthewbauer, I'd be fine with moving nixpkgs-update into the NixOS team, if people want it. Currently, @jtojnar is a co-maintainer of the nixpkgs-update repo, so if I get hit by a bus, it won't be totally unavailable.

I'm not aware of if there is a general licensing policy for projects under the NixOS team. Currently the nixpkgs-update repo is in the public domain.

[7c6f434c]

I think most of the vendor-site-scraping updaters inside Nixpkgs contain more code for version retrieval than for actual updates; so maybe they could just lose their expression-editing logic so the build-checks and GitHub-communication could be unified.

I wonder if adding an obligatory «backend» and optional «changelog» entry is a cheap change which could be used as an excuse to announce that backends are welcome.

[taku0]

Building, checking, and PRing features are attractive to me. I still need custom version checker and updater because those packages I update consist with multiple variants; firefox have standard version and ESR version, each locale of firefox-bin have its own archive file, flashplayer have standard and debug variants for x86-64 and i686 architectures, and adoptopenjdk have HotSpot and OpenJ9 variants for various OSs and architectures.

[FRidh]

https://github.com/NixOS/nixpkgs/pull/50983 gives all Python packages an updateScript attribute. I'm considering also taking into account dependencies using https://github.com/costrouc/nixpkgs-dev-tools/blob/master/python/python-package-init.py

For me it's at least important that the updater can also be run separately because I need to be able to pass in additional arguments.

[Synthetica9]

I think nixpkgs-update (or some other bot) should also have the capability to run scripts with a defined interval. There are a lot of packages that have update scripts (Spotify and Steam come to mind).

Maybe nixpkgs-update could have both backends (to determine which packages need updating, like repology, rss, github releases, etc) and front-ends (to do the actual updates. This would include the current updater and update-scripts)

Preferably both of these would be configurable from within the package definition (probably from the meta field?).

[Chiiruno]

This would be a good change I think, however I'm worried about 2 things:

1. How can we guarantee that we won't update a package that has breaking changes to other packages that depend on it? It's going to be a problem if there's a major breaking revision from say, GTK. Now, that probably won't happen since (most) developers keep backwards compatibility and a stable API in mind, but I'd rather not rely on trust. This goes moreso for ABI changes with GCC and LLVM, I sure don't want gcrypt to start having problems, because then I won't be able to boot into my system. Perhaps only update on minor revision? Have it package-configurable?
2. How huge of a refactor in nixpkgs in total will this be? I'd imagine you'd need to list all packages that need a refactor and put together a project in the repo so contributers can submits PRs for it.

Overall though, I think it's worth the effort, I love NixOS, but I find the version bump process a little tedious to always do, especially when it can be done automatically.

Ah, something else I'm wondering is will it handle git-based packages? I'd love to see meguca and hydron automatically updated. Which brings another point, will this handle regenerating stuff like node-packages periodically?

And even further now that I think about it, how will failed updates be handled? Retry once a day?
I think only trying once and never trying again would be bad, since say the updater might update meguca first, but not nodePackages: meguca, then it might fail meguca.

[CMCDragonkai]

The only way to guarantee compatibility is with automated tests.

[matthewbauer]

@jtojnar has a good pr for using commits with update.nix:

https://github.com/NixOS/nixpkgs/pull/59372

Otherwise, I'm going to close this for now, not a whole lot can be done without some sort of RFC.