Nixpkgs: Agree on common high-level packaging guidelines

[copumpkin]

I think it would be nice to have a section in the nixpkgs manual describing how we like to think about packages at a fairly high level. Here are a few candidate ideas (possibly controversial):

1. One of the key goals of nixpkgs is to improve determinism: we strive to eliminate "works for me, dunno" as a response to help requests by minimizing the pieces of state that can affect the behavior of a program. Determinism not only helps users gain confidence in the system, but also helps developers better debug user errors.
2. Where possible, we like to lock down all versions of programs used. Often a package with a fully pure build can still pull in implicit dependencies at runtime from the PATH environment variable. For example, a script might look for a java VM in the path to run its jars with, or attempt to shell out to tar at runtime. Because those tool references are not "locked down" and could vary at runtime, a script using java on one machine might work (because perhaps it has JVM 1.8 in its PATH for other reasons) and it might fail on another machine (because nobody realize they had to nix-env install java before running it, or possibly because the jar was built with a newer version of bytecode than the JVM in path supports)
3. Satisfying #2 might not always be possible. Sometimes there are acceptable reasons to have runtime dependencies that aren't locked down at compile time, but if you think you have one, please call it out explicitly in a comment in your package expression so future maintainers (possibly trying to figure out a principled way to do runtime dependencies) can understand your reasoning and whether they should fix it. Of course, in some cases we simply don't realize that a package is shelling out to an unstated dependency because it's super common on everyone's machine, and there's no problem with that either.
4. Self-updating programs are generally unfriendly to nixpkgs. If you have a program that attempts to update itself, you'll probably need to patch it to stop doing that. Given the ease of updating individual programs "out of line" with the nixpkgs distribution, patches should preserve update notifications, but since the Nix store is in principle (and often in practice) read-only, the program will be unable to change its own program files.
5. "Vendored" dependencies are becoming more and more common in the world: many packages (especially in some ecosystems) will bundle their full dependency closure with their source distribution, often as a workaround for poor dependency management solutions, and in the Nix world we're faced with a dilemma: respect the exact versions of dependencies that the author bundled, or repoint the package to more standard "common" dependencies that we provide elsewhere in nixpkgs. Both answers unfortunately have pros and cons: using the vendored dependencies preserves the exact behavior that the author intended, but severely hinders distro-specific patches (e.g., perhaps one of those vendored packages references a hardcoded /usr/bin/tar and we need to patch that path every time it gets bundled with a package we're building; or worse, one of those vendored packages might have a security vulnerability that we don't catch when we patch the common package!); if we force packages to use common dependencies, then the versions might not match the author's wishes, but we have a much cleaner ecosystem where we have reasonable certainty about all the versions of packages that someone depends on, at the cost of higher packager maintenance. I think the right answer here will depend on which language ecosystem is being used, but in general I'd prefer avoiding vendored packages to better reason about dependencies in use in a system and share work across them.
6. Try to minimize the amount of not-specific-to-Nix content that goes into the nixpkgs repo: big diffs containing files from upstream just clutter the repo and make it harder to understand what belongs to us and what doesn't. A consequence of this is that many patches should be retrieved directly from upstream using fetchurl or fetchpatch, rather than bundling them in nixpkgs. Patch files in the nixpkgs repo should be mostly nix-specific packages that exist nowhere else. If for whatever reason (perhaps upstream provides no persistent link) you do need to bundle external content into our repository, please clearly mark its provenance either with a comment if the file format supports comments or in some other way if it doesn't.

There are probably several others of these that I've missed, but I figured I'd try to start a discussion by putting out some proposed principles. Once we've settled on some such principles, I'd be happy to submit a PR against the nixpkgs manual with whatever we agree on (or perhaps present that "there are two schools of thought" in some cases and actively discuss the pros and cons of each side to let newcomers decide where they fall)

cc @domenkozar @edolstra @peti @shlevy @vcunat @aszlig @7c6f434c @garbas as comparative old-timers who have been around far longer than I have and probably have far more useful opinions than I do 😄

[matthewbauer]

Another question I've had:

When should older versions of packages be left in nixpkgs?

There are some cases where there's enough compatability issues and other differences to keep it and it's pretty established practice that it's not just any old version update (gnome3 vs gnome2, python3 vs python2, gtk3 vs gtk2)... I would prefer nixpkgs to be up to date as possible (rolling release) and just use old nixpkgs trees for those special cases when you need older stuff. If you look in all-packages.nix, you'll see quite a few unused older versions. I hate to break stuff but we may need to have a special policy on what's "allowed" to have older duplicates and remove the rest.

[copumpkin]

@matthewbauer yes, thanks! I knew there was a major one I was forgetting. I'd definitely want to include a point about how we think about multiple versions.

I personally think the policy on older versions should be "keep them around as long as the latest version of something else needs them". We'll probably need better tooling to discover reverse build-time dependencies of course.

[peti]

I'm closing this issue because it seems like the discussion is stalled.