Nixpkgs: Standard gentoo-like "use" flags across nixpkgs

[copumpkin]

Before telling me nix already supports this, hear me out :smile:

Take a "headless" Nix (NixOS or otherwise) system. A whole load of packages have slightly different options to disable GUI components, but none of them are standardized, and thus we can end up with a completely headless server bundling a ton of graphics stuff because it depends for example on pinentry which optionally supports a GUI prompt and thus pulls in a ton of graphical crap.

What if we instituted a set of standard flags that would be available through stdenv that would help us determine those options? For example:

{ stdenv, fetchurl, mesa, gnome }:

stdenv.mkDerivation {
  buildInputs = stdenv.lib.optional (!stdenv.flags.headless) [ mesa gnome ];
}

And then our closures get small again. Individual packages could still keep ad-hoc flags in their parameter set, but this would help us standardize on a set of common flags of this sort.

Off the top of my head, I'd start with these common flags:

• headless: disable any GUI components
• hardened: use any additional package-specific hardening measures available (the stdenv would also apply some default hardening steps, but certain packages might need additional knowledge of this) [this is already on by default, now]
• monolingual: get rid of language translations (and default to whatever you specify if not english)

There are probably others, but I think we could get a lot of value out of just these two.

cc:

• @edolstra @shlevy @vcunat: general knowing of things
• @globin @thoughtpolice: probably care about the hardened flag

[copumpkin]

I guess this is basically the config field, but cast in a slightly different light, with a centrally determined schema.

[cleverca22]

one issue i have noticed is that while there is a config.pulseaudio to force it on/off, every app has its own default, so its a total mess until you set it, and once you do, half of them miss the binary cache

[copumpkin]

I'd like to keep the space of flags fairly small so that we can conceivably get Hydra to build common combinations of them. Some sort of audio enable/disable flag also sounds reasonable, but I might just lump it under headless to keep the combination set small.

[domenkozar]

I think we all agree on this one, we need to define an API and the rest will follow :)

[vcunat]

Well, just a single flag may double the number of jobs on Hydra, and I imagine the hardened flag would want to affect some mass-rebuild stuff such as glibc. But we could have a smaller jobset for the hardened config.

What's missing in the config field? Centrally defined defaults and perhaps centrally checked basic consistency (like nixos types)? Why not add such features to config instead?

[jagajaga]

I suppose there was a discussion somehow related to this question.

[jagajaga]

Ooops, here is a link: https://github.com/NixOS/nixpkgs/pull/6968

[wmertens]

Good idea! The current implementation of this is basically the profiles like https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/profiles/headless.nix , but having actual boolean values to check would indeed be better

[spacekitteh]

@grahamc policy discussion, security

[vcunat]

The hardening is on by default, as amended in the issue, so I don't see any security-related stuff remaining here.

[spacekitteh]

the security aspect comes from disabling unnecessary options. for example, a web server has no use for audio support or screen reader support. additionally, it allows the specifying of more-secure-yet-less-performing replacements, such as specifying constraints on allowable crypto algorithms and implementations.

[vcunat]

Side note: if we do have such extreme cases, I think we should remove those options from the default config (and perhaps add a packageFull version as we have in a few cases already).

[spacekitteh]

Reguarding the actual proposal, though, I am highly in favour of it. I think with the purely functional approach we can avoid most of the pitfalls of Gentoo's implementation (I spent a week trying to sort out a broken gentoo install due to conflicting flags, which happened because I performed installs in the wrong order)

If we put some thought into our flags and their defaults, we could probably have a large range of them yet not blow up hydra - one major thing to think about is sets of related flags (e.g. a "hardened" set) which sets multiple individual flags. Then hydra could just build the flagset defaults, and let tinkerers build locally.

[oxij]

This is exactly what the original version of #6968 did with
nixpkgs.config.common option (see several of my first comments in that
PR), but those changes were cut as unorthodox and the PR was trivialized
to the point of doing nothing in that regard.

However, I still rebase those original changes on top of master since I
can't live without them (they bitrotted considerably after two years of
rebases, though).

Personally, I do some crazy stuff with nixpkgs.config, I turned it
into something like "NixOS options", pretty much. But even old mild
version of that was rejected as unorthodox.

Still, I think I can extract something agreeable for everyone from my
personal branch

• if we can agree that nixpkgs.config option is useful and not
  actually deprecated,

• we force *Support and with* package options defaults into package
  definitions (this is the part I constantly fix on rebasing)

~~
pkg = callPackage ./something.nix {
someSupport = true;
};
~~

breaks the thing. That someSupport need to be in ./something.nix,
not in all-packages.nix (it's also cleaner that way, IMO).

• (optionally) we agree to rename all *Support options into with* or
  vice versa, or you'd have

~~
nixpkgs.config.common = {
x11Support = false;
withX = false;
withX11 = false;
};
~~

in your configuration.nix, which is meh.

What I do in my branch to achieve this is simple: I have callPackages
look at nixpkgs.config.common and apply those options to everything
(my version does much more, including several different callPackages,
but we shall ignore that for now).

But that change alone is useless since defaults in all-packages.nix
will override the common options. The change needed to make it
actually work for all packages is huge.

I'm not extracting it from my wizardry (it's like a day of work, if not
more) until people say they sure they want it.

[MarcWeber]

http://lists.science.uu.nl/pipermail/nix-dev/2017-March/023248.html
Well -> having flags doesn't mean all combinations have to be provided as binaries by Hydra or that all combinations must be tested. A warning line could be enough. Having a sink to express the need and setup such flags (even without providing binaries) could be helpful. Flags can also be commented

[copumpkin]

Yeah, as long as people don't go adding them willy-nilly for tiny package-specific flags that's probably fine. I'd want there to be a bit of governance around what makes sense to go into a flag. It also gets interesting when you consider that we support more than one platform.

[spacekitteh]

Yeah, single-package-specific flags don't make sense as global flags. On the other hand, things like "headless" still have to be implemented carefully - e.g. openjdk needs some AWT components, even in headless mode, in order to support generating images, and currently the headless flag there overlooks that.

[twhitehead]

You could technically add common flags to the top level callPackage namespace (see _pkgs/top-level/splice.nix_). This way they would automatically become available to satisfy the arguments of all packages.

I think the real key, however, would be to take the NixOS option declaration and checking machinery and apply it to defining and declaring these options. This way they would be documented, defaulted, checked, and ideally automatically extracted and made available on a website to lookup as with NixOS options.

Imagine if the top-level config attribute was not just sucked in blindly from _~/.config/nixpkgs/config.nix_, but rather generated from this file in the manner of the NixOS machine configuration. That is, verified against a declaration of valid options, which itself is declared in a modular manner.

Packages could also have the option of extending the config with package specific options by providing an optional options attribute (or perhaps meta.options?). This would functions in the same manner as the NixOS modules options attribute. So, for example, a config.emacs entry _~/.config/nixpkgs/config.nix_ would be matched up and checked against an emacs.options declaration.

[twhitehead]

One way to test this would be to extend the current nixpkgs NixOS module to provide a verified config attribute set to nixpkgs (i.e., one constructed using the NixOS option machinery) instead of just the current anything goes nixpkgs.config option.

Ultimately you could imagine nixpkgs itself being changed to not default the config attribute to just importing _~/.nixpkgs/config.nix_, but rather to use the NixOS module system to treat _~/.config/nixpkgs/config,nix_ as a module config declaration for nixpkgs.config to be merged with the system declared one from _/etc/nixos/configuration.nix_ to generate the actual config attribute in a verified manner.

Note also how this is nicely orthogonal to the use of overlaying and overriding which provide a nice modular way for the user to actually make changes to nixpkgs itself without having to fork it.

[oxij]

Anyone still interested in this, please see https://github.com/NixOS/nixpkgs/issues/36229#issuecomment-370180008
It describes how SLNOS implements this feature (and gives some social commentary on related issues).

[oxij]

A demo implementation #56110 with some discussion of what a proper implementation will take.

[Ekleog]

(triage) I think this will require an actual RFC in order to move forward. Leaving open until someone actually writes it, though, so this issue can track the need to write down the RFC.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[Pitometsu]

For me it's important.

[wmertens]

I agree, this is important, and as @Ekleog said this should be an RFC with a single proposal for the way forward. Any champions amongst our midst?