Nixpkgs: Unfree software is unintuitively hidden for new users

[grahamc]

Issue description

New users try to install unfree software, search for it with nix-env -qaP, and find it missing. Some more enterprising users search nixpkgs on GitHub and find it is _definitely there_ but don't know how to find it.

Example chat where this happened: https://botbot.me/freenode/nixos/2016-07-20/?msg=69976623&page=9

Perhaps a way to improve this would be show the unfree software in the list, but with a note about it being unfree.

Steps to reproduce

1. Be a new user who is evaluating NixOS/Nixpkgs
2. Try to install hipchat
3. Find it isn't in nix-env's list
4. Confirm it is in nixpkgs via GitHub
5. ??? until they find { allowUnfree = true; } or ask in IRC

[grahamc]

re:

• https://github.com/NixOS/nixpkgs/pull/5628
• https://github.com/NixOS/nix/issues/432

I think this should be revisited. I think we're doing users a disservice by making it difficult to find the software they're wanting to choose. Right now we're choosing to appease strict FOSS users over reducing the headache for newcomers.

I'd rather the onus on more advanced users to get rid of it, than new users to get started with the tools they need and are familiar with.

[copumpkin]

Agree. I think this is largely an incidental consequence of not wanting Hydra to do unfree stuff, but I think the mechanism should be a made a bit more fine-grained to be friendlier to users.

[vcunat]

I tried, and I think it's hard to find strong consensus on this point.

Personally I'm comfortable that official nixpkgs accepts significantly wider range of packages than it's usual for most other large distributions. BTW, we're slowly shifting to specifying packages directly by attribute names, and in that case the user gets a useful evaluation error.

[grahamc]

BTW, we're slowly shifting to specifying packages directly by attribute names, and in that case the user gets a useful evaluation error.

Yeah, this error is really helpful if they get far enough to try to install it that way. A bit tough, though, to get that far when a nix-env search tells you it isn't there.

[abbradar]

I think we really do need to show the user somehow that there _are_ other packages. I've taken a liking to this suggestion, option II. Other solution is to show such packages but place big red label "UNFREE" next to them.

EDIT: after some thoughts I like my solution less, because this heavily depends on what do we consider "promotion". The other suggestion, however, is very unobtrusive IMHO and doesn't even name anything, just says that there are matches.

[PlasmaShift]

Put the allow unfree information where people go and look at how to install software

When they generate your Nixos configuration and have kde and wget " I think that is the one package they show as an example to install" commented out put allow unfree in a comment there as well

When they look at the manual on how to install packages put a note somewhere really close so they see it. Probably at two places, one for nix "nix-env -i ... or nix-env -qa ..." and one for Nixos "inside your configuration.nix

when they search on Nixos.org packages in the description make a note, ex nonfree -- rest of description
in package name or attribute name it would would get in the way of normal operations. I am not concerned about nonfree when I am greping around nix-env -qa

These are the places I would go/went to find out how to install applications
This way would also not interfere with the nix way as describing your os

As a side note it would also be nice if when you generated configuraiton.nix you would get a stateless user example.

[peti]

The real issue is that nix-env -qa searches are fundamentally broken and don't find literally thousands of useful packages that we have. This phenomenon is not limited to unfree software.

[domenkozar]

This came up again today on irc:

22:50:36           tosmi | hi, i'm a nixos newbie and unable to install the citrix-receiver package   
22:51:09           tosmi | nix-env -qa | grep doesn't find citrix-receiver               
22:51:34           tosmi | i suppose because it depends on an EULA protected blob...                                                                  

[domenkozar]

I think Nix UI redesign is a good opportunity to fix this.

[grahamc]

Today a user on IRC (ToxicFrog) spent four hours trying to package Sublime3 only to find out it was hidden behind unfree.

[grahamc]

Today another user on IRC (riptawr) spent quite some time trying to figure out discrepancies between nixos-rebuild (as root) and nix-env (as his user) due to his profile not having allowUnfree.

[vcunat]

It's very common that people are confused from nix-env not using nixpkgs config from nixos config. I put it into a separate *.nix file and import in both; I haven't seen a better approach yet – maybe it could be suggested in the docs. (But it seems a digression from the original topic.)

[Eilie]

Hello, I'm a new NixOs user and I've spent an hour trying to figure that out and then got help on IRC. Would be great to have this resolved in some intuitive way.

[alecmev]

Just spent 30~60 minutes trying to figure out why I can't find google-chrome via nix-env -qa, even though it's clearly in the repo, until I've been pointed to this issue by kind people in #nixos. Maybe add a quick note about allowUnfree somewhere high up in the manual, at least in Chapter 2 and Chapter 8, until a warning is implemented? ~Also, I don't see allowUnfree documented anywhere, it juts shows up in random issue descriptions and config excerpts~ (e.g. this wasn't obvious, spent extra 15 minutes on that).

[vcunat]

For reference, the documentation is in nixpkgs manual.

[vcunat]

Also note that restriction of unfree stuff is primarily a "political decision" IMHO. We've had lots of discussions, e.g. on https://github.com/NixOS/nixpkgs/pull/5628.

[7c6f434c]

I think the discussion right now is not as much a plea to reverse that
decision as to declare it more visibly. Proudly declaring this decision
is not against this decision, and those who want the override will know
what to look for.

For the record: I support requiring an explicit action for unfree
packages, and I do have some unfree packages installed. I also think
that mentioning this filter close to the beginning of NixPkgs manual is
a good idea.

[ToxicFrog]

My preference would be for package search (both the website and nix-env -qa) to list nonfree packages by default, but display some kind of indication that they're nonfree. nixos-homepage PR 120 does this for the website, but is blocked on license information being included in the search page. (I noted on that PR that this is the approach taken by other free-software distros such as Debian.)

Failing that, there should at least be (by default) a prominent warning that non-free software isn't being included in the search; a large part of the problem here is that it's not at all evident to the user that their query isn't actually finding everything. An option to enable it would also be nice (manually searching the nixpkgs repo is pretty awful UX), but making sure the user knows that they need to do that at all is a necessary first step.

I wasted a lot of hours trying to repackage nonfree software like Steam, Chrome and Sublime3 before someone on IRC pointed out that the package search was lying to me and I needed to instead search the nixpkgs repo directly. Someone else repeats this experience on a regular basis, as evinced by this very thread.

[vcunat]

Suggestions on showing unfree packages in searches by default are consistently disliked by some core devs (Eelco, Rob, 7c6f434c above, maybe more), so I don't feel like pushing that point anymore.

I like the approach of mentioning the fact that they're hidden on some (more) prominent place. By itself that would seem more like propagating "free" SW than "unfree". However, I don't have a good idea about what parts of docs/website are best visible to newbies.

[ToxicFrog]

For my part I tended to lean heavily on https://nixos.org/nixos/packages.html

Even if they aren't shown by default there should be some way to enable them rather than just shrugging and going "use github search then".

[7c6f434c]

I think all the way that allow installation ($NIXPKGS_ALLOW_UNFREE and Nixpkgs config) enable search, too.

[ToxicFrog]

Good point. That works for people using the command line, and in that case I think the suggestion on #5628 of including allowUnfree = false in the autogenerated templates, with comments, will help a lot with making people aware that the option exists.

That said, the website is both a nicer UI and much faster than nix-env -qa even on a fast machine; it shouldn't be a second class citizen in what it allows you to search for.

Perhaps the answer is just to add "include unfree software" and "include broken packages" checkboxes next to the search box, and default both to off?

[7c6f434c]

I wonder if regardless of our current discussion it is a good idea to make it easy to generate a JavaScript-only local-HTML-file package list with search. Of course, it will follow the brokenness/insecurity/lack-of-freedom choices set by the user.

The problem with package list on the site is that it is a list of Hydra jobs. Of course, Hydra won't build things where it is unclear whether they are OK to distribute.

[jcrben]

Perhaps an interactive prompt when installing could be a compromise? Also spent some time trying to figure out what was going on before hopping on IRC. Not all beginners are up to ask on IRC. Note also that this is documented in the nixpkgs manual but not, as far as I can tell, in the nix manual, which is where I started...

[7c6f434c]

Technically, this is not a Nix-level decision, so it is not in the Nix manual — the license attribute lives in NixPkgs, the license check is performed by code in NixPkgs repository without any special support from Nix, etc.

I somewhat like being able to use NixPkgs for free software with no NixPkgs config at all, so I can understand unwillingness to put warnings unless configured.

Maybe enumerate the natural places like NixPkgs package search where adding word «free» and a footnote «unfree packages need to be explicitly enabled and are never built on Hydra» would make sense?

[MaikKlein]

It was really confusing for me.

I know that I enabled nixpkgs.config.allowUnfree = true; in my configuartion.nix but I was unable to find packages with nix-env -qa.

I was almost convinced that nixos was broken. Also allowUnfree is not in https://nixos.org/nixos/options.html

[domenkozar]

Another instance: https://www.reddit.com/r/NixOS/comments/6mtt5d/how_to_install_atom_on_mac/dk4gy3v/

[butterflya]

In this message I discuss the causes of this problem and some possible solutions.

Causes

There are a few causes of this problem.

• The Nix manual is sub-optimal:
  There are at least 80 occurrences of the word "available" in the Nix manual, but there is no good definition of what "available" means. Additionally, the word available is interpreted by the author to mean something very specific; a logical relationship involving unfree. This meaning will never be the one a first time reader will be able to comprehend, especially considering that the word unfree is not even listed in the Nix manual.

The Nixpkgs manual does contain this information, but that makes it appear as if Nix and Nixpkgs are written by two independent entities, which have never even heard of each other, but in fact the manual is written largely by the same person.

The various manuals need to be heavily cross-linked and intra-linked to make them more useful. In at least 20 places I would expect the word "available" to be hyperlinked.

• A minority wants to impose their political agenda onto others by making it more difficult to install proprietary software and claim that a system which by default is not fully operational is better.

Solutions

• A solution that probably everyone could live with would be modifying the installer to generate a configuration.nix file containing by default all the code for controlling free/non-free code as in the manual in a commented block with additionally a big warning explaining the user what it means, especially in relation to the behaviour of nix-env -qaP. There are ways to enforce that the user has actually read this information, if this wouldn't already solve the problem. For systems other than NixOS, this could be output on stdout.

• Another solution which would cover 99% of the cases is to identify popular packages and import those string constants into nix-env -qaP and inform the user that enabling unfree is required via the well-known snippet. This would cost 5KB in the binary size, while saving a latency of seconds (and possibly an expensive search of minutes to hours with the worst case being deciding that NixOS is not worth the trouble).

• Yet another solution is to print to stderr a warning that nix-env -qaP output is incomplete when unfree software is disabled while introducing another option to disable such warnings.

• Finally, nix-env -qaP could also just search on nixpkgs as a fallback code path and tell the user the exact same response as was given via IRC. The latency of this would be slightly higher, but considering that reading a manual and finding the right section for a user will take at least 1 minute, I expect that a computer is faster in operation in most cases. Additionally, nix-env could output to the user that the quick path failed, and that it will now go to a slower mode in order to help the user as a service to the user.

I recommend to implement all solutions until the number of questions on IRC about this topic goes to zero.

[ToxicFrog]

@butterflya: This should also be fixed on the nixos.org package search (which a lot of people, including me, use in preference to nix-env -q because it is much faster). There's some thoughts on this earlier in the thread, but maybe it should be split out into a different issue -- there's basically three underlying problems there:

• the package list on the website is generated from a build without nonfree software enabled
• the package list doesn't expose the free-ness of software anywhere, and
• the package list has no way to filter by free-ness

I suspect that fixing 2 and 3 on that list are prerequisites for not having fixes to 1 rejected.

E: oh hey, there are existing bugs for this on the nixos-homepage repo that I already mentioned earlier in this thread and then forgot about. Whoops.

[butterflya]

@ToxicFrog Don't you think it's a little bit weird that on NixOS there is this great module system and they could just have implemented the exact same service behind the nixos.org package search on NixOS and even enabled by default such that it would be indexed during idle time?

The only reason nixos.org package search exists, is because they want to spy on you.

[copumpkin]

The only reason nixos.org package search exists, is because they want to spy on you.

Come on man, I agree with you that these packages shouldn't be hidden by default, but stuff like this just makes it harder to take anything else you say seriously. This is a FOSS project full of folks working on it in their free time. We're not a shady cabal with ulterior motives. The search is there because someone felt like implementing it because they thought it would help the project.

[butterflya]

@copumpkin Point me to a privacy policy claiming that all search queries go to /dev/null and I might start to take you seriously (if you like to do ad hominem arguments, two can play that game).

[copumpkin]

Sigh. Look at the source to https://nixos.org/nixos/packages.html and tell me what you see. Note that it's just grabbing a gzipped tarball of JSON and processing it on the client side. The server never sees individual queries. The source to that page is here.

Granted, there's no guarantee that the deployed webpage and the underlying source code will stay in sync, or that the html won't silently change to record your inputs, but that would be work and frankly I doubt anyone in this project gives a damn about what packages folks search for. We should probably get a privacy policy someday, but accusing the project of spying on you when it's trivial to see that it isn't seems silly.

[ToxicFrog]

@butterflya The search queries are never sent to the server, but are executed entirely in the browser using a package database downloaded when the page first loads. You can trivially verify this for yourself with any combination of source inspection, browser debug tools, and packet capture that you care to use. The NixOS option search page works the same way.

[U007D]

This is me. Right now. After a couple of hours I did stumble across nixpkgs.config.allowUnfree = true;, but even after adding it to my configuration.nix, I still cannot get nix-env to show me all packages (eg. google-chrome is nowhere to be found), and I still don't know why (even after reading this thread, unfortunately).

I can't claim to understand why "the FOSS contingent" would object to unfree software being shown under an Unfree heading. I'll just accept that this is a thing.

What would help me right now, and users like me in the future is for the tool to behave as it does, with a note (make it silenceable if the experts don't want to see the reminder) stating that unfree software is not being shown and the step(s) required to show them. No kidding--I have read through most of the chapters of the manual (everything through ch. 12, and browsed the rest) and see no mention of how to list all (including unfree) packages.

Do a find in your browser on https://nixos.org/nixos/manual/index.html right now, and you'll find there is no mention of allowUnfree anywhere on the page!

I'm 3 hours into my quest now, and still without a solution. NixOS is a great idea, but easily fixable issues like this make the initial experience tough to stick with. Lucky I'm stubborn, I guess... ;)

[U007D]

Solved. For anyone else running into this problem:

@butterflya's description of possible solutions mentions a 2nd manual, namely the nixpkgs manual (https://nixos.org/nixpkgs/manual/), which you may not have known about (I did not).

Unlike the nixos manual (https://nixos.org/nixos/manual/index.html), which does not mention allowUnfree, nixpkgs manual does explain how to get this working. If you're like me, you've had a long search just to get here, so here is the relevant part:

Nix comes with certain defaults about what packages can and cannot be installed, based on a package's metadata. By default, Nix will prevent installation if any of the following criteria are true:

The package is thought to be broken, and has had its meta.broken set to true.

The package's meta.license is set to a license which is considered to be unfree.

The package has known security vulnerabilities but has not or can not be updated for some reason, and a list of issues has been entered in to the package's meta.knownVulnerabilities.

Note that all this is checked during evaluation already, and the check includes any package that is evaluated. In particular, all build-time dependencies are checked. nix-env -qa will (attempt to) hide any packages that would be refused.

Each of these criteria can be altered in the nixpkgs configuration.

The nixpkgs configuration for a NixOS system is set in the configuration.nix, as in the following example:

{
 nixpkgs.config = {
   allowUnfree = true;
 };
}

However, this does not allow unfree software for individual users. Their configurations are managed separately.

A user's of nixpkgs configuration is stored in a user-specific configuration file located at ~/.config/nixpkgs/config.nix. For example:

{
 allowUnfree = true;
}

(Note: you may have to create the ~/.config/nixpkgs folder).

After making both of the above changes, listing available packages (nix-env -qaP '*' --description) will now include unfree packages.

I hope that is helpful for someone.

[zoosky]

Great. I was in this journey too a few months ago.

[LisannaAtHome]

Another one: https://botbot.me/freenode/nixos/2017-12-27/?msg=95086544&page=3

[lboklin]

To add to this; for one-offs, before you go enabling unfree packages system- or user-wide, I recommend simply setting the NIXPKGS_ALLOW_UNFREE environment variable to 1, like so: NIXPKGS_ALLOW_UNFREE=1 nix-env -iA nixpkgs.steam (or export NIXPKGS_ALLOW_UNFREE=1 for persistence within the shell session).

[bbigras]

I'm one of the confused new users. This morning I saw "Nix 2.0 Released" on various websites and I wanted to give NixOS a try.

I wanted to install dropbox. It wasn't showing up with nix-env -qaP '*' --description nor on https://nixos.org/nixos/packages.html but weirdly I found https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/networking/dropbox/default.nix using google. I was wondering if it wasn't available on the stable channel or something like that and I finally saw license = licenses.unfree;. I ended up here after another google search.

Thanks to @U007D's comment I was able to figure out how to use { allowUnfree = true; }.

[alexfmpe]

Just wasted some time over this.
I don't understand why the "unfree" message only shows when using attribute names.
I get the "don't promote unfree" policy, but would we be violating it by switching the message in these cases from error: selector 'whatever' matches no derivations
to error: selector 'whatever' matches only unfree derivations ?

[copumpkin]

@alexfmpe I think there are two issues at play here that are easy to confuse in this issue:

1. Hydra doesn't want to build unfree software (or host it on cache.nixos.org)
2. Some folks in the community don't want to encourage use of unfree software

I believe the current unfree solution was added initially to address point 1, and then it incidentally also covered point 2 and people didn't care enough to change that. But I still don't think point 2 is a primary goal, and if we can come up with a sensible mechanism to ensure that Hydra doesn't have to worry about distributing unfriendly licenses, I don't imagine most of the community would complain about the lack of point 2 (or making point 2 an opt-in version, since most users don't feel that strongly about licenses and just want software to work).

So in short, the unfree-blocking mechanism for Hydra could certainly be improved, but someone would need to do it.

[7c6f434c]

@copumpkin what you say is implementable by just setting allowUnfree to true by default and overriding it on Hydra. Of course, with multiple ways to set it needed for different operations, it will destroy option 2, not just make it opt-in — and this would be bad.

I hope that the new nix UI wouldn't have any operations that care about the derivation names as opposed to attribute names.

For search reasons — maybe we should permit evaluating name in addition to meta. Will it be enough?

[cx405]

I am new to nixos (just few months, but using it as main system now), and this is how to search for unfree packages:

nix-env -qaP --arg config '{ allowUnfree = true; }' | grep <package_name>

This shows all occurrences and lists packages in all accessible channels.

[cdyson37]

Just had exactly the same experience as the other n00bs. Couldn't find it with nix-env, couldn't find it with the online search, but stumbled across it in github. And now I'm here :)

Perhaps a stopgap solution would be to ask the user when running the magic install script whether allowUnfree should be set?

[cx405]

@cdyson37 If you remove the "--arg config {}" option in my post above, it will work normally. So this feature is already implemented, just not behind an extra trigger. There is new "nix" interface that supports search.

I assume the best way is to create the built-in option, which would in the end - add the option that I provided above. So, in a curtain way, - you can already use it today.

[lexleogryfon]

I`m new user, and prefer to use https://nixos.org/nixos/packages.html over nix-env for package search, because of beautiful UI.
I've spent months with thought that there is no good commercial packages packages like vscode, sublime on nixpkgs. That makes impression that nixpkgs less mature and provide less choices than Arch AUR, or Ubuntu.
Recently accidentally discovered that mentioned packages are available as unfree.
Please let newcomers know about all available packages in https://nixos.org/nixos/packages.html and merge https://github.com/NixOS/nixos-homepage/pull/120

An absence of packages in https://nixos.org/nixos/packages.html may prevent potential user from switching to NixOs.

It is 2018 already, case - example of bad user experience, issue doesn't have technical obstacles, how it can't be solved so long only because of FOSS zealotry?

[ThomasTheGerman]

I just had the same problem trying to install steam, just tried installing steam-run on a whim, and got a beautiful error message that dependancies are unfree and how to enable unfree packages.

This is especially stupid from my side since there's a FAQ-Entry for this...

[GlennS]

I've been using Nix for about a year and a half now, and just encountered this problem while trying to install VSCode. It wasted probably half an hour to an hour of my time.

I'd really appreciate it if it was mentioned in the NixOS manual.

[goertzenator]

This issue keeps biting me even though I've been running Nixos a few months now. It is frustrating to have to deal with this in so many different contexts:

• Nixos config file
• root user environment
• my regular user environment
• package search on the web

I wish there was a global switch at install time that could cut through all this friction (web search is separate of course).

[samueldr]

Spotted in the wild:

• https://old.reddit.com/r/NixOS/comments/8z87pe/nixos_on_prgmr_and_failing_to_learn_nix_pushcx/e2hrqq8/?context=3
  
  
  
  • https://push.cx/2018/nixos
  
  

[philipcristiano]

Just switched over to NixOS and ran into this issue. The original steps to reproduce are what I went through, but also went trying to use unstable thinking that the packages might be new.

It would have helped me if the packages came up when searching, maybe when trying to install get a message saying it's unfree and that I need to enable unfree to get it installed.

Eventually reached step 5, and had to ask someone in Slack.

[Ekleog]

(triage) @samueldr, ISTR you were rewriting the package search page (hopefully not mistaking with someone else), which would be a first good place to have unfree packages listed. In addition, the screen real estate could include a big warning sign for unfree packages, so that ideological decisions wouldn't have to come into play. Does your prototype include that? Did it make any progress?

[samueldr]

@Ekleog though its data set is severely stale, here's the re-implementation:

• https://nix.samueldr.com/explorer/?channel=nixos-18.03
• https://github.com/NixOS/nixos-homepage/pull/209

Though I haven't felt there was strong interest into merging this, no response to the feature, no acknowledgement. Let's allow 18.09 some time to crystallize and stabilize, nixcon to pass, and then maybe a stab at dusting off the ol' js-hands routine to spit shine it and rattle it some more in front of the crowd.

[pvgoran]

I was just hit by this. I spent quite some time trying to figure out how to enable ImageMagick to use the Arial font required by my software (which was a task by itself, because Linux font management still looks like purple magic to me).

Eventually I found out that on my Gentoo system (which works with Arial just fine) it's provided by corefonts package, but I failed to find it in NixOS by looking at https://nixos.org/nixos/packages.html, and only discovered it by a series of searches on github and in a local nixpkgs repository.

On top of this, https://nixos.org/nixos/options.html doesn't list nixpkgs.config.allowUnfree.

[tbenst]

Just got confused by this. Already have nixpkgs.config.allowUnfree = true; in configuration.nix and in (rycee) home.nix but also need it in config.nix. Perhaps we only need one opt-in in configuration.nix?

[AMDphreak]

I have an answer for this. Didn't realize this was already a thread. Here's my answer, with proof that allowUnfree should be true by default, on top of the great arguments here.

Set allowUnfree = true by default and change name #55616

As for this response:

I tried, and I think it's hard to find strong consensus on this point.

You're not going to get consensus when those who disagree are disagreeing for highly individualized reasons. Their reasons are inherently incorrect for use in a policy. Policy should be approached from the point of view of the person you are applying the policy to, not the person making the policy. It's retarded to do it any other way.

[teto]

There is a rational for disallowing by default unfree software. The issue is about discoverability where there is clearly room for improvement. Heck even for free software nix search won't return all results (like in haskellPackagesetc) nix search haskellPackages.greenclip or nix search greenclip returns nothing while the program is in nixpkgs.

[7c6f434c]

disagreeing for highly individualized reasons

… which happen to be well-aligned between those disagreeing. Maybe they are not so individualised.

As for approaches to policy — running nix-serve on an externally accessible machine is probably illegal if you have some of the unfree packages installed (because that would be distributing modified versions); server users are a large fraction of developers of Nixpkgs (maybe even a majority depending on how you count), and probably a significant fraction of all users of Nixpkgs. «Using the full range of Nix tooling is a legal liability for our target market» is _also_ not a good policy.

[AMDphreak]

There is a rational for disallowing by default unfree software. The issue is about discoverability where there is clearly room for improvement. Heck even for free software nix search won't return all results (like in haskellPackagesetc) nix search haskellPackages.greenclip or nix search greenclip returns nothing while the program is in nixpkgs.

@teto I take it there is a typo in your message. You contradict yourself. True: the issue is about discoverability. False: there is a rational[e] for disallowing by default unfree software.

Free software is a subset of all software. If the issue is about discoverability, then you would by default allow users to access all software.

Free software ∈ all software ∴ make all software default.
The main issues is selectivity. Open doors for your users and let them choose whether to be specific later on. There is zero harm in letting them have access to unfree software. Sure some people would like for people not to use unfree software, but that is not their call. They have to work on making free software an attractive option, not punishing people for needing unfree software. It's basic operant conditioning from psychology. You're not going to get more people to use NixOS if you make installing the OS a friggin obstacle course.

[teto]

If the issue is about discoverability, then you would by default allow users to access all software.

To put things differently, Nixos could do a better job at showing the user that "yes it is possible to install this free software with this extra step". The extra step is the difference I make between "discoverabililty" and "allowing by default unfree software" (usability ?).
I don't know where the rationale for the decision to hide free software is but it's easy to imagine the (good) reasons why. Anyway I doubt you can sway this default.

[AMDphreak]

@teto: I did not say to hide free software by default. Not sure if you're confused or those are typos.

The default should be to allow users to access all software, since something they want could require it. The distinction between free and nonFree is only a problem for users who ban nonFree on the principle of:

1. Philosophy, or
2. Security (their own personal machine)

NixOS should not "do a better job" of making this little annoying detail more annoying for new users. They need to get rid of it. Apply the principles of design that Dieter Rams used: "Less but better". Users who are concerned about the debate between nonFree and Free software will GO LOOKING for this setting. New users don't care about this topic and shouldn't be required to think about it.

[puffnfresh]

@AMDphreak it's not about security nor philosophy. There can be legal issues with distributing non-free software.

For example, our CI system Hydra evaluates all packages and the caches them. This would be problematic for non-free packages. The default is to be safe for use cases such as this.

[7c6f434c]

@AMDphreak replying here to messages from both threads to make this detailed explanation a bit more findable.

Set-theoretical arguments

It is easy to misread this argument to imply that packaging malware (including a variety password-collecting and exfiltrating payloads, not just the general penetration-testing-optimised exploits that we sometimes do package) in the main repository is a good idea. I won't try to engage this part of your arguments because apparently I am misreading something.

Re: what is easy and what is hard to implement/use (and is there a rationale)

What I was saying in the new separate issue is: there are things that are easy to set _as the global default_ and there are things that are easy to _override_; then there are things that are easy to _set_ and things that are easy to _verify_.

Note that Nixpkgs does not have any special status from Nix language point of view, this is an important fact.

It is easy to allow unfree packages globally and it is easy to disable them in the current sense, as in «touching them throws an error immediately, so nix-env ignores them». The «nix-env shows that package names and descriptions but installation requires opt-in» is probably recognised as better than the current situation by a vast majority, but is harder, and there is no real incentive for people writing most of the code to put in all that debugging effort.

It is easy to override a value for a _single_ way of importing Nixpkgs. A typical NixOS system imports Nixpkgs in at least _three_ ways (sometimes more, we are trying to clean that up). Atypical use might include additional ways, but these are easier for the user to enumerate because they were added intentionally. System rebuild, installation of packages as root and installation of packages as user can use three different Nixpkgs config sets naturally. This is natural just because the import of Nixpkgs happens in different envirnoments.

Note that a non-NixOS system with Nix might still have the user/root distinction.

Therefore if a user wants to have a whitelist of unfree packages, or forbid them completely, there are multiple situations to override.

Moreover, it is not completely trivial to check if you have found all of the situations. As I said, from time to time we end up having some remiports of Nixpkgs with yet different settings done behind the scenes (which is something to clean up, but it does happen).

Therefore, it is easy to set globally any zero-nuance policy, and it is easy to override the policy in a specific case, and it is easy to see if that specific case works — but it is hard to see if you override _all_ existing cases. The people writing the code do care about tightly controlling installation of unfree software on their machines, so the default will make this tight control easy.

Also note that making sure that our documentation on _enabling_ unfree software is correct is easy — people will complain (and submit fixes) if it gets out of date. Checking the instructions on _disabling_ is a more involved task. So while the unintuitiveness of enabling unfree software lasts until reading the proper part of the manual, complexity of disabling doesn't go away even for experienced Nixpkgs developers.

In addition to this minor question of functionality desired by people currently writing the code being feasible without constant effort, and to already mentioned legal issues, the unfree software is and always will be supported in a technically inferior way in Nixpkgs, because our main strategy of achieving various useful properties of Nix relies on build-time infrastructure. Redefining installation prefix during the build is cleaner and more predictable and works better than binary patching.

Technical quality of support

As for attracting more users — we currently have bottlenecks in proper onboarding of users who share our view of unfree software (possible, but requires deliberate decisions) and who succeed in reading the manual (there are things to improve to make sure they find the manual they need, not a random of the three major official ones) and who are in favour of opensource development model.

Philosophy

Users who put higher value on unfree software than our current user base (and are also not experienced enough to easily navigate our documentation) need the technically weaker part of Nixpkgs to work well. They are harder to support well, and will take more time to become qualified enough to contribute.

People who share pro-free-software philosophy are more likely to become eager and qualified contributors. We need them. The filtering of allies by mutual tolerance of project-relevant philosophical values has externalities. Some of these externalities are positive.

(And yes, I think I know of multiple Nixpkgs forks that go on for years — if you think you can attract users by making some technically small policy changes, you can try…)

[alexeymuranov]

A minor remark about "set-theoretic" notation: it must have been "Free software ⊂ all software", not "Free software ∈ all software". :)

[grahamc]

If it makes a difference to how people spend their effort on this thread, I think I can conclusively say we will not simply allowUnfree = true by default.

[tbenst]

I think there are a lot of ways of resolving this problem that do not involve changing the default. For example, allowUnfree = false could be explicitly included in the default configuration.nix file. Also, the system allowUnfree setting could propagate rather than each user needing to opt in as well. Right now this is particularly confusing on single user systems which I imagine is a common use case.

[ToxicFrog]

Explicitly including it in the config file doesn't really help that much, because part of the problem is that these packages don't show up in e.g. the command-not-found handler or the nixos.org package search, which leads users to believe that the software isn't in nixpkgs at all.

Personally, I would rather see this software appear in search (with appropriate warnings if necessary) but still default allowUnfree to false, than have it default true but have the software still hidden in search.

[samueldr]

The command-not-found handler, as it is, cannot have unfree software in it because it would imply the hydra build farm would need to build said software to list the binaries available. Same for nix-index or related friends; basically any introspection into the built package is probably out of the question with how things are for unfree.

As far as not introspecting too far, meaning only listing what is known into the Nixpkgs repository, it is possible to list unfree software, meaning it could be possible to list it on the website.

Maybe a middle ground could be reached, where a list of available binaries names (bin/*) is added into meta.bin, allowing command-not-found to be made aware of unfree software. It is technically possible to help make the situation better.

Though, this said, it probably wouldn't happen without the allowUnfree opt-in, if at all, unless @edolstra's opinion has changed, and other contributors have no similar moral objection.

As for my opinion, I'm definitely torn over the issue, the pragmatic in me says: "list the software already", the idealist in me says: "eww proprietary and unfree software", which is probably why I would implement it behind a checkbox or a boolean option, defaulting to unfree software being out of the system.

As this is a discussion about the discoverability of unfree software, let's not steer into "enableUnfree by default", and think of ways to help the UX while still holding on to our convictions.

^{N.B. This is all my personal opinion.}

[7c6f434c]

I wonder how acceptable or non-acceptable is having a NixOS option that simultaneously changes nixpkgs.config and adds NIXPKGS_ALLOW_UNFREE=1 in the default environment (obviously, a user's profile can unset the variable). Maybe also set allowUnfree=false; in generated profile, so that users can know that the flag exists.

[ToxicFrog]

@samueldr Also relevant is https://github.com/NixOS/nixos-homepage/pull/120 (include all packages in the index). It was blocked on package licenses not being properly displayed, but that should have been fixed by https://github.com/NixOS/nixos-homepage/commit/728f6fdb2f6220e7d3b365bb2b767cea1365a1a5

[Ekleog]

@samueldr Personal opinion: the technical solution you put forward for not introspecting too far sounds workable, and I think that unfree packages should be displayed with a big warning “This is unfree!” (or “This is broken!” or “This has security issues!”) sign next to it (not a checkbox, a checkbox will be missed and generate useless duplicate work). Something similar to F-Droid and its “anti-features” list, though F-Droid lists only free software because they have to redistribute binaries.

[ninjin]

I would consider myself fairly close to being a FOSS hardliner, but there are situations – video calls – where even I have to run unfree software on my system and I find the best way to do so is to whitelist unfree packages rather than use allowUnfree=true; which would give me less control:

nixpkgs.config.allowUnfreePredicate = (pkg: builtins.elem (builtins.parseDrvName pkg.name).name [
    "unfree1"
    "unfree2"
    …
  ]);

But this makes unfree packages unsearchable by default using nix-search, which is a pain. I thus use the environment variable pointed out by @7c6f434c for a quick dirty workaround:

NIXPKGS_ALLOW_UNFREE=1 nix search ${pkg}

Just use a shell alias for increased convenience:

alias nix-search='NIXPKGS_ALLOW_UNFREE=1 nix search'

Hopefully this may be of use to some fellow user that does not feel like opening the unfree floodgates just to find all available software from the command line.

[samueldr]

@Ekleog right, I'm using "checkbox" here as a way to keep myself flexible because it depends on the direction the project wants to take as far as publicizing unfree software. One could go the idealist way and ensure it's hidden by default (current situation, minus ability to list in some search), or the pragmatic way, list everything and label accordingly.

I have no strong preference for one or the other, both generally have the same basic requirements (knowing the unfreeness), the rest is mainly an implementation detail according to the vision of the distro.

[jappeace]

Hi, I stumbled on this through google.
So I'm trying to use nix for a client, and it's a bit of a pain I have to opt in (not that I disagree with it in principle). For future reference, this is how you enable non-free in a pin: https://github.com/jappeace/haskell-template-project/blob/45fa410733376d588f7d29d7a9b642189fc8e402/pin.nix#L13

[gloaming]

For the record, I'm not in favour of information hiding in general. However:

Suggestions on showing unfree packages in searches by default are consistently disliked by some core devs

Nixpkgs is intended as a free software distribution, so it should not promote the availability of unfree packages.

The project's view about unfree software is known. We shouldn't promote unfree software.

I like my solution less, because this heavily depends on what do we consider "promotion"

I think it would be good to clarify this. If a user asks for a specific package, then they must already know about it; so does it count as "promotion" to acknowledge its existence? I think there are two use cases of package search that should be distinguished:

1) I want to learn about what packages are available to do a task:

$ nix search play music
...

2) I want to install a particular package, and I want to see its attribute name and version:

$ nix search spotify
...

Of course, we can't know what the user is trying to do. But if we choose to, we can make a best effort guess. I suggest the following compromise behaviour when search for unfree packages has not been explicitly enabled or disabled:

• Don't search unfree package descriptions or other meta
• Only search unfree names and attribute names
• Only return matches with reasonable word boundaries

With rules like these, a search for play music or spot would not return spotify, but spotify could. A search for lime or text editor would not return sublime3, but sublime could.

(The obvious UX disadvantage is that the search behaviour is unusual, potentially confusing, and inconsistent between free and unfree packages.)

These matching rules could also be combined with something like https://github.com/NixOS/nixpkgs/pull/5628#issuecomment-69469083, if we still don't want to display the results by default.

Also, I'm sure this must have been discussed before, but the current behaviour also has the problem that it actually _encourages_ users to set allowUnfree so that they don't miss any packages. It might be better to have e.g. searchUnfree so that users don't have to do that.

[7c6f434c]

… and a problem of trying to go into maximum-purity evaluation mode is that NIXPKGS_ALLOW_UNFREE is very often the best way for dealing with one-off unfree package task.

[Dema]

Frankly speaking I, as a fellow NixOS user, don't really care if "some core devs don't like non-free software". If you don't want your users to use non-free stuff, just simply ban it completely from a distro. And if you allow it, make it first-class citizen.

[7c6f434c]

If you don't want your users to use non-free stuff, just simply ban it completely from a distro.

This is too black and white.

The point is that some core devs want to interact with Nixpkgs _as if_ there were no non-free packages, while not banning such packages completely for those who wants them. Others have a need to install a specific nonfree package, but want to retain tight control over the list of installed nonfree packages.

As there are many different ways to evaluate Nixpkgs on a typical system and they are controlled by slightly different settings, the experience code devs want for themselves requires default-off nonfree software.

And if you allow it, make it first-class citizen.

This can never happen. First-class whatever is supported by the binary cache, and nonfree packages are often illegal to redistribute in modified form.

[SRGOM]

Frankly speaking I, as a fellow NixOS user, don't really care if "some core devs don't like non-free software". If you don't want your users to use non-free stuff, just simply ban it completely from a distro. And if you allow it, make it first-class citizen.

I don't even think anybody on the Nix core team dislikes non-free software. I know one person did but he left to create his own GNU distribution which uses hurd ... GUIX.

[tbenst]

As a user of non-free software, it bothers me the number of times that an update to, say CUDA or MKL breaks a bunch of nix packages. I recently fixed scikitlearn to build with MKL, and less than a week later it’s broken again after MKL got updated. Even with the newly fixed tensorflow it seems that the default CUDA 10(.1) fails to build and user must manually override to 10.0.

First-class whatever is supported by the binary cache, and nonfree packages are often illegal to redistribute in modified form.

Anyone know how Anaconda gets away with distributing binaries of MKL and CUDA?

I wonder if there’s a middle ground where even if we don’t distribute binaries of unfree software, we might configure hydra to test builds against common unfree software? Would make packaging a lot more stable for common workflows.

[AMDphreak]

Frankly speaking I, as a fellow NixOS user, don't really care if "some core devs don't like non-free software". If you don't want your users to use non-free stuff, just simply ban it completely from a distro. And if you allow it, make it first-class citizen.

I don't even think anybody on the Nix core team dislikes non-free software. I know one person did but he left to create his own GNU distribution which uses hurd ... GUIX.

Interesting.

[vyp]

This is off topic because I think this issue is more of a discoverability/documentation/search issue that could be solved without defaulting to unfree = true, but I just wanted to share my opinion on issues regarding nonfree software to everyone arguing to allow it by default.

In my opinion, nonfree software goes against some of the core values of nix, such as reproducibility of packages and the source-based model of nix. (I know many would disagree with that. :woman_shrugging: ) And really we should all be working towards removing our dependencies on any nonfree software we use, to achieve full reproducibility, thereby making our lives easier, instead of having it by default and probably introducing ourselves to more headaches over the long term. That's an idealistic view, and even if people agree with it, they can't just be expected to create their perfect free alternatives overnight to nonfree software that they need to use today, and it's definitely not easy to achieve long-term either (software's always changing and evolving).

But on the flip side I always see people coming to free software communities and almost shifting the blame onto them for not having good integration with nonfree software. We have to recognize that nonfree software is usually designed to be hard to work with or integrate into 'unsupported' systems such as linux distributions. I think in practice this usually comes down to having to modify a binary rather than having access to source code, but at the very least by definition there's always going to be the legal question of distributing modified nonfree software by default. (And yeah, there are lots of free software packages in nixpkgs where the packages are binary-based and not source-based but I see that as more of a manpower issue really.)

I see people invest years of effort into integration/support with nonfree software only for the vendor to remove something or make a breaking change. Forking a project can be difficult and time consuming, but at least the option is there with free software (especially if there's an active community around the project to help), meaning years of effort aren't in principle in danger of being wasted. So sometimes I wonder if they would have been better off investing their time into a free software project instead. This is all subjective and anecdotal, and also tangential to the original issue, and of course there are many cases where integration with nonfree software is desirable to the free software project/community, but just my observations.

I don't even think anybody on the Nix core team dislikes non-free software. I know one person did but he left to create his own GNU distribution which uses hurd ... GUIX.

The Guix System uses the linux-libre kernel by default. So primarily it's a linux system and its packages usually support linux the best.

[7c6f434c]

I don't even think anybody on the Nix core team dislikes non-free software.

Well, disliking is not a binary, it is a matter of degree. And as already linked in this discussion,
https://github.com/NixOS/nixos-homepage/pull/120#issuecomment-268485650 (to link a statement by a person whose membership in the set of core developers cannot be reasonably disputed)

[stale[bot]]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[davidak]

there is still no good solution

Am 1. Juni 2020 07:25:26 MESZ schrieb "stale[bot]" [e-mail]:

Thank you for your contributions.
This has been automatically marked as stale because it has had no
activity for 180 days.
If this is still important to you, we ask that you leave a comment
below. Your comment can be as simple as "still important to me". This
lets people see that at least one person still cares about this.
Someone will have to do this at most twice a year if there is no other
activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask
   on the #nixos channel on
   irc.freenode.net.

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/NixOS/nixpkgs/issues/17126#issuecomment-636619762

--
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

[Enteee]

I think this is more of a documentation issue rather than a technical issue that can easily be resolved by changing the way how the nix tooling works. So why do we not just add a sub-section to the NixOs doc - Package Management that talks about this?

And maybe include a in the default config generated by nixos-generate-config a few lines like the following:

# If set to true, this option will allow you to
# install proprietary software.
# Please note: If set to false, searching for
# derivations using nix-env -qaP won't yield
# any proprietary results.
nixpkgs.config.allowUnfree = false;

... or similar. What do you think?

[7c6f434c]

I think this is more of a documentation issue rather than a technical issue that can easily be resolved by changing the way how the nix tooling works. So why do we not just add a sub-section to the NixOs doc - Package Management that talks about this?

… and a link to Nixpkgs manual, because there is already a subsection there.