Nix: User land nix packages support

[pjotrp]

For cluster computing where you don't get root it would be very useful to have nixpkgs in user space. It would be so very useful!

Also, if that facility existed - package managers for languages and browsers could be based on Nix packaging (think Ruby/rvm, Firefox plugins, etc).

[pjotrp]

Oh yes, I would be the first tester :)

[shlevy]

Perhaps I'm misunderstanding exactly what you mean, but you can already use nix on a system where you don't have root access. Just put your nix store in, say, $HOME/nix and nix itself will work. We try to avoid having any hard-codings of /nix/store in nixpkgs, so most of nixpkgs should work as well, though you won't get the benefit of the build farm.

[pjotrp]

Binary nixpkgs is what I am talking about. Do you think a firefox plugin wants to be built? With binary packages we can do away with most packaging systems, such as ruby gems, python eggs etc. These are all hacks, reinventing the wheel. Get one of these projects on board, and Nix will rule.

I discussed this with Eelco at FOSDEM 2012. We were talking path rewriting.

[edolstra]

@pjotrp Some follow-up to our discussion yesterday:

So store path rewriting would allow sharing of binaries between stores with different prefixes, as long as the prefixes have the same length. So /home/alice/store and /home/bob/store would be incompatible, but /home/alice/store and /home/bob/store__ would work because they have the same length.

Currently, the store path hashes that Nix computes depend on the store prefix. For instance:

$ NIX_STORE_DIR=/foo nix-instantiate --readonly-mode '<nixpkgs>' -A hello
/foo/wrd8l1wzk91l0wmq60rzcadxsgjyr3pw-hello-2.8.drv

$ NIX_STORE_DIR=/bar nix-instantiate --readonly-mode '<nixpkgs>' -A hello
/bar/10y5nydvyb5z29cwlhk5scf2kpzq14i7-hello-2.8.drv

This means that things like the binary cache mechanism won't work properly between different store prefixes, because the binary cache is indexed by the store path hash.

The solution would be to distinguish between the actual store prefix (e.g. /home/alice/store) and the "canonical" store prefix, which would have to be of the same length (e.g. /abcdefghijklmnop). The latter would be used for computing store path hashes and for binary cache lookups. So when Alice does "nix-env -i hello", Nix will compute the store path /abcdefghijklmnop/-hello-1.2.3 and look it up in the binary cache. If it exists, it will fetch the NAR, and unpack it in /home/alice/store/-hello-1.2.3, rewriting all occurences of /abcdefghijklmnop to /home/alice/store. Export operations like nix-push should do the reverse.

Doesn't sound too hard...

[shlevy]

@edolstra so will /nix/store become /nix/store__________ by default, for some reasonable number of underscores?

[edolstra]

Not by default, only for people who want this (e.g. Pjotr).

The default prefix /nix/store would actually allow you to put the store in /tmp (issue NixOS/nixpkgs#1605), which might be good enough for some use cases.

[pjotrp]

How hard would it be to make the hash calculation start from the Nix STORE path, rather than include it?

Essentially that would free up all user land software. We could have two versions of binaries, a 'hard' one including /nix/store and a 'soft' one, excluding the prefix. The former would be the default, but the build farm could generate both and caching would work too. It would be up to the user which version to choose. In addition we can provide in-line patching for migrations of linked binaries.

I have three use cases now. (1) cluster installs where I don't get root access (2) GRID computing where I don't even get the same $HOME, let alone root and (3) simple user land cases, such as firefox plugins and Ruby gems. I think Guix will start to give Nix broader acceptance as a generic solution. User land support would be the icing on the cake.

[edolstra]

I'm trying to remember if there is a good reason for including the store prefix in the hash calculation, but I can't think of one. It ensures uniqueness of hashes across different store prefixes, but that might not be very important.

[shlevy]

@edolstra but even if we didn't include the store prefix, it's not like packages would actually be portable between stores with different prefix lengths right?

[shlevy]

I mean I suppose we could try writing a stdenv that used relative paths for rpaths and such...

[pjotrp]

It would also relieve us of the FSH objection. Install the store in /usr/local/nix

[edolstra]

@shlevy No.

[shlevy]

@pjotrp That doesn't actually relieve us of the FHS objection, as /usr or /usr/local are not semantically correct places for the store to live. nix uses a directory not recognized by FHS precisely because the FHS simply doesn't cover nix's use case.

[pjotrp]

That is semantically correct, I presume, but since almost all systems make the /usr/local open for shared software installation I don't see why the store can't live in /usr/local/share/nix/store, for example.

http://www.pathname.com/fhs/2.2/fhs-4.9.html

[vcunat]

Hashes are not the problem... not just Rpaths are hardcoded at configure/compile-time -- also data directories, etc... these get compiled as strings into binaries and I don't see any generic way of fixing that, except for some rewriting, which is only safe for paths of the same length (or shorter if we use symlinks in a clever way).

FHS: even /usr/local/... isn't usually writable by regular users AFAIK.

[edolstra]

@vcunat See above, the idea is to do rewriting of store prefixes that must have the same length.

[vcunat]

Yes, I do agree that's well doable. Then people can have nix store even in their home (unless their home path is much too long). But for all this to be useful, we need to have the things built by Hydra with a long-enough nix-store prefix (e.g. elongated by some random string to be easily recognizable), which won't be used for the default platforms as we have now, I presume. Thus, it would be a bit more pressure on the build farm, but I suppose similar exotic versions could be built less frequently than the default stuff.

I'm a little worried about increasing the maintenance complexity, having something like another "path-portable" version of platforms we care about, but hopefully a different prefix will cause very few differences in errors.

[shlevy]

@vcunat +1. This seems like an added maintenance burden without much corresponding benefit IMO. But ultimately up to @edolstra of course :)

[domenkozar]

Having a way to install Nix and packages without root would mean users can replace their package manager with Nix pretty much everywhere.

Someone wants a specific, working version of firefox? Bootstrap Nix, install, go.

[shlevy]

But this proposal won't give that. It will allow packages to be portable from one store to another only if their store prefixes are the same in length, and since /nix/store isn't going to change that means for packages we'll need to _also_ build for some /prefix/that/is/really/really/really/long if we want most users to be able to use it.

[vcunat]

@shlevy: yes, it would certainly mean a separate building for these portable packages. What about some other, cheaper way: I suppose regular user can do stuff like LD_PRELOADing -- a simple libc wrapper could convert all syscalls that include paths -- substitute all occurrences of /nix/store/ to your custom path, all without rebuilding anything.

[edolstra]

@shlevy I never said _we_ are going to build those packages. But it does allow other people to do that.

[matejc]

Right now I am at work.. and I am going to be quick here .. but this might be relevant on this topic...

This is my Hydra script to build Nix and/or other packages with custom prefix https://github.com/matejc/hydra_scripts/blob/master/release/3_vm.nix

And here are build inputs for example: https://hydra-ssl.scriptores.com/jobset/projecttwo/custom_prefix#tabs-configuration

Script builds in any directory .. I am using virtual machine because I do not want those directories to be directly on my server, and chroot is not an option here because you need to be root to use it.

Please do not all go download build output, my server has slow connection :P
Script is in development and pull requests are welcome...

[vcunat]

I haven't noticed we have a related PR https://github.com/NixOS/nixpkgs/pull/1650.

[offlinehacker]

Could user namespaces be used for that? Also, could be patchelf changed in a way that it would do static "LD_PRELOAD"?

[edolstra]

Well, Pjotr's particular use case was deployment in grid environments were you can't use namespaces (at least not anytime soon). LD_PRELOAD might be an option.

[offlinehacker]

I had an idea that we could change the default nix store location which would get mount --bind on /tmp/store or somewhere in tmp on boot. This way would allow to have all packages build for path that is writable by almost all users on almost all systems, and in case of binary deployments only small bootstrap script would be needed that would copy all files to that tmp location before start. I also made an issue on nixpkgs for that https://github.com/NixOS/nixpkgs/issues/1605

[offlinehacker]

You would do this only on systems running nixos, on non nixos systems you
would have bootstrap script that copies all files to tmp if they are not
yet there.
On Feb 6, 2014 8:10 AM, "Vladimír Čunát" [email protected] wrote:

@offlinehacker https://github.com/offlinehacker: how would you do bind
mount without root access?? To me this seems to defeat the main purpose...

—
Reply to this email directly or view it on GitHubhttps://github.com/NixOS/nix/issues/16#issuecomment-34297927
.

[offlinehacker]

Okay i will explain again and make it more clear:

To have a nix store on common location and writable by all users i would
place it somewhere in /tmp by default. Now it becomes problematic, because
/tmp gets usually cleared on reboot. I would solve this by mount --bind
/nix/store /tmp/... on any system running nix where you have root
access(not only nixos). On binary deployment case, where you usually don't
have root access, you can't do mount bind, in that case you would have a
bootstrap script that would copy store closure to /tmp before starting of
any other binary.
On Feb 6, 2014 8:52 AM, "Vladimír Čunát" [email protected] wrote:

On NixOS any user can install with nix, so there's no need. On others I
still don't see how you do this without root.

—
Reply to this email directly or view it on GitHubhttps://github.com/NixOS/nix/issues/16#issuecomment-34299818
.

[vcunat]

EDIT: I believe I understand @offlinehacker now, so I cleaned my questions from the thread.

[brodul]

It would be nice to have at least source build support in the home.

The problem is to find the dependencies for the latest nix. If I want to compile nix with custom store path as a normal user I would need to compile also the dependencies ( https://nixos.org/wiki/How_to_install_nix_in_home_(on_another_distribution)
). It would be nice to have a binary nix with the option to override the /nix prefix ( I didn't really checked if this is possible at the moment ).

Personally I don't care that much about binary support for me it would be fine to compile from source so I would than have different hashes than hydra. Maybe this could be improved later.

[offlinehacker]

Well i think this is not even the biggest issue right now, i would like to see systemd user services support first. I opened this issue https://github.com/NixOS/nixpkgs/issues/1689

[offlinehacker]

@brodul what @matejc did (https://github.com/matejc/hydra_scripts/blob/master/release/3_vm.nix) is basicly what you want, but still needs a few improvements(like source cleanup, blog post or wiki entry of course). This is still not really usefull for me until user systemd services work.

[wavewave]

Recently, I found a nice solution to this problem using http://proot.me
You can download http://static.proot.me/proot-x86_64
proot-x86_64 is an executable. (need to do chmod u+x)
Download nix binary and unzip it somewhere (for example /tmp/wavewave/nix)
Then I can run it with ./proot-x86_64 -b /tmp/wavewave/nix:/nix
Then /nix/store is there and runnable.
You can still access your home directory and other directory. It's user-land mount bind.

[pjotrp]

Very interesting!

[pjotrp]

We can close this issue. proot solves the problem of installing Nix in userland. It is the dog's bollocks for cluster computing setups where scientists have no root!! But it is going to be more important than that. Userland rules.

@wavewave thanks for spotting and sharing this solution. You can not believe how excited I am.

[knedlsepp]

I just saw the NixCon presentation of @domenkozar, where he talked about the state of python packaging. In the last few minutes he mentioned that even though python packaging is made simpler on nix, there is no real movement of pythonistas towards nix because the conda package manager already offers what nix has to offer. In my opinion this is mainly because of exactly this issue. The conda people solve the $HOME installation via a relative setting of the RUNPATH in libraries, whereas in nix this requires rebuilding everything.

What exactly would be necessary to make this happen?

• [ ] Removing the store prefix from being included in the hashing. (https://github.com/NixOS/nix/issues/166)
• [ ] Making sure no absolute paths to /nix/... (in binaries/scripts), but only relative ones make it into the store.

Implementing such a solution would make nix so much more versatile. Thus I ask the question again: Is the second goal really that unreachable? How do the conda people do it?

Edit:
Not as many packages as on nix, but I think they are doing a really great job. And the UX is great!
https://conda-forge.github.io/feedstocks.html