Nixpkgs: Remove myEnvFun

[domenkozar]

We should remove myEnvFun and provide instructions how to replace it for different use cases.

$ git grep myEnvFun
pkgs/misc/my-env/default.nix:      sdlEnv = pkgs.myEnvFun {
pkgs/misc/my-env/default.nix:            pkgs.myEnvFun {
pkgs/top-level/all-packages.nix:  myEnvFun = callPackage ../misc/my-env {

This has been discussed before and nix-shell should be cover the same usecases.

Please state why you use myEnvFun if you do so we can provide alternative ways.

[FRidh]

I wouldn't mind not having myEnvFun around.

Am I correct that it doesn't set any additional environment variables, contrary to nix-shell? If true, then I would argue that myEnvFun can't yet be replaced by nix-shell until we get say a nix use which doesn't set any additional variables. Since I think this is how most people use myEnvFun and "misuse" nix-shell.

[avnik]

I used it a lot, to provide dev environments, which updates all-at-once on system rebuild

@domenkozar would be nice to refactor myEnvFun to myInteractiveShell, to allow create load-env-xxx which just provide custom PATH and other vars, with custom set of packages. (and kick all other stuff out of myEnvFun)

[jwiegley]

I use myEnvFun quite extensively, but I'd like to see how to replace my use cases with nix-shell. Let me know when you have those examples, and I'd be happy to try them out by switching over my configuration.

[teh]

@domenkozar - do you want to remove myEnvFun because it fulfills a similar role to nix-shell?

@jwiegley - I had a look at your public git repo and AFAICT you mostly use myEnvFun for getting some binaries into your path? Please correct me if that's not accurate :)

I think that use case would be covered by e.g.:

nix-shell -p clang llvm

[teh]

Here's a search: https://github.com/search?l=Nix&q=myenvfun&type=Code&utf8=%E2%9C%93 for use cases.

[jwiegley]

@teh Several of the environments that I define for use with myEnvFun involve many dependencies, so enumerating them with nix-shell -p would simply be unacceptable. Also, I want to ensure that every dependency is rebuilt when I upgrade the environment; anonymously named dependencies passed to nix-shell may not still be available in the future, and if I'm on an airplane, that doesn't help.

So myEnvFun gives me a few things:

1. A single name for a large set of dependencies that are not immediately accessible from my user environment.
2. A convenient command to bring these dependencies into scope.
3. Because of the relationship of the command to the dependencies, they are all upgraded together when using -u --leq.

As far as I can tell, nix-shell gives me none of these benefits.

[avnik]

On Wed, Feb 22, 2017 at 09:38:36AM -0800, teh wrote:

@domenkozar - do you want to remove myEnvFun because it fulfills a similar role to nix-shell?

@jwiegley - I had a look at your public git repo and AFAICT you mostly use myEnvFun for getting some binaries into your path? Please correct me if that's not accurate :)

I think that use case would be covered by e.g.:

nix-shell -p clang llvm

It contrast with myEnvFun it not allow to tie that to build process for
entrie system -- for example -- I have building load-env's scripts for
golang, haskell, ocaml and wine.

I definelly won't wait build/download when I do load-env-golang, and
moreover, I won't fix errors on this time. I have dedicated time for it,
when I update whole system. (It only my own use cases, other folks can
have own).

So we need revamp myEnvFun, and possible rename to myInteractiveShell,
or likewise.

[teh]

Those seem like valid use cases to me. @domenkozar is myEnvFun a support burden? Can I do anything to make it nicer that would allow us to close this bug?

@avnik renaming seems like unnecessary churn unless you feel strongly about this.

[domenkozar]

All those use cases work with nix-shell used together with a default.nix. You can specify as many dependencies as you want, fix the NIX_PATH, etc.

The only thing you'd need to do is something like a bash alias my-project = nix-shell /path/to/my/default.nix.

Am I missing something here?

[wizeman]

@domenkozar How do you tie that to the build process for the entire system?

I have a local server with Hydra. I want the configuration of all my machines (including laptop, server and others) to build in my Hydra server at the same time, including all my programming environments.

With myEnvFun it's very simple - I just add my programming environments (one for each programming language) to environment.systemPackages and that's the end of it.

If I have a bunch of default.nix files scattered all over my filesystem, wouldn't it be more difficult to keep track of them and include them in my system configuration so that I can send it to my Hydra instance? I'm not even sure how I could accomplish that reliably.

[FRidh]

@wizeman you could define your environments in just one file, even in your configuration.nix, and add your envs to the Nixpkgs tree with say packageOverrides.

Anyway, nix-shell executes a shellHook which I don't think is always desirable.

[avnik]

On Thu, Feb 23, 2017 at 01:20:13AM -0800, teh wrote:

Those seem like valid use cases to me. @domenkozar is myEnvFun a support burden?

It have a lot of bare understandable hacks inside, so I believe it can
be simplified.

Also I feel it can be extended by executing pre-defined, or customized
shell instead bash.

[FRidh]

I suppose something like PATH=:$(nix-build -A someDerivation)/bin $SHELL would typically be enough?

[avnik]

On Fri, Feb 24, 2017 at 09:56:05AM -0800, Frederik Rietdijk wrote:

I suppose something like PATH=:$(nix-build -A someDerivation)/bin $SHELL would typically be enough?

Doesn't cover case for "I want all my language specific dev environments
built at once with all system"

[dhess]

I'm not a Nix expert, so I might be missing something; but as far as I can tell, besides the changes to the prompt that myEnvFun performs, the buildEnvs I create in my config.nix do the same thing:

https://gist.github.com/dhess/2ca7e5e836c52fd0187637a49ace04fb

I can nix-shell -p shell-env to get an environment with my shell tools, nix-env -i haskell-env to ensure that GHC and friends are always available, or nix-env -i meta-env to install the lot.

I can build everything in advance (for occasions when I don't have Internet access, which, as mentioned by others, is often very convenient) just by calling nix-instantiate, which I do whenever I update my nixpkgs git repo from upstream, along with pushing everything to my own binary cache so that I only have to build the new derivations once, and can re-use them on my other machines.

In fact, by changing just buildEnv → myEnvFun, paths → buildInputs, and meta-env → env-meta (etc.) in that confix.nix file, I was able to trivially swap between the two methods. None of the packages even needed to be re-built. (I ended up switching back to my system because I don't like the changes that myEnvFun makes to the environment.)

The only thing I'm unhappy about is this hack:

    ## This exposes the locally-defined package to other Nix files,
    ## e.g., default.nix in a git repo which you're using to develop a
    ## Haskell package.
    dhess-ssh-keygen = (my-haskell-packages super.haskell.packages.ghc801).dhess-ssh-keygen;
    fm-assistant = (my-haskell-packages super.haskell.packages.ghc801).fm-assistant;
    hpio = (my-haskell-packages super.haskell.packages.ghc801).hpio;

If I don't do that, nix-env complains that the dhess-ssh-keygen and fm-assistant packages that I refer to in my haskell-core function aren't defined; and it picks up the nixpkgs version of hpio, rather than my override. If someone can help me with the Nix-foo required to get rid of that hack, that would be awesome. The reason I define the haskell-core function is that I want the same set of Haskell packages for each of the GHC environments that I might build/install. (Using myEnvFun had the same issue, so this is not specific to my environment.) The convoluted Haskell package stuff I'm doing at the bottom of the config.nix file is my way of removing problematic packages from the master list if particular packages don't work with particular versions of GHC.

[jwiegley]

I use both buildEnv and myEnvFun. The problem is that if I install a buildEnv, all of its dependencies become visible at user scope, meaning I can't install two buildEnvs with conflicting paths. And if I don't install them, then they don't rebuild when I upgrade my environment, but only when I try to use them.

myEnvFun both keeps the dependencies built, and makes them accessible only when requested.

Now, if there were a way to provide a list that meant "always build these, but don't install them", then I could introduce the buildEnvs using nix-shell when needed.

[dhess]

@jwiegley I believe you could do that with nix-instantiate '<nixpkgs>' -A meta-env where meta-env is a buildEnv with all of the environments you want to build, but not install. I do that for my stack-env, which is never installed in my environment, but I like to have it ready to go on a moment's notice.

I guess you just want to run nixos-rebuild switch or whatever and not have to worry about the extra step? (I'm on a Mac, so for me, that's not an option.)

[jwiegley]

Yes, right now nix-env -u --leq handles everything, and that's what has made myEnvFun the right tool for the job so far.

[cruegge]

A simple way to get the automatic updating might be to install a script that artificially depends on everything you want and calls nix-shell on itself, something like

runCommand "fooenv"
{ buildInputs = [ <...> ]; }
''
  mkdir -p $out/bin
  cat > $out/bin/fooenv <<EOF
  #! ${stdenv.shell}
  exec nix-shell \$(nix-store --query --deriver $out)
  # $buildInputs $nativeBuildInputs
  EOF
  chmod +x $out/bin/fooenv
'';

[LnL7]

Why not generate a wrapper that just executes nix-shell or is there something that myEnvFun does differently?

[Gabriel439]

The issues I've run into with nix-shell are:

• You can't build and cache the environment it assembles using Hydra
• You can't install the environment on a NixOS system using environment.systemPackages
• It's difficult to override nixpkgs when using the --packages flag

[FRidh]

Simple solution:

• specify environments with buildEnv and make them accessible by overriding the package set, e.g. in config.nix. You could e.g. add myEnvs.someEnv = buildEnv { ... };
• nix-shell -p myEnvs.buildEnv to access your env

[jwiegley]

@FRidh That's an interesting idea! I'll give that a try and see how it works.

[srhb]

@jwiegley How'd it work?

[jwiegley]

I haven't had the chance yet to retool my environment; I rely on myEnvFun in quite a few ways.

[jwiegley]

@FRidh But how do you keep myEnvs.buildEnv up-to-date when not using the nix-shell? On my system, I have several Coq environments:

env-coq85
env-coq86
env-coq87

Thanks to myEnvFun, nix-env -u --leq will update all 4 and then nix copy --to ssh://hermes will copy all over (and their many dependencies) to my laptop for taking out of the home where I might not have Internet.

As as I can tell, just declaring the package sets in config.nix and invoking them with nix-shell when needed (a) won't keep them up-to-date until used and (b) won't copy them as part of my home environment's closure to the other machine.

[dhess]

@jwiegley Here is how I work with buildEnvs. My approach has changed quite a bit since my last reply to this thread in Feb 2017.

I've restructured all of my buildEnvs, plus the packages that they incorporate, so that they're part of an overlay. I import this overlay into my Nix environment to make the buildEnvs visible to nix-env.

I've also created a Hydra-style release.nix that instantiates the buildEnvs, plus whatever other one-off packages I might want to have available, as Hydra jobs. My Hydra builds everything that's part of the release.nix output attrset whenever the repo changes. You can see this in action here: https://github.com/dhess/nixpkgs-dhess/blob/4e814fab7e2eb58ed5353b19c87f5221c7c56d9e/jobsets/release.nix

You don't need a Hydra to take advantage of this approach. You can just run nix-build jobset/release.nix -A x86_64-darwin from the top of the repo and build the products directly on your workstation (or even remotely if you have remote build machines defined).

The nix-build is an extra step versus what you're doing now, but it's easy enough to add that to a script that also wraps the nix-env -u --leq bit as well.

Of course, this is not a drop-in replacement for what you're doing with myEnvFun due to the functional differences between nix-shell and myEnvFun, but it solves the problem of making packages available at all times, even if they're not currently in your environment.

I don't know if there's a direct solution to your nix copy issue as I don't take that approach to get packages on my other Macs. With this new system I'm using, the Hydra provides a binary cache of the release.nix evaluations, so that is where all of my Macs get their updates. Before I set up a Hydra, I just wrote a simple nix-binary-cache-update script that pushed the products I built on my desktop Mac to an S3 binary cache, and I added the URL for that binary cache to the nix.conf on my other Macs so that they would look there first before building updates themselves. Again, not the same as what you're doing now, and probably not as efficient as an scp over the local network, but do-able. (One advantage of the S3 approach is that you can still get the packages if you leave the house and forgot to nix copy to the laptop first.)

[jwiegley]

If I can build all the dependencies with a single nix-build call, then I should be able to copy the closure of that output derivation to the other Mac directly. It adds one more step, but that's not insufferable, since I already have to run three commands (nix-env -u, darwin-rebuild switch, home-manager switch).

So, as long as I have a way to keep all the dependencies updated and copied en masse, and then easily produce aggregated environments, I suppose I can relax my clawed fingers from grasping myEnvFun so tightly.

[jwiegley]

@dhess Many of my environments have overlapping files. I've not yet found a way to aggregate them with nixpkgs in such a way that I can build them all by just building one attribute.

[jwiegley]

Another downside to the nix-shell approach: https://github.com/NixOS/nix/issues/459

[jwiegley]

OK, I've tried in earnest over the last three months to make nix-shell an integral part of my workflow, and I have to say at this point that it's still very much inferior to myEnvFun, on multiple counts. Here are all the problems I'm still having with it, none of which are a problem for myEnvFun:

• nix-shell always uses <nixpkgs>, despite giving it a different path (see https://github.com/NixOS/nix/issues/459, linked above). For my use case, this means I have to hack the default.nix file to use <darwin> for all of my projects.

• You cannot copy the closure for a nix-shell environment after it is built, requiring that all machines redo that work. I currently work around this by copying every file from /nix/store to my other machines -- since that's faster than rebuilding everywhere -- but it's still quite inefficient to solve this problem.

• nix-shell appears to randomly decide that it needs the Internet to build your shell. This happened to me on the airplane the other day, in mid-air: I hadn't changed anything in my Nix configuration, nor in my system configuration, nor even in the dependencies of my project, yet I couldn't enter nix-shell because it suddenly required the Internet. The fix: enter the myEnvFun I keep around with the same environment, since that never asks for the Internet after being built.

• You cannot "upgrade all of your nix-shells" with a single command. Instead, it must be done shell by shell, using hand-rolled scripting that's much easier to get wrong than nix-env -u --leq.

• You must manually manage GC roots to avoid your nix-shell dependencies from being garbage collected, especially if you have multiple projects pinned to different versions of their related dependencies.

So far, the only upside I've discovered with using nix-shell is that I can use haskellPackages.developPackage so that it provides whatever Haskell dependencies my project thinks it needs, rather than a static list I must keep in my Nix configuration. This is quite cool.

In sum, if the above issues concerning nix-shell could be fixed, I'd happily switch to it for its per-project convenience. But until then, the risk of not being able to use nix-shell at all when the internet is missing is a show-stopper. Please keep myEnvFun around until at least that is fixed!

[Gabriel439]

@jwiegley very accurately summarizes the issues I've had with nix-shell. I've been bitten by every single bullet point he mentioned

[teh]

I was trying to drive-by unblock this ticket, but am also now a user of myEnvFun. @domenkozar unless myEnvFun is a support burden [1] maybe we can close this ticket?

[1]
I think the evidence shows it's not a burden: 1 commit in 2018 (by @jwiegley) and 3 commits in 2017.

[lheckemann]

Now, if there were a way to provide a list that meant "always build these, but don't install them",

system.extraDependencies I think?

[jwiegley]

I will say that right now, even though I'm using nix-shell far more than not now, my biggest blocker is the seemingly unpredictable nature of its dependence on the Internet. The beauty of myEnvFun lies in avoiding the case of wanting to work while on the plane, but getting stuck because nix-shell wants the Internet and I don't have it.

[lheckemann]

nix-shell with a .drv file that's been registered as a GC root and previously been shelled into should work consistently without having to download anything as far as I know. (as long as the keep-outputs option is set)

$ grep keep-outputs /etc/nix/nix.conf
gc-keep-outputs = true
$ nix-instantiate shell.nix --indirect --add-root $PWD/shell-drv
$ nix-shell ./shell-drv --run true
<stuff is downloaded>
$ nix-store --gc
$ nix-shell ./shell-drv --run true
<nothing downloaded>

[FRidh]

Nix 2.0 is out for some time now. How is your experience with using nix run?

default.nix in project folder

with import <nixpkgs> {};
buildEnv {
  name = "myEnv";
  paths = [ python2 python3 ];
}

and execute nix run.

overlay

self: super: {

  hello = super.hello.overrideAttrs(oldAttrs: { name = "foo"; });

  envs = let
    items = {

      env1 = super.buildEnv {
        name = "env1";
        paths = with super; [ python2 python3 ];
      };

      env2 = super.buildEnv {
        name = "env2";
        paths = with self; [ hello ];
      };
    };
  # Create a text file that references every env.
  # Install this, and you will have built versions of
  # the envs available.:
  #
  # `nix-env -iA nixpkgs.envs`
  #
  # Activate env:
  #
  # `nix run nixpkgs.envs.env1`
  in super.writeTextFile {
    name = "envs";
    text = super.lib.attrValues items;
    destination = "/nix/envs";
  } // items;
}

Of course, using a different Nixpkgs is always possible with say fetchTarball.

[lheckemann]

@FRidh s/nix-run -iA/nix run/ ?

[jwiegley]

@lheckemann Such management is highly manual right now, whereas myEnvFun's can be collected into a list in the main system configuration, and even upgraded all at once.

I'd like to step back and see if a redesign of nix-shell is in order, so that we merge the benefits of both approaches, and expel some of these weaknesses.

[cruegge]

Another advantage of myEnvFun and similar constructions is that the environments can be pre-built on one machine and transferred to others. Not GC'ing the drv only works on the machine where the evaluation was performed.

[FRidh]

@jwiegley well, nix-shell does what it needs to do, at least in my opinion. It's just that nix-shell is often also used for just running programs, something which it is not completely intended for, considering the hooks it runs.

nix run on the other is intended for starting up programs or environments.

[jwiegley]

@FRidh I have no problems with nix-shell being a special-purpose tool. My problem lies with suggesting that a tool which does work, myEnvFun, should be removed, and I should be using nix-shell for purposes it clearly wasn't intended for. This happens in the #nixos channel too, where the answer to so many questions has been "just use nix-shell". That makes it sound like it's not actually a special purpose tool.

If it is special purpose, and we want to deprecate myEnvFun, then let's make a new tool that combines their functionality.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.