Nixpkgs: emacsWithPackages uses dependencies from wrong package set

[rzetterberg]

When I use the emacs package counsel-projectile from emacs.melpaPackages it
includes the dependency counsel from emacs.melpaStablePackages.

Here's my minimal exampel:

emacs1.nix

{ system ? builtins.currentSystem }:

let
  pkgs = import <nixpkgs> { inherit system; };
  emacsWithPackages = with pkgs; (emacsPackagesNgGen emacs).emacsWithPackages;
in
  emacsWithPackages (epkgs: (with epkgs.melpaPackages; [
    counsel-projectile
  ]))

After building with nix-build emacs1.nix, starting it with result/bin/emacs -Q
and running package-initialize inside the started emacs I show the version of
counsel and get the following result: counsel-0.8.0.

That is the wrong version of counsel, it should be counsel-20170219.2346.

If I only install counsel in my wrapped emacs:

emacs2.nix

{ system ? builtins.currentSystem }:

let
  pkgs = import <nixpkgs> { inherit system; };
  emacsWithPackages = with pkgs; (emacsPackagesNgGen emacs).emacsWithPackages;
in
  emacsWithPackages (epkgs: (with epkgs.melpaPackages; [
    counsel
  ]))

After building with nix-build emacs2.nix, starting it with result/bin/emacs -Q
and running package-initialize inside the started emacs I show the version of
counsel and get the following result: counsel-20170219.2346.

So there is something that goes wrong when counsel is installed as a
dependency of counsel-projectile. It seems the it is using counsel from
melpa-stable-generated.nix and not melpa-generated.nix.

My environment

$ nixos-version
17.03.1458.057f89b934 (Gorilla)
$ nix-env --version
nix-env (Nix) 1.11.11

Additional information

Here's how counsel is defined in melpa-generated.nix:

    counsel = callPackage ({ emacs, fetchFromGitHub, fetchurl, lib, melpaBuild, swiper }:
    melpaBuild {
        pname = "counsel";
        version = "20170219.2346";
        src = fetchFromGitHub {
          owner = "abo-abo";
          repo = "swiper";
          rev = "950545ba0e0bde6b4662aa769f60ac3c768ffeb2";
          sha256 = "0mq1id9xv4rn13y95n98pyywgrll2yb69v1cqacq8rq7my08awsq";
        };
        recipeFile = fetchurl {
          url = "https://raw.githubusercontent.com/milkypostman/melpa/06c50f32b8d603db0d70e77907e36862cd66b811/recipes/counsel";
          sha256 = "0y8cb2q4mqvzan5n8ws5pjpm7bkjcghg5q19mzc3gqrq9vrvyzi6";
          name = "counsel";
        };
        packageRequires = [ emacs swiper ];
        meta = {
          homepage = "https://melpa.org/#/counsel";
          license = lib.licenses.free;
        };
      }) {};

Here's how counsel is defined in melpa-stable-generated.nix:

    counsel = callPackage ({ emacs, fetchFromGitHub, fetchurl, lib, melpaBuild, swiper }:
    melpaBuild {
        pname = "counsel";
        version = "0.8.0";
        src = fetchFromGitHub {
          owner = "abo-abo";
          repo = "swiper";
          rev = "c24a3728538dd7d11de9f141b3ad1d8e0996c330";
          sha256 = "19vfj01x7b8f7wyx7m51z00la2r7jcwzv0n06srkvcls0wm5s1h3";
        };
        recipeFile = fetchurl {
          url = "https://raw.githubusercontent.com/milkypostman/melpa/06c50f32b8d603db0d70e77907e36862cd66b811/recipes/counsel";
          sha256 = "0y8cb2q4mqvzan5n8ws5pjpm7bkjcghg5q19mzc3gqrq9vrvyzi6";
          name = "counsel";
        };
        packageRequires = [ emacs swiper ];
        meta = {
          homepage = "https://melpa.org/#/counsel";
          license = lib.licenses.free;
        };
      }) {};

[mdorman]

Perhaps @ttuegel will correct me, but unless you specifically specify that a package should come from a particular source, packages from melpa-stable take precedence over packages from melpa, and the only relationship between packages is based on the name---so you will always get the melpa-stable version (if there is one).

If you want the unstable counsel package, list it alongside counsel-projectile so it, too, will specifically be pulled from melpa.

[rzetterberg]

@mdorman thanks for the tip!

I tried adding counsel as a separate dependency:

emacs3.nix

{ system ? builtins.currentSystem }:

let
  pkgs = import <nixpkgs> { inherit system; };
  emacsWithPackages = with pkgs; (emacsPackagesNgGen emacs).emacsWithPackages;
in
  emacsWithPackages (epkgs: (with epkgs.melpaPackages; [
    counsel
    counsel-projectile
  ]))

But it doesn't change anything, I still get the same result as in my example emacs1.nix when only counsel-projectile is added.

[rzetterberg]

I tried to override the counsel-projectile package specifically:

emacs4.nix

{ system ? builtins.currentSystem }:

let
  pkgs = import <nixpkgs> { inherit system; };
  emacsWithPackages = with pkgs; (emacsPackagesNgGen emacs).emacsWithPackages;
in
  emacsWithPackages (epkgs: (with epkgs.melpaPackages; [
    counsel
    (counsel-projectile.override {
      inherit counsel;
      inherit projectile;
    })
  ]))

That helped me get counsel from melpaPackages instead of melpaStablePackages, but now ivy that counsel uses is the wrong version (taken from melpaStablePackages).

Since overriding the whole dependency tree manually to use the right package set is not viable, what are my options? Can I somehow say that all packages in melpaPackages should only use dependencies from melpaPackages?

Would really appreciate some input from @ttuegel or @mdorman since I'm at loss what my next step should be. Thank you!

[peterhoeg]

This isn't specific to counsel-projetile - I have noticed that a number of other packages also pull in the wrong dependencies causing emacs to add both stable and unstable versions to the load path which obviously gives a range of fun to troubleshoot errors when the wrong version is being loaded.

[peterhoeg]

Ironically this makes makes letting nix handle the emacs config less deterministic than plain use-package package-install. @ttuegel, anything I can provide to help get to the bottom of this?

[vyp]

@peterhoeg What do you mean "plain use-package"? Isn't nix and use-package orthogonal?

[peterhoeg]

@vyp I meant package-install (comment updated). package-install will honor archive priorities. See package-archive-priorities.

Because you are of course right!

[peterhoeg]

I'm taking the liberty of roping in @the-kenny, @peti, @mdorman, @oxij and @vyp who as emacs users may have something to add.

[oxij]

I follow this thread (actually, any thread that mentions "emacs") but I
still don't understand which semantics you want in the end.

• Do you want to have packages prefer deps from the same set when
  possible?

• Do you want another set of expressions like melpaPackagesPreferSelf
  that will do the above?

• Do you want the above, but only for some packages?

The latter can be done today with emacs-packages.nix. Just override
the package in question there and get happy. The second option will need
some boiler-plate, but should be easy enough to do. The first option
needs consensus.

[peterhoeg]

As it is right now, I have several packages that are duplicated from both stable and regular melpa which breaks things.

The way I see it, there are two options:

1) Always install dependencies from the same set but don't pull in an older package. Examples:

• Package A in melpa depends on package B..
• Package C in stable also depends on package B.
  Package B should only be pulled in once and it should be the one from Melpa.

2) Support priorities the same way that emacs (or rather package-install) does so the user can explicitly choose priorities.

[oxij]

Both of your options have very non-nix semantics. From nix point of view
in both cases you want buildEnv to override package deps when the same
package gets two versions.

This is impossible. Example

• Package A in melpa depends on package B.
• Package C in stable also depends on package B.

but both A and C refer to some files (e.g. a binary) in their respective
versions of B by /nix/store path. Do you expect buildEnv to patch
A and C to refer to the single version of B?

The only reasonable way to fix such a case is to override it by hand and
make nix rebuild the expression.

That said, I think that if we switch to "package from the same set
first" approach and make emacsWithPackages fail when it has two
versions of the same package, we might end up with a better overall
usability (users will need to do less overrides) than now.

This is the second or third issue on the subject. I think that going
"same set" will remove the element of surprise that prompts opening of
these issues.

[peterhoeg]

Do you have an example of the manual override for use as a temporary workaround?

if we switch to "package from the same set first" approach and make emacsWithPackages fail when it has two versions of the same package

👍

[oxij]

Do you have an example of the manual override for use as a temporary workaround?

put into the package declaration block in emacs-packages.nix:

evil-org = (melpaPackages self).evil-org.override {
  org = (elpaPackages self).org;
};

Seems to work.

if we switch to "package from the same set first" approach and make emacsWithPackages fail when it has two versions of the same package

👍

Let's wait and see what @ttuegel has to say about this.

[ttuegel]

I'm not sure I understand what is wanted here, because several people have chimed in with what seem to be disparate goals. So, I'm going to address the original poster and explain what is going on and how we might solve it.

The original problem is that the following installs the stable version of counsel as a dependency of counsel-projectile:
```.nix
emacsWithPackages (epkgs: (with epkgs.melpaPackages; [ counsel-projectile ]))

while this gives the latest unstable version of `counsel`:
```.nix
emacsWithPackages (epkgs: (with epkgs.melpaPackages; [ counsel ]))

Why does this happen? epkgs is a single package set with consistent dependencies where stable versions are preferred where they exist. epkgs.melpaPackages is the set of unstable packages with dependencies resolved in epkgs. Others in this thread have alluded to the fact that this is fundamentally inconsistent, and that is true.

If we want to use the unstable version of some packages, we should override the set consistently,
.nix let emacsPackages = (emacsPackagesNgGen emacs).overrideScope (super: self: { counsel = self.melpaPackages.counsel; counsel-projectile = self.melpaPackages.counsel; }); in emacsPackages.emacsWithPackages (epkgs: with epkgs; [ counsel-projectile counsel ])
This ensures that the packages seen by Emacs are a single consistent set.

This is not very ergonomic and we should discuss how to improve that. For example, we could remove the melpaPackages etc. sets because they should not be used. (Exception: if you are overriding a package that isn't a dependency.)

That said, I think that if we switch to "package from the same set first" approach and make emacsWithPackages fail when it has two versions of the same package, we might end up with a better overall usability (users will need to do less overrides) than now.

The fix-point construction I described above guarantees a consistent package set, so I don't understand what this would be good for.

[oxij]

That said, I think that if we switch to "package from the same set first" approach and make emacsWithPackages fail when it has two versions of the same package, we might end up with a better overall usability (users will need to do less overrides) than now.

The fix-point construction I described above guarantees a consistent package set, so I don't understand what this would be good for.

That's correct. My point, however, is that the current behavior

The original problem is that the following installs the stable version of counsel as a dependency of counsel-projectile:
```.nix
emacsWithPackages (epkgs: (with epkgs.melpaPackages; [ counsel-projectile ]))

while this gives the latest unstable version of `counsel`:
```.nix
emacsWithPackages (epkgs: (with epkgs.melpaPackages; [ counsel ]))

is too much of a surprise for new users of emacsPackagesNg set. E.g.
exactly the same issue was reported before: #12962 (I think there was
another not the same, but a similar one, but I couldn't find it within a
couple of minutes).

My point here:

• Its too surprising.

This ensures that the packages seen by Emacs are a single consistent set.

Except when I override something on a per-package basis by hand without
constructing my own emacsPackages set first (as per your example).

My point here:

• It's still easy to break emacsWithPackages with misconfiguration if
  we don't check consistency in emacsWithPackages.

• The rest of nixpkgs does overriding with overrides. Having a way to do
  it another way can be useful (e.g. "use flags" are very useful,
  despite being rejected from the official nixpkgs). Not properly
  supporting overrides, however, is plain surprising.

I think there should be an option to simply write

emacsWithPackages (epkgs: [
  (epkgs.melpaPackages.evil-org.override { org = epkgs.elpaPackages.org; })
])

(when it doesn't break anything) instead of overriding the whole set.

Btw, we should check the packages were built with the correct emacs
version too. Just in case.

For example, we could remove the melpaPackages etc. sets because they should not be used. (Exception: if you are overriding a package that isn't a dependency.)

I fail to see how that would help with the above points.

[ttuegel]

The rest of nixpkgs does overriding with overrides. Having a way to do it another way can be useful (e.g. "use flags" are very useful, despite being rejected from the official nixpkgs). Not properly supporting overrides, however, is plain surprising.

The Haskell package sets in Nixpkgs have the same problem and use the same solution.

I fail to see how that would help with the above points.

It makes the bad override scenario in the original post impossible to construct.

I think there should be an option to simply write ... (when it doesn't break anything) instead of overriding the whole set.

I would be happy to merge that PR.

[vyp]

(@peterhoeg Forgot to reply to you but for reference, except for emacs itself, I don't use nix for emacs packages so I don't have much else to say about this.)

[peterhoeg]

Thanks @vyp - I got it working - full config available here in case someone else stumbles over this:

https://gist.github.com/ad6870c1004a932368c3c69f3fe321ff

I guess this issue can be closed as "working as intended although not quite obvious for the casual user", @ttuegel ?

[ttuegel]

@peterhoeg Yes, that is how it is intended to work. I am a little surprised you need so many overrides! But, you also have quite a few packages installed in Emacs too, I guess.

[peterhoeg]

The combination of essentially living in emacs and using spacemacs means a lot of packages...

I'll go ahead and close this and I've made a note to update the manual "RealSoonNow(tm)".

[rzetterberg]

Thanks for the help everyone!

I settled on overriding specific packages and their dependencies, then I picked all packages from epkgs instead of a mix of specific package set (such as epkgs.melpaPackages). overrideScope was not available to me (I guess that it's something that was introduced after the 17.03 release), but override worked fine.

Here's what my full package selection looks like, in case someone else stumbles upon this thread and use NixOS 17.03:

  emacsPackages = (emacsPackagesNgGen emacs).override (super: self: {
    counsel            = self.melpaPackages.counsel;
    counsel-projectile = self.melpaPackages.counsel-projectile;
    google-c-style     = self.melpaPackages.google-c-style;
    hamlet-mode        = self.melpaPackages.hamlet-mode;
    ivy                = self.melpaPackages.ivy;
    ob-php             = self.melpaPackages.ob-php;
    ob-restclient      = self.melpaPackages.ob-restclient;
    php-mode           = self.melpaPackages.php-mode;
    projectile         = self.melpaPackages.projectile;
    restclient         = self.melpaPackages.restclient;
    swiper             = self.melpaPackages.swiper;
    undo-tree          = self.melpaPackages.undo-tree;
  });
  emacsP = emacsPackages.emacsWithPackages (epkgs: with epkgs; [
    ace-jump-mode
    counsel
    counsel-projectile
    diminish
    dockerfile-mode
    editorconfig
    elm-mode
    engine-mode
    ess
    evil
    evil-magit
    feature-mode
    go-mode
    google-c-style
    gotham-theme
    guide-key
    hamlet-mode
    haskell-mode
    htmlize
    lua-mode
    magit
    markdown-mode
    neotree
    nix-mode
    ob-php
    ob-restclient
    php-mode
    powerline
    projectile
    request
    restclient
    sass-mode
    terraform-mode
    undo-tree
    use-package
    web-mode
    xterm-color
    yaml-mode
    yasnippet
  ]);

[binarin]

There is an annoying emacsPackageOverrides option, which is a no-op now and probably should be removed to avoid confusion. I've even fixed it initially, before learning about overrideScope in this thread.