Nixpkgs: gccemacs packaging

[bhipple]

In the emacs world, recently there's been a lot of interest in trying to improve performance of elisp, including adding some threading support, adding libjansson for C JSON serialization/deserialization, etc. One such project in this camp is gccemacs, which adds native compilation to elisp files with libgccjit:
http://akrl.sdf.org/gccemacs.html
https://lists.gnu.org/archive/html/emacs-devel/2019-11/msg01137.html

We now have a working libgccjit as of #75288, and I have a basic gccemacs working here in my NUR:
https://github.com/bhipple/nur-packages/blob/master/pkgs/gccemacs/default.nix

This appears much snappier than my previous emacs, but there are some issues for improvement. This ticket will track some of them:

1. The deferred compilation of .eln files does not work for elpa/melpa packages provided by the system, since they are in the nix store and owned by root. Investigate if there's a way to make the deferred eln compilation go into the user's $XDG_CACHE_DIR.

2. When building the system packages, it would be nice to do AOT eln compilation on all the plugins when building the packages in Nix, like we currently do for elc files. Investigate if there's a cmdline arg to build a package.

3. This project is probably not appropriate to build into NixPkgs master itself until it gets into upstream emacs master, but I'd like to integrate it with https://github.com/nix-community/emacs-overlay so users can use it in their NUR overlays easily.

4. I'd like to setup cachix for pre-compiled binaries.

[bhipple]

CC @AndreaCorallo, do you have an opinion on items (1) and (2)? Specifically, I'd like a hook to set where to put the eln files in the case where the user's emacs packages are not writable for the user, because they were installed by the system package manager:
https://github.com/emacs-mirror/emacs/blob/feature/native-comp/lisp/emacs-lisp/comp.el#L460-L466

It's possible that with a good solution for (2), we don't need (1) at all, since if the system package manager builds the package it can build the eln file too. I haven't looked into this too much, but presumably there's inspiration we can draw from how elc files are handled?

[adisbladis]

I think the emacs overlay is a better place for this package until there is a stable release.

There we already have infrastructure to keep things automatically up to date and we don't have to pollute nixpkgs with highly experimental packages.

Also we have a Hydra instance pushing to the nix-community binary cache.

[bhipple]

Agreed (see point (3) above)! I'm actually using the emacs overlay pretty heavily myself, and once I get this fully working in the bhipple NUR I'll send a PR.

I didn't realize we have a full blown Hydra and binary cache for this; that's awesome!

[matthewbauer]

For gccemacs, I have this at the end of emacs/wrapper.nix:

  for dir in $out/share/emacs/site-lisp/elpa/*; do
    rm -f $dir/*.elc
    pushd $dir
    emacs --batch --quick \
          --eval \
          "(let ((default-directory \"$out/share/emacs/site-lisp\"))
          (normal-top-level-add-subdirs-to-load-path))" \
          -L . -f batch-native-compile $(printf '%s\n' $dir/*.el | grep -v \\\(-autoloads\\.el$\\\|-pkg\\.el$\\\)) || true
    popd
  done

This compiles native versions of all of the packages in elpa. In the future, we'd probably want to do batch-native-compile for every Emacs package.

It takes a long time to compile but it looks like it works okay. The only issue with gccemacs I've found is that startup is really really slow even without comp-deferred-compilation. Previously it takes like ~1s to start Emacs, with gccemacs it's now like ~4s.

[AndreaCorallo]

CC @AndreaCorallo, do you have an opinion on items (1) and (2)?

I believe should be possible to AOT compile all packages.

batch-native-compile should help you with that and if it does not we have to consider it a bug.

For (2) I think you already got an option.

[AndreaCorallo]

It takes a long time to compile but it looks like it works okay. The only issue with gccemacs I've found is that startup is really really slow even without comp-deferred-compilation. Previously it takes like ~1s to start Emacs, with gccemacs it's now like ~4s.

This is quite unexpected, in my experience and people I know the startup time is about the same. Does NixOS has something peculiar that can influence load time of shared libraries (maybe dynamic linker related)?

[matthewbauer]

The two things I can think of that might be different from other distros:

• I have tons of ELPA packages in my config (125 from /elpa/ in load-path + builtin paths), which get doubled when we add eln-* suffix to each one.
• NixOS disables ld.so.cache: https://github.com/NixOS/nixpkgs/blob/3155fbff0adcd8abf762a5689de4af1c7b93b974/pkgs/development/libraries/glibc/common.nix#L62

[AndreaCorallo]

Mmhh it's very hard to tell from here... Lots of components are involved including file system and dynamic linker.

AFAIK it should not be slower to start even with many packages, but I guess we'll get more data point in the future to understand this better.

Maybe you can have a profile run to get an idea on where do we loose so much time.

[bhipple]

For reference, I have not yet noticed any startup slowness in my setup -- though since I use an emacs daemon and emacsclient, I've only started it up once or twice, and since that daemon boots with systemd user mode at startup it's usually up by the time I look at it ;)

[mjlbach]

Now that libgccjit in is unstable, one issue is for those of us not using nix to manage our emacs packages. I use doom and straight.el; gccemacs seems to require the following in LIBRARY_PATH in order to compile any packages:

pkgs.lib.getLib pkgs.stdenv.cc + /lib
pkgs.lib.getLib pkgs.stdenv.glibc + /lib
pkgs.lib.getLib pkgs.libgccjit + /lib/gcc/x86_64-unknown-linux-gnu/9.3.0

we could wrap emacs, but as @adisbladis mentioned via IRC this has the negative side effect of polluting the LIBRARY_PATH of any application launched by emacs.

[Mic92]

Now that libgccjit in is unstable, one issue is for those of us not using nix to manage our emacs packages. I use doom and straight.el; gccemacs seems to require the following in LIBRARY_PATH in order to compile any packages:

pkgs.lib.getLib pkgs.stdenv.cc + /lib
pkgs.lib.getLib pkgs.stdenv.glibc + /lib
pkgs.lib.getLib pkgs.libgccjit + /lib/gcc/x86_64-unknown-linux-gnu/9.3.0

we could wrap emacs, but as @adisbladis mentioned via IRC this has the negative side effect of polluting the LIBRARY_PATH of any application launched by emacs.

Is it not possible to put it in the RPATH of emacs?

[adisbladis]

A few updates on this topic:

• General build infrastructure support for AOT compilation has been added to nixpkgs: https://github.com/NixOS/nixpkgs/pull/93716.
• Which means we're now AOT compiling if the native-comp branch is used.
• This support is used in the overlay packaging.
• A hydra jobset has been added for native packages.

@AndreaCorallo You may be especially interested to take a look at the Hydra jobset at https://hydra.nix-community.org/jobset/emacs-overlay/unstable-gcc-pkgs. It's probably the biggest test of native-comp that exists currently.

[adisbladis]

Is it not possible to put it in the RPATH of emacs?

No, binutils & friends are still called.

[bhipple]

@AndreaCorallo I'm not sure where you'd like gccemacs bug reports, but I found an issue: the byte-compile function respects a file-local var no-byte-compile: t, like so:

https://www.gnu.org/software/emacs/manual/html_node/elisp/Byte-Compilation.html
https://github.com/emacs-mirror/emacs/blob/878924e881528f8b87216f571db91a960c733d9a/lisp/emacs-lisp/bytecomp.el#L1818-L1823

The native-compile does not respect this and compiles files that it shouldn't, like this one:

https://github.com/emacs-mirror/emacs/blob/878924e881528f8b87216f571db91a960c733d9a/lisp/org/org-version.el

which results in a bunch of bugs when we have a native compiled org-version on the "wrong" version (e.g., https://github.com/raxod502/straight.el/issues/211)

---

I have a temporary workaround inside NixPkgs here:

diff --git a/pkgs/build-support/emacs/generic.nix b/pkgs/build-support/emacs/generic.nix
index 956787ad59e..42cd996c0fb 100644
--- a/pkgs/build-support/emacs/generic.nix
+++ b/pkgs/build-support/emacs/generic.nix
@@ -60,8 +60,19 @@ stdenv.mkDerivation ({

   LIBRARY_PATH = "${lib.getLib stdenv.cc.libc}/lib";

+  # FIXME: This should be handled upstream in gccemacs by having
+  # (native-compile) respect the file local variable no-byte-compile the way
+  # (byte-compile) does:
+  # https://www.gnu.org/software/emacs/manual/html_node/elisp/Byte-Compilation.html
   postInstall = ''
-    find $out/share/emacs -type f -name '*.el' -print0 | xargs -0 -n 1 -I {} -P $NIX_BUILD_CORES sh -c "emacs --batch -f batch-native-compile {} || true"
+    for f in $(find $out/share/emacs -type f -name '*.el'); do
+        if grep -q " no-byte-compile: t" $f; then
+            echo "Skipping native compilation of $f"
+            continue
+        fi
+        echo "Native compiling $f"
+        sh -c "emacs --batch -f batch-native-compile $f || true"
+    done
   '';

 }

I took a look at contributing a fix upstream, but am not particularly familiar with the emacs code here so figured I'd raise the issue first with a workaround in case anyone else hits it.

[AndreaCorallo]

Hi @bhipple
thanks for the interesting observation!

The right place to report bugs or suggest patches is the Emacs bug tracker:
https://www.gnu.org/software/emacs/manual/html_node/efaq/Reporting-bugs.html

Please file a bug here adding feature/native-comp in the subject, (eg https://debbugs.gnu.org/cgi/pkgreport.cgi?package=emacs;include=subject%3Anative-comp)

[tadfisher]

The existing Emacs package builder doesn't seem to work, as comp-eln-load-path isn't populated in the builder sandbox. Here's a PR toward making it work again: https://github.com/NixOS/nixpkgs/pull/104010