Doom-emacs: doom build or doom upgrade stuck on "Waiting for 1 async jobs..." when using emacs native-comp branch.

[h0cheung]

What did you expect to happen?
It should finish building normally.

What actually happened?
It stuck on "Waiting for 1 async jobs...", and the CPU usage of Emacs is almost zero.

Additional details:

• Include a link to your private config
  https://gitlab.com/h-cheung/dotfiles/-/tree/master/doom.d

System information:

Package subdirs is deprecated

((emacs

  (version . "28.0.50")

  (features . "XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GLIB NOTIFY INOTIFY ACL GNUTLS LIBXML2 FREETYPE HARFBUZZ LIBOTF ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES NATIVE_COMP THREADS XWIDGETS LIBSYSTEMD JSON PDUMPER LCMS2")

  (build . "7月 28, 2020")

  (buildopts "--prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games --with-sound=alsa --with-modules --without-gconf --without-gsettings --with-nativecomp --with-x-toolkit=gtk3 --without-xaw3d --without-m17n-flt --with-cairo --with-xwidgets --without-compress-install 'CFLAGS=-march=native -O2 -pipe -fuse-ld=gold' CPPFLAGS=-D_FORTIFY_SOURCE=2 LDFLAGS=-Wl,-O2,--sort-common,--as-needed,-z,relro,-z,now")

  (windowsys . batch)

  (daemonp . server-running))

 (doom

  (version . "2.0.9")

  (build . "HEAD -> develop 092480152 2020-07-30 18:57:53 -0400")

  (dir . "~/.doom.d/"))

 (system

  (type . gnu/linux)

  (config . "x86_64-pc-linux-gnu")

  (shell . "/usr/bin/zsh")

  (uname . "Linux 5.7.9-zen1-1-zen-anbox #1 ZEN SMP PREEMPT Sun, 19 Jul 2020 13:08:36 +0000 x86_64")

  (path "/usr/local/sbin" "/usr/local/bin" "/usr/bin" "/opt/cxoffice/bin" "/var/lib/flatpak/exports/bin" "/usr/lib/jvm/default/bin" "/usr/bin/site_perl" "/usr/bin/vendor_perl" "/usr/bin/core_perl" "~/.local/bin" "/opt/android-sdk/platform-tools/" "~/.emacs.d/bin/" "/usr/lib/emacs/28.0.50/x86_64-pc-linux-gnu"))

 (config

  (envfile)

  (elc-files . 0)

  (modules :completion (company +childframe) (ivy +childframe +icons +prescient) :ui doom doom-dashboard doom-quit hl-todo hydra modeline ophints (popup +all +defaults) pretty-code treemacs vc-gutter vi-tilde-fringe window-select workspaces zen :editor (evil +everywhere) file-templates fold format multiple-cursors parinfer rotate-text snippets :emacs (dired +ranger +icons) electric (ibuffer +icons) undo vc :term eshell vterm :checkers (syntax +childframe) :tools (debugger +lsp) (eval +overlay) gist (lookup +dictionary +docsets) (lsp +peek) (magit +forge) make pass pdf rgb :lang (cc +lsp) emacs-lisp (json +lsp) (markdown +grip) (org +brain +dragndrop +gnuplot +hugo +journal +jupyter +pandoc +pomodoro +present) (python +lsp) qt (sh +lsp) (yaml +lsp) :config (default +bindings +smartparens) :h-cheung vterm-run chinese (telega +auto-method) tools ui cc-extra systemd pkgbuild)

  (packages "n/a")

  (unpin "n/a")

  (elpa "n/a")))

[hlissner]

0c47a03 should fix this issue (thanks to @flatwhatson on Discord). Let me know if that isn't the case and I'll reopen this. Thanks for bringing it to our attention!

[zackteo]

I just encountered this. Had let my computer run for several hours. Fortunately, I opened a separate window to run doom sync and it looked like doom was fine already (opened emacs too) So I closed the other process. Not sure why it was stuck

[ludwj61]

I get the same thing, it hangs at waiting for 1 async job. I don't see any issues from it but it happens on every doom build.

[flatwhatson]

@ludwj61 While it is hanging, please run ps auxf | grep emacs in another terminal. That should show the subprocess which is hanging and allow us to make some progress. Perhaps we need another entry on the compilation blacklist.

[ludwj61]

@flatwhatson This is what I get from running that command after it hangs on the "last" async job:

also thanks for your arch packages!

[flatwhatson]

@ludwj61 Interesting! I can't reproduce this problem, with-editor compiles fine for me, though you're the second person I've seen with the same hang. Would you mind running emacs --batch --eval '(native-compile "~/.emacs.d/.local/straight/build/with-editor/with-editor.el")' to check whether that reports any errors?

[ludwj61]

@flatwhatson sure, this is what I get:

[flatwhatson]

@ludwj61 Hmm, I would expect that file to exist. Have you installed doom into a different location? Perhaps you need to run doom sync first? Try find ~/.emacs.d -name with-editor.el to locate the file.

[ludwj61]

@flatwhatson Sure, I just double checked and I only have doom installed in my home directory as usual. Running the find command gives me this:

The previous compile command still says it can't find anything. And both of these outputs are the same after running doom sync (but I had run doom sync after doom build already).

[lucasweiblen]

I have a similar issue while running doom build.. it hangs with 1800+ async jobs then I have to cancel it and run doom sync then it starts counting down the async jobs but now it has hanged with "waiting for 1 async jobs" as well..

I've done the ps stuff as well:

[flatwhatson]

@ludwj61 Ah, you need to tweak the command a bit, the path is slightly different on your system:

emacs --batch --eval '(native-compile "~/.emacs.d/.local/straight/build-28.0.50/with-editor/with-editor.el")

[ludwj61]

@flatwhatson yep, running that command gives no output instead of the file not found error earlier. Any idea why the path is different? My emacs was built with your AUR package.

[benneti]

The async jobs do not even start (sometimes) when having doom installed as .config/emacs symlinking to .emacs.d seems to fix the issue. But using the XDG directory should be supported, too.

This was before symlinking incase it helps anyone

$ ps auxf | grep emacs
benneti    78607  0.0  0.0   8584  3932 pts/2    S+   09:11   0:00      |   \_ sh /home/benneti/.config/emacs/bin/doom sync
benneti    78609 15.9  2.6 671764 421320 pts/2   Sl+  09:11   0:35      |       \_ emacs --no-site-file --script /home/benneti/.config/emacs/bin/doom -- sync
benneti    78654  5.1  0.0      0     0 ?        Zs   09:11   0:11      |           \_ [emacs] <defunct>
benneti    78810  0.2  0.0      0     0 ?        Zs   09:12   0:00      |           \_ [emacs] <defunct>
benneti    78811  0.8  0.0      0     0 ?        Zs   09:12   0:01      |           \_ [emacs] <defunct>
benneti    78812  0.1  0.0      0     0 ?        Zs   09:12   0:00      |           \_ [emacs] <defunct>
benneti    79510  0.0  0.0   7648  2584 pts/1    S+   09:15   0:00          \_ grep --color=auto emacs

[flatwhatson]

Let's be clear; there are two different and (probably) unrelated kinds of bugs related to hanging doom build.

Hanging with >1000 jobs pending

This occurs reliably when running doom build on a fresh install or after clearing ~/.emacs.d/.local/cache/eln. It's caused by crashed (defunct) worker processes not getting reaped, so the queue manager thinks it can't create any more worker process and the queue stalls. This has been happening for a while, and only occurs when the cache is empty.

Workaround is to Ctrl-c and re-run doom build, it will work on the second attempt.

Hanging with 1 job pending

This occurs because an async worker has hung instead of crashing, so all the async jobs complete except this one. Originally we saw this behaviour with evil-collection-vterm.el, and these recent reports it seems to be happening on with-editor.el.

Workaround is to use ps to get the PID of the hung worker and kill it to allow the build to complete gracefully.

---

@ludwj61 The path is not important, I guess straight.el can choose a different build path for different versions of Emacs. The important thing is that the file was compiled successfully, because no errors were printed. Thanks for testing this, it should help to narrow down the cause of the bug.

---

@benneti I suspect your bug is the "hanging with >1000 jobs" kind described above and not actually related to XDG path. Could you try removing the ~/.emacs.d symlink, clearing the .local/cache/eln directory, running doom build, hitting Ctrl-c when it hangs, then running doom build again?

[benneti]

indeed you are right.

[Thaodan]

Is this maybe a bug in the native-comp branch? There was a similar bug like this yesterday breaking certain packages when using native-comp before https://github.com/emacs-mirror/emacs/commit/898f929215cf644c651abf789b564fcbc50ffbdd was committed.