Doom-emacs: [BUG] `doom update` fails with timeouts, hangs and/or errors

[utrack]

Describe the issue
bin/doom re hangs for a long time, then fails for some of the packages.

End of output:

  .
  .
  .
  ! A thread has timed out. The following packages were skipped: restclient, company-restclient, company-shell, mu4e-maildirs-extension, calfw-org, org-gcal, synosaurus, mixed-pitch, wordnut, visual-fill-column, avy, ace-link, memoize
  . 72%

Output:
/dev/null

--------------------------------------------------

There was an unexpected error:
  error (error)
Wrong type argument: listp, "There were errors:

   - bui: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')
   - tree-mode: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')
   - spinner: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')
   - ht: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')
   - epl: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')
   - pkg-info: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')
   - cl-lib: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')
   - loop: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')
   - elisp-refs: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')
   - f: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')
   - s: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')
   - dash: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')
   - parent-mode: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')"

doom/info:

emacs   version    26.2
        features   XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS GLIB NOTIFY ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD LCMS2
        build      Apr 12, 2019
        buildopts  (--prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --with-x-toolkit=gtk3 --with-xft --with-modules 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt' CPPFLAGS=-D_FORTIFY_SOURCE=2 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now)
doom    version    2.0.9
        build      HEAD -> develop, origin/develop, origin/HEAD cbd3a3b2 2019-08-28 00:40:50 -0400
system  type       gnu/linux
        config     x86_64-pc-linux-gnu
        shell      /bin/zsh
        uname      Linux 5.1.14-arch1-1-ARCH #1 SMP PREEMPT Sat Jun 22 16:28:48 UTC 2019 x86_64
        path       (~/.bin ~/go/bin /usr/local/bin /usr/local/sbin ~/.bin ~/go/bin /usr/local/bin /usr/local/sbin /usr/local/bin /usr/bin /bin /usr/local/sbin /usr/lib/jvm/default/bin /usr/bin/site_perl /usr/bin/vendor_perl /usr/bin/core_perl /usr/lib/emacs/26.2/x86_64-pc-linux-gnu)
config  envfile    nil
        elc-files  32
        modules    (:completion company (ivy +childframe +fuzzy) :ui doom fill-column hl-todo modeline nav-flash ophints (popup +all +defaults) (pretty-code +fira) treemacs vc-gutter vi-tilde-fringe window-select workspaces :editor evil file-templates fold (format +onsave) multiple-cursors rotate-text snippets word-wrap :emacs dired electric vc :term term :tools debugger editorconfig eval flycheck (lookup +docsets) lsp magit make pdf :lang data emacs-lisp ess go latex markdown (org +dragndrop +present) plantuml rest sh :email mu4e :app calendar (write +wordnut) :config default)
        packages   (ivy-rich counsel-tramp dap-mode zoom evil-snipe evil-visual-mark-mode outline-magic outshine poporg org-jira)
        elpa-packages (n/a)

[hlissner]

This is a known issue that has no elegant solution atm. For now, here are a few things to try:

• Run doom up multiple times
• Run doom purge -r to see if its cleanup process will fix the issue
• If it continues to hang, you'll need to delete the affected plugins manually from ~/.emacs.d/.local/straight/repos/ then run bin/doom refresh -f to reinstall them (this will update them as well)

[smypf]

Not sure if this results in the same outcome, but running the straight-pull-all command in emacs then running doom refresh in terminal resolves this issue.

[myrgy]

I faced the same issue.
Issue was fixed only after ~/.emacs.d/.local/straight/repos/affected-repo and bin/doom refresh -f

doom up and doom purge -r didn't help

[hlissner]

Folks have reported several issues with doom upgrade.

I am aware of these and am working on a solution, and have decided to use this issue as a catch-all for them. For the time being, here are the common issues and their respective workarounds:

doom upgrade hangs indefinitely while updating some packages

You'll need to delete said package(s) manually from ~/.emacs.d/.local/straight/{build,repos}/, then run doom refresh -f.

EDIT: If this doesn't work, you'll need to delete ~/.emacs.d/.local/straight/build-cache.el as well, then run doom build -f.

"No upstream configured for branch" errors

Example:

There was an unexpected error:
  error (error)
Wrong type argument: listp, "There were errors:

   - company-anaconda: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')
   - anaconda-mode: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')"

Run this for each problem package:

cd ~/.emacs.d/.local/straight/repos/PACKAGE
git branch -u origin/master
doom upgrade

If that doesn't work, then:

1. Delete the package from both ~/.emacs.d/.local/straight/build and ~/.emacs.d/.local/straight/repos
2. Delete ~/.emacs.d/.local/straight/build-cache.el
3. Run doom refresh -f

If that still doesn't work, delete ~/.emacs.d/.local and run doom refresh -f.

Threads time out frequently.

Example:

 .
  .
  ! A thread has timed out. The following packages were skipped: melpa, gnu-elpa-mirror, emacsmirror-mirror, straight, bind-key, async, dotenv-mode, auto-minor-mode, all-the-icons, hide-mode-line, highlight-numbers, rainbow-delimiters, restart-emacs, better-jumper, dtrt-indent, helpful, ns-auto-titlebar, pcre2el, smartparens, so-long, osx-clipboard, undo-tree, ws-butler, projectile
  . 11%
  . 22%
  . 33%
  . 44%

For this, you'll need to either:

• Increase the timeout (from the default 45s): doom upgrade -t 160 or
• Delete the packages listed in the "thread has timed out" message from ~/.emacs.d/.local/straight/{build,repos}/, then run doom refresh -f.

Doom throws "file-missing" errors at startup after doom upgrade

This means the autoloads files weren't generated correctly. doom refresh -f should address this. If not, run doom rebuild -f

[mattsawyer77]

FWIW, to get around the undo-tree problem mentioned above, I did this and it worked OK:

git clean -xdf
bin/doom install

[rscircus]

Ran into the undo-tree issue after doom update.

Solution:

• delete: ~/.emacs.d/.local/straight/build/undo-tree
• doom rebuild -f

Here is my line of thought:

I wondered what git clean -xdf does. It does the following for the curious of minds:

• git clean: removes untracked files from directory,
• -d: remove directories, too,
• -x: ignore the standard ignore rules, ha!,
• -f: shoot, ask questions later.

which didn't solve the issue for me, though. However, M-x, straight-pull-all, followed by doom refresh in the CLI - which is similar to @hlissner's solution afais - gave a better hint:

I decided to reset (r) to origin/master. Didn't work. doom update still threw the same error. However, things diverged in this repo somehow. Hence I constructed the solution above using @hlissner's hints. Which worked. There may come other packages with a similar error... rinse, repeat.

[Emantor]

FWIW I ran into the same issue. The history within ~/.emacs.d/.local/straight/build/undo-tree contained only cc7354c which is not related to the upstream history at all. I added the upstream branch via git branch --set-upstream-to=origin/master master and than reset the branch (thereby throwing away the commit, I retrieved the commit via reflog) with git reset --hard origin/master, afterwards doom update ran correctly.

[hlissner]

undo-tree's remote was changed from http://www.dr-qubit.org/git/undo-tree.git to https://github.com/emacsmirror/undo-tree a while ago. This is what caused the issues for this plugin, in particular -- straight and Doom can't deal with recipe changes atm, and git clean alone will not fix that. A complete git-based solution would is more complicated than the solution I offered:

You'll need to delete said package(s) manually from ~/.emacs.d/.local/straight/{build,repos}/, then run doom refresh -f.

[mattsawyer77]

@hlissner This just happened to me again with company-anaconda and anaconda-mode.

There was an unexpected error:
  error (error)
Wrong type argument: listp, "There were errors:

   - company-anaconda: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')
   - anaconda-mode: ((error Failed to run \"git\"; see buffer *straight-process*) fatal: no upstream configured for branch 'master')"

So this time I tried doing

rm -rf .local/straight/{build,repos}/company-anaconda
rm -rf .local/straight/{build,repos}/anaconda-mode
bin/doom refresh -f

but the subsequent bin/doom update fails again with the same error message as before.

[rscircus]

@mattsawyer77 - that's what I meant with rinse repeat above...

In the end I was so annoyed, that I deleted all subdirectories in {build,repos} and doom rebuild -f again. That solved everything for me.

So the final solution is:

• rm -rf ~/.emacs.d/.local/straight/{build,repos}/*
• ~/.emacs.d/bin/doom rebuild -f

After that ~/.emacs.d/bin/doom refresh should work fine.

[hlissner]

Ah, it turns out that the recipe for these anaconda packages (upstream, on MELPA) have changed sometime in the past month or two. In that case, you'll need to delete ~/.emacs.d/.local/straight/build-cache.el as well. I'll update my original post.

[mattsawyer77]

@hlissner it seems something's still not quite right. When I got this error, I did not have the build-cache.el file. I went ahead and removed the whole straight directory and reinstalled everything; now all is well again.

[hlissner]

As of 873fc5c this issue should finally be resolved. Since it changes a fair bit of update logic, it may be a safer bet to update doom with the following:

cd ~/.emacs.d
bin/doom clean
git pull --rebase
bin/doom refresh -f
bin/doom update

doom update is now synchronous and will display more information about the update process than it did before. I've also modified straight to be able to prompt you for decisions, which should hopefully replace these unhelpful errors with a prompt.

Let me know if the issue hasn't been resolved and I'll reopen this. Thanks for bringing it to my attention and for putting up with these issues for so long!

[rscircus]

One gets asked stuff like this, with the recipe above:

  - (19/245) so-long is up-to-date
  > In repository "undo-tree", remote "origin" has URL
    "https://github.com/emacs-straight/undo-tree.git"
  but recipe specifies a URL of
    "https://github.com/emacsmirror/undo-tree.git"

       1) Abort
       2) Rename remote "origin" to "origin-1", re-create "origin" with correct URL, and fetch
       3) Rename remote "origin" manually, re-create it with correct URL, and fetch
       4) Delete remote "origin", re-create it with correct URL, and fetch
       5) Delete remote "origin", re-create it with manually set URL, and fetch

    How to proceed? (1, 2, 3, 4, 5) 

I went with 4 followed by a 3 in the next question:

> In repository "undo-tree", branch "master" has diverged from "origin/master"

       1) Abort
       2) Merge "origin/master" to branch "master"
       3) Reset branch "master" to "origin/master"
       4) Rebase branch "master" onto "origin/master"

    How to proceed? (1, 2, 3, 4)

[rileyrg]

Having similar issues. Nightmare. Keep getting errors saying git cant run on boxquote.el and dap-mode. Trying the complete wipe solution and will get back.