Doom-emacs: [BUG] doom refresh fails in Windows

[r-darwish]

What did you expect to happen?
doom refresh should work in Windows.

What actually happened?
I got this:
```x There was an unexpected error:
Type: file-error
Message:
File error
Data:
"Spawning child process"
"Exec format error"

Run the command again with the -d (or --debug) switch to enable debug
mode and (hopefully) generate a backtrace from this error:

doom -d refresh

If you file a bug report, please include it!

DOOM Running doom hook: doom-init-theme-h
DOOM Running doom hook: window-divider-mode
Searching for program: xclip, no such file

But after the xclip patch I still get this

Loading package general...
Loading package general...done (0.024s)

Recompiling stale elc files...
Recompiled 0/0 file(s) (168 ignored)
x There was an unexpected error:
Type: file-error
Message:
File error
Data:
"Spawning child process"
"Exec format error"

Run the command again with the -d (or --debug) switch to enable debug
mode and (hopefully) generate a backtrace from this error:

doom -d refresh

If you file a bug report, please include it!

DOOM Running doom hook: doom-init-theme-h
DOOM Running doom hook: window-divider-mode
```

Which is very similar, but without any message indicating what's the bad executable.

Additional details:
https://github.com/r-darwish/.doom.d

I know that Windows isn't "tier 1" in Doom so I'll gladly fix the problem and open a PR. However, since I don't see error message this time I'd be happy to get some advise about how to debug it.

Steps to reproduce:
Run doom refresh

System information:

((emacs

  (version . "26.2")

  (features . "XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS THREADS LCMS2")

  (build . "Apr 13, 2019")

  (buildopts "--without-dbus --host=x86_64-w64-mingw32 --without-compress-install 'CFLAGS=-O2 -static -g3'")

  (windowsys . batch)

  (daemonp))

 (doom

  (version . "2.0.9")

  (build . "HEAD -> develop, origin/develop, origin/HEAD 0d4315960 2020-01-06 15:08:15 -0500")

  (dir . "c:/Users/roey/.doom.d/"))

 (system

  (type . windows-nt)

  (config . "x86_64-w64-mingw32")

  (shell . "C:/Users/roey/scoop/apps/emacs/current/libexec/emacs/26.2/x86_64-w64-mingw32/cmdproxy.exe")

  (uname . "n/a")

  (path "c:/Program Files (x86)/Intel/iCLS Client/" "C:/Program Files/Intel/iCLS Client/" "C:/Windows/system32" "C:/Windows" "C:/Windows/System32/Wbem" "C:/Windows/System32/WindowsPowerShell/v1.0/" "C:/Program Files (x86)/NVIDIA Corporation/PhysX/Common" "C:/Program Files (x86)/MiKTeX 2.9/miktex/bin/" "C:/Program Files (x86)/LyX 2.2/Perl/bin" "C:/WINDOWS/system32" "C:/WINDOWS" "C:/WINDOWS/System32/Wbem" "C:/WINDOWS/System32/WindowsPowerShell/v1.0/" "C:/Program Files (x86)/Intel/Intel(R) Management Engine Components/DAL" "C:/Program Files/Intel/Intel(R) Management Engine Components/DAL" "C:/Program Files (x86)/Intel/Intel(R) Management Engine Components/IPT" "C:/Program Files/Intel/Intel(R) Management Engine Components/IPT" "C:/ProgramData/chocolatey/bin" "C:/Program Files (x86)/vim/vim80" "C:/WINDOWS/System32/OpenSSH/" "C:/Program Files/Calibre2/" "C:/Program Files (x86)/Windows Kits/8.1/Windows Performance Toolkit/" "C:/Program Files/NVIDIA Corporation/NVIDIA NvDLISR" "C:/WINDOWS/system32" "C:/WINDOWS" "C:/WINDOWS/System32/Wbem" "C:/WINDOWS/System32/WindowsPowerShell/v1.0/" "C:/WINDOWS/System32/OpenSSH/" "C:/Program Files/PowerShell/6/" "C:/Users/roey/scoop/apps/python/current" "C:/Users/roey/scoop/apps/python/current/Scripts" "C:/Users/roey/AppData/Local/Programs/Python/Launcher/" "C:/Users/roey/scoop/shims" "C:/Users/roey/.cargo/bin" "C:/Users/roey/AppData/Local/Microsoft/WindowsApps" "C:/Program Files/Oracle/VirtualBox" "C:/ProgramData/emacs/bin" "C:/Users/roey/.local/bin" "C:/Users/roey/AppData/Roaming/Python/Python37/Scripts" "C:/Users/roey/AppData/Local/Microsoft/WindowsApps" "c:/Users/roey/scoop/apps/emacs/current/libexec/emacs/26.2/x86_64-w64-mingw32"))

 (config

  (envfile)

  (elc-files . 0)

  (modules :completion company (ivy +childframe +icons) :ui doom doom-dashboard hl-todo hydra modeline nav-flash ophints (popup +all +defaults) vc-gutter vi-tilde-fringe window-select :editor (evil +everywhere) file-templates fold (format +onsave) lispy multiple-cursors rotate-text snippets :emacs dired electric ibuffer vc :tools (eval +overlay) flycheck (lookup +docsets) lsp macos magit :lang data emacs-lisp (java +lsp) javascript markdown (org +dragndrop +present) python rust sh web :config (default +bindings))

  (packages highlight-symbol git-link syntax-subword)

  (elpa "n/a")))

[hlissner]

Hmm. That isn't a very helpful error message, and I assume that it isn't producing a backtrace? I'm afraid I have no idea where to go from here.

[r-darwish]

There's no traceback and -d doesn't help much. I'll try to add some print statements until I find something useful

[r-darwish]

@hlissner is there any way to run doom refresh from a running instance of Emacs, so that toggle-debug-on-error will pop up on errors?

[r-darwish]

That seems to be caused by https://github.com/raxod502/straight.el/issues/393.

Adding (print! (with-current-buffer "*straight-process*" (buffer-string))) helped debugging. Maybe it's best to always do that if -d is specified?

Anyway, the suggested mitigation in https://github.com/raxod502/straight.el/issues/393 is to set exec-path before loading straight. This is currently something that can't be done in ~/.doom.d and have to be set by editing ~/.emacs.d/bin/doom. Do you happen to have any suggestion about how can Doom provide a mechanism to allow setting exec-path?

[hlissner]

Perhaps by setting PATH directly while running bin/doom. e.g.

PATH="some/path:$PATH" doom refresh

[dp12]

I ran into the same problem and had to do the following hack to straight.el to get around it.

diff --git a/straight.el b/straight.el
index 93f137d..ff0982a 100644
--- a/straight.el
+++ b/straight.el
@@ -3584,7 +3584,7 @@ modified since their last builds.")
                           (append (list "-o"
                                         "-path"
                                         (expand-file-name
-                                         "*" (straight--repos-dir local-repo))
+                                         "\"*\"" (straight--repos-dir local-repo))
                                         newer-or-newermt
                                         mtime-or-file
                                         "-print")

[skywind3000]

@dp12 I still have "Exec format error" after applying your patch manually to:

C:\Users\UserName\.emacs.d\.local\straight\repos\straight.el\straight.el

Did I edit the wrong file ??

[hlissner]

@skywind3000 You must rebuild the package with doom build -r. This byte-compiles and symlinks straight's files to .local/straight/build/straight/.

If you don't see it rebuilding straight on doom build -r, try doom build instead.

[skywind3000]

@hlissner , after running doom build, I can confirm that straight.el in .local/straight/build/straight/ has been changed, but it still print "Exec format error".

[nikodemus]

I had the same issue and can confirm this fixed it for me.

Specifically:

1. Initially I had the issue described in https://github.com/hlissner/doom-emacs/issues/2323 with the complaint "/usr/bin/find: paths must precede expression: `c:/Users/myname/.emacs.d/.local/straight/repos/link-hint.el/README.org'"
2. Edited C:\Users\myname\.emacs.d\.local\straight\repos\straight.el\straight.el as per patch above.
3. Ran bin/doom build -r which reported "No packages need rebuilding".
4. Ran bin/doom build which rebuilt straight.

After these steps bin/doom sync passed, and changes in .doom/init.el take effect.

[ctm22396]

I was getting an "Exec format error" when trying to sync. I would get through a doom build call without any trouble, but doom sync gave me trouble sometime during the "(Re)building packages..." stage.

I traced the source to the file core/cli/packages.el in the function doom-cli-packages-build. There are two function calls wrapped in unless:

(unless force-p
    (straight--make-build-cache-available)
    (straight--make-package-modifications-available))

There's a find call in the straight--make-package-modifications-available with many arguments. I apologize for my lack of knowledge, but it might be an error in the way the arguments are constructed that is leading to the "Exec format error". I believe that is what was targeted by the fix suggested above with the "\"*\"". However, this fix did not solve the problem for me.

As it is not run when force-p is true, I commented out the code and doom sync was able to run for me without error.

So to be clear my solution was to comment out the preceding code from core/cli/packages.el in the function doom-cli-packages-build

[r-darwish]

Hopefully this will be resolved once https://github.com/raxod502/straight.el/pull/473 is merged

[hlissner]

With raxod502/straight.el/pull/473 merged this issue should be resolved. A doom upgrade should fix it. If you have any trouble, delete ~/.emacs.d/.local/straight/{build,repos}/{straight,straight.el}, then runu doom sync.

Let me know if the issue resurfaces and I will reopen the issue. Thanks for bringing it to my attention!

[Bram-Boris]

Hi,

I think I still have this issue using Emacs 26.3 on Windows using a fresh install of doom-emacs.
It looks related so I commented it here. Let me know if you need more information.

> doom refresh
> Synchronizing your config with Doom Emacs...
  > Regenerating envvars file at "c:/Users/bram.boris.meerlo/.emacs.d/.local/env"
  - Scraping shell environment
    - Ignoring PROMPT=$P$G
        Successfully generated "c:/Users/bram.boris.meerlo/.emacs.d/.local/env"
  > (Re)generating core autoloads...
    > Generating core autoloads...
    > Byte-compiling core autoloads file...
        Generated .local/autoloads.elc
  > Installing packages...
    - No packages need to be installed
  > (Re)building packages...
There was an unexpected error:
  Type: file-error
  Message:
    File error
  Data:
    "Spawning child process"
    "Exec format error"

Debug mode
I have created a gist

[ArturMroz]

I'm also still having this issue on a fresh doom install, however, @ctm22396's fix worked for me (@dp12's fix used to work before but stopped recently).

[ctm22396]

I'm noticing that the error is dependent on which modules I have activated in my .doom.d/init.el. treemacs and neotree always breaks it. If I comment them out and refresh, I do not get the error and the refresh process completes successfully. Perhaps there are some paths in certain packages that make the find command invalid?

In any case, it's still either directly or indirectly an upstream error in straight.el.

[skywind3000]

I encounter another issue when I was trying to reinstall doom after straight's patch merged:

https://github.com/hlissner/doom-emacs/issues/2513

Now, "doom install" will become unresponsive if you are trying to reinstall it in Windows.

[rjl6789]

@ctm22396 can I just ask you to confirm how you implement your work around. Is it:

comment out as below in function doom-cli-packages-build from file core/cli/packages.el

;;(unless force-p
;;    (straight--make-build-cache-available)
;;    (straight--make-package-modifications-available))

then doom build -r followed by doom sync ? - I've also tried a doom refresh - and your fix seems to quel all errors. is that correct? which command makes force-p true? Sorry - new to this

EDIT: I should add that making changes in the ~\.doom.d\init.el was implemented when I did a doom refresh with the changes to core\cli\packages.el

[ctm22396]

@rjl6789

That's correct. I believe force-p is set to t when you run bin/doom build without the -r flag. When force-p is nil, the doom-cli-packages-build function appears to use the straight package to find packages in your emacs ecosystem that are outdated so that they can be rebuilt. When it is t, the command just (re)builds every package. In fact a better workaround than the one I suggested would be to set force-p to t during the reload/refresh/sync call, so that you ensure that all packages are rebuilt. However, it takes quite some time to have them all rebuilt.

EDIT: I feel like there must be a way to implement this selective rebuilding that doesn't rely on find, which seems to be causing a lot of trouble on Windows

[xeijin]

@ctm22396 haven't tried your fix yet, but I have the gnu version of find.exe in my path now and still get the error, which is strange as it was resolved after straight.el fixed the issue on their side

P:\home\.emacs.d\bin>where find
P:\apps\cmder-full\current\vendor\git-for-windows\usr\bin\find.exe
C:\Windows\System32\find.exe
P:\shims\find.exe

[maciekk]

FWIW, I too came here looking for help with exact same problem (Windows 10 "doom refresh" fails with above error messages), and @ctm22396 mitigation worked for me as well.

[skywind3000]

@xeijin, I solved this problem by installing another "find.exe" (with scoop install findutils) and set the scoop binary path before c:\windows\system32 temporary when using doom refresh:

set "PATH=%USERPROFILE%\scoop\shims;%PATH%"

And without installing somepackages like cc, my doom refresh works correctly now.

btw: I haven't applied @ctm22396's patch, just change to another find tool, but if I enable some packages like cc , it will break again.

Is it a helpful information for debugging ??

[hlissner]

@skywind3000 @maciekk @xeijin @ctm22396 @Bram-Boris @rjl6789 Another issue has been made for your Exec format error issue. See #2556.