Doom-emacs: Consider using straight as the package manager

[rgrinberg]

I'm currently a user of spacemacs, but spacemacs is quite heavy, buggy, and contains lots of cruft I will never use. The layer system in particular is a bit of a nightmare.

But my biggest complaint with spacemacs is that it doesn't use https://github.com/raxod502/straight.el/. I'd really like to use straight as my package manager because I want to use lock files to secure my configuration against random breakage. I'd like a convenient workflow for contributing upstream. And all that other git goodness that comes with having access to the full git history of every package. If doom were to use straight, I'd definitely switch from spacemacs.

[hlissner]

I hear that. I already have straight.el in my crosshairs, as well as a local branch where I'm experimenting with straight.el integration. However, I'm waiting on some key features promised for the v1.0 release (of straight.el) before I can move forward with it. I can't predict when that'll be, but I'll keep this issue open and keep you posted.

[rgrinberg]

Would it be possible to list which features are still missing in straight? I'd like to make sure that there's progress on those and perhaps work on some of them myself.

[hlissner]

I haven't looked at straight for a while, but at the time this issue was created, what Doom needed most (from straight) was a non-interactive workflow for updating packages (straight-pull-packages minus the prompt for each and every package) and GNU ELPA support.

I had a couple other misgivings about poor error handling, confusing pull/push prompts when you run straight-freeze-versions, and unimplemented support for VCSes other than git, but I'm not sure if these issues have been addressed yet.

This week is really busy for me. After the 16th I'll revisit straight and see where we stand.

[rgrinberg]

All those issues are straight forward except for:

and unimplemented support for VCSes other than git

Is this really a blocker though? I don't think there are any issues serious packages out there that aren't hosted on git.

[hlissner]

You're right, it's not really an issue. I was under the impression that there were (relatively common) non-git/svn dependencies out there, which straight would presumably be unable to fetch, much less manage (and so, Doom wouldn't be able to either). Not sure where I got that idea, I'll have to dig around straight's issue tracker.

[hlissner]

Adopt straight.el as our package manager

[hlissner]

After a long wait, it's done! I published the straight branch a short while ago, which replaces package.el and quelpa with straight.el. This sidesteps all the recent gnutls and gnu elpa issues (#1582 and #1566).

I'll be spending the next week ironing out the kinks before merging it in. It'd be a huge help if you could give a try. Here are a few things you should know going in:

1. Make a backup of your old .emacs.d! You have to reinstall all your packages. Simply switch branches, delete ~/.emacs.d/.local, and doom refresh away!

2. Two naming conventions for functions/variables have changed

   • Hooks were doom|hook and are now doom-hook-h
   • Advice functions were doom*advice and are now doom-advice-a

3. doom quickstart is now doom install (though quickstart is still
   around, it is deprecated).

4. doom compile :plugins is now doom build -f

5. doom env will now do what doom env refresh did. doom env clear still exists too, but doom env {auto,enable} are gone (doom refresh will always regenerate your envvar file if it exists).

6. The package management system is young. Occasionally, Doom may throw weird errors shortly after startup following package updates. If that happens, try doom autoloads followed by doom rebuild all. It doesn't happen often, fortunately.

7. I haven't employed lockfiles or pinned any packages yet. That comes after documentation.

8. If you have package! declarations with a :recipe, the recipe format has changed ever so slightly. You'll find more about it on straight's project page, but it usually won't amount to any more than changing :fetcher to :host.

9. def-package! is now use-package!. This is to make it absolutely clear that this macro is simply a thin wrapper around use-package!.

---

For Doom users that have been with us for a while, this may cause some breakages. Sorry for the trouble, but the good news is, these are all the breaking changes I have in store.

Let me know if it gives you any trouble. I will keep this thread open until straight makes it to develop.

[credmp]

Don't know if this is the right place for this, but I tested this branch on Ubuntu 18.04 LTS with a clean Doom Emacs install. It works correctly, the gnutls.el errors (invalid request) are gone.

[ydewit]

I just switched to straight, rm -rf .local/ and ran ./bin/doom refresh but got the following message:../../../autoloads.el:1410:1:Error: Symbol’s value as variable is void: next`

> Checking core autoloads file
  > Regenerating core autoloads file
  ✓ Scanned 66 file(s)
  ✓ Generated new autoloads.el
  ✓ Expanded module autoload paths
  ✓ Generated autodefs
  ✓ Clean up autoloads
  > Byte-compiling .local/autoloads.el...


In toplevel form:
../../../autoloads.el:1410:1:Error: Symbol’s value as variable is void: next
> Installing & building packages...

[hlissner]

@ydewit Are you try this out with a pre-existing private config, or a blank one?

Also try bin/doom --debug refresh to see if it will produce a backtrace from the error.

[hlissner]

@ydewit Nevermind, I found the cause and fixed the issue in 457b542

[ydewit]

@hlissner From a blank one.

I just tried the same using the latest and it worked without issues. Thank you.

[HaoZeke]

So quick question, do we now need to switch to using the use-package integration? So we don't need to separately declare the package and configure it in different files?

[hlissner]

@HaoZeke No, it's business as usual. Your package declarations go in DOOMDIR/packages.el, and your configuration goes into DOOMDIR/config.el. Doom's package management will always follow this declarative paradigm.

However, this is just Doom's (strong) suggestion. If you _really_ don't like this paradigm, you could forget about packages.el (delete it if you'd like) and simply load straight (or package.el) right away, in config.el, and install your packages with the :straight or :ensure properties that use-package provides. That way you won't have to consult the bin/doom utility nearly as often (it would only be used to manage the built-in packages, then yours will be auto-downloaded/installed on startup), but you'd lose out on a few startup optimizations that Doom does, under the assumption that you will do things its way.

[madhat2r]

Switched to straight branch and this is what I get when opening emacs now

Error in an autoloads file: autoloads.pkg.el, (void-variable sly-contribs)

Full trace:

Debugger entered--Lisp error: (void-variable sly-contribs)
  add-to-list(sly-contribs sly-macrostep append)
  byte-code("\300\301\302\303#\210\304\305!\203\021\0\305\306\307\"\210\304\305!\203\034\0\305\310\311\"\210\300\207" [add-to-list sly-contribs sly-macrostep append fboundp register-definition-prefixes "sly-macrostep" ("sly-macrostep") "sly-macrostep-tests" ("sly-macrostep-")] 4)
  load("/home/micah/.emacs.d.doom/.local/autoloads.pkg" noerror nomessage)
  (let (command-switch-alist) (load (if noninteractive file (file-name-sans-extension file)) (quote noerror) (quote nomessage)))
  (condition-case e (let (command-switch-alist) (load (if noninteractive file (file-name-sans-extension file)) (quote noerror) (quote nomessage))) ((debug error) (if noninteractive (message "Autoload file warning: %s -> %s" (car e) (error-message-string e)) (signal (quote doom-autoload-error) (list (file-name-nondirectory file) e)))))
  doom-load-autoloads-file("/home/micah/.emacs.d.doom/.local/autoloads.pkg.el")
  (let ((core-autoloads-p (doom-load-autoloads-file doom-autoload-file)) (pkg-autoloads-p (doom-load-autoloads-file doom-package-autoload-file))) (if (and core-autoloads-p (not force-p)) (progn (after! (:or package straight) (require (quote core-packages))) (after! straight (doom-initialize-packages))) (let ((default-directory doom-core-dir)) (let ((--dolist-tail-- (file-expand-wildcards "autoload/*.el"))) (while --dolist-tail-- (let ((file ...)) (load file t t) (setq --dolist-tail-- (cdr --dolist-tail--)))))) (let ((--dolist-tail-- (list doom-local-dir doom-etc-dir doom-cache-dir doom-elpa-dir))) (while --dolist-tail-- (let ((dir (car --dolist-tail--))) (if (file-directory-p dir) nil (make-directory dir (quote parents))) (setq --dolist-tail-- (cdr --dolist-tail--))))) (require (quote core-packages)) (doom-ensure-straight) (doom-initialize-packages force-p)) (if (or (and core-autoloads-p pkg-autoloads-p) force-p noninteractive) nil (if core-autoloads-p nil (message "Your Doom core autoloads file is missing")) (if pkg-autoloads-p nil (message "Your package autoloads file is missing")) (user-error "Run `bin/doom refresh' to generate them")))
  (progn (setq doom-init-p t) (setq exec-path doom--initial-exec-path load-path doom--initial-load-path process-environment doom--initial-process-environment) (require (quote core-lib)) (require (quote core-modules)) (let ((core-autoloads-p (doom-load-autoloads-file doom-autoload-file)) (pkg-autoloads-p (doom-load-autoloads-file doom-package-autoload-file))) (if (and core-autoloads-p (not force-p)) (progn (after! (:or package straight) (require (quote core-packages))) (after! straight (doom-initialize-packages))) (let ((default-directory doom-core-dir)) (let ((--dolist-tail-- (file-expand-wildcards "autoload/*.el"))) (while --dolist-tail-- (let (...) (load file t t) (setq --dolist-tail-- ...))))) (let ((--dolist-tail-- (list doom-local-dir doom-etc-dir doom-cache-dir doom-elpa-dir))) (while --dolist-tail-- (let ((dir ...)) (if (file-directory-p dir) nil (make-directory dir ...)) (setq --dolist-tail-- (cdr --dolist-tail--))))) (require (quote core-packages)) (doom-ensure-straight) (doom-initialize-packages force-p)) (if (or (and core-autoloads-p pkg-autoloads-p) force-p noninteractive) nil (if core-autoloads-p nil (message "Your Doom core autoloads file is missing")) (if pkg-autoloads-p nil (message "Your package autoloads file is missing")) (user-error "Run `bin/doom refresh' to generate them"))) (if noninteractive nil (doom-load-envvars-file doom-env-file (quote noerror))))
  (if (or force-p (not doom-init-p)) (progn (setq doom-init-p t) (setq exec-path doom--initial-exec-path load-path doom--initial-load-path process-environment doom--initial-process-environment) (require (quote core-lib)) (require (quote core-modules)) (let ((core-autoloads-p (doom-load-autoloads-file doom-autoload-file)) (pkg-autoloads-p (doom-load-autoloads-file doom-package-autoload-file))) (if (and core-autoloads-p (not force-p)) (progn (after! (:or package straight) (require (quote core-packages))) (after! straight (doom-initialize-packages))) (let ((default-directory doom-core-dir)) (let ((--dolist-tail-- ...)) (while --dolist-tail-- (let ... ... ...)))) (let ((--dolist-tail-- (list doom-local-dir doom-etc-dir doom-cache-dir doom-elpa-dir))) (while --dolist-tail-- (let (...) (if ... nil ...) (setq --dolist-tail-- ...)))) (require (quote core-packages)) (doom-ensure-straight) (doom-initialize-packages force-p)) (if (or (and core-autoloads-p pkg-autoloads-p) force-p noninteractive) nil (if core-autoloads-p nil (message "Your Doom core autoloads file is missing")) (if pkg-autoloads-p nil (message "Your package autoloads file is missing")) (user-error "Run `bin/doom refresh' to generate them"))) (if noninteractive nil (doom-load-envvars-file doom-env-file (quote noerror)))))
  doom-initialize(nil)
  eval-buffer(#<buffer  *load*-786279> nil "/home/micah/.emacs.d.doom/core/core.el" nil t)  ; Reading at buffer position 23013
  load-with-code-conversion("/home/micah/.emacs.d.doom/core/core.el" "/home/micah/.emacs.d.doom/core/core.el" nil t)
  require(core "/home/micah/.emacs.d.doom/core/core")
  eval-buffer(#<buffer  *load*-595926> nil "/home/micah/.emacs.d.doom/init.el" nil t)  ; Reading at buffer position 2680
  load-with-code-conversion("/home/micah/.emacs.d.doom/init.el" "/home/micah/.emacs.d.doom/init.el" nil nil)
  load("/home/micah/.emacs.d.doom/init.el")
  (let* ((emacs-directory (file-name-as-directory (chemacs-emacs-profile-key (quote user-emacs-directory)))) (init-file (expand-file-name "init.el" emacs-directory)) (custom-file- (chemacs-emacs-profile-key (quote custom-file) init-file)) (server-name- (chemacs-emacs-profile-key (quote server-name)))) (setq user-emacs-directory emacs-directory) (if server-name- (progn (setq server-name server-name-))) (mapcar (function (lambda (env) (setenv (car env) (cdr env)))) (chemacs-emacs-profile-key (quote env))) (load init-file) (if (not custom-file) (progn (setq custom-file custom-file-) (load custom-file))))
  chemacs-load-profile("doom")
  (if args (let ((s (split-string (car args) "="))) (cond ((equal (car args) "--with-profile") (add-to-list (quote command-switch-alist) (quote ("--with-profile" lambda (_) (pop command-line-args-left)))) (chemacs-load-profile (car (cdr args)))) ((equal (car s) "--with-profile") (add-to-list (quote command-switch-alist) (cons (car args) (quote (lambda ...)))) (chemacs-load-profile (mapconcat (quote identity) (cdr s) "="))) (t (chemacs-check-command-line-args (cdr args))))) (chemacs-load-profile (chemacs-detect-default-profile)))
  chemacs-check-command-line-args(nil)
  (cond ((equal (car args) "--with-profile") (add-to-list (quote command-switch-alist) (quote ("--with-profile" lambda (_) (pop command-line-args-left)))) (chemacs-load-profile (car (cdr args)))) ((equal (car s) "--with-profile") (add-to-list (quote command-switch-alist) (cons (car args) (quote (lambda (_))))) (chemacs-load-profile (mapconcat (quote identity) (cdr s) "="))) (t (chemacs-check-command-line-args (cdr args))))
  (let ((s (split-string (car args) "="))) (cond ((equal (car args) "--with-profile") (add-to-list (quote command-switch-alist) (quote ("--with-profile" lambda (_) (pop command-line-args-left)))) (chemacs-load-profile (car (cdr args)))) ((equal (car s) "--with-profile") (add-to-list (quote command-switch-alist) (cons (car args) (quote (lambda (_))))) (chemacs-load-profile (mapconcat (quote identity) (cdr s) "="))) (t (chemacs-check-command-line-args (cdr args)))))
  (if args (let ((s (split-string (car args) "="))) (cond ((equal (car args) "--with-profile") (add-to-list (quote command-switch-alist) (quote ("--with-profile" lambda (_) (pop command-line-args-left)))) (chemacs-load-profile (car (cdr args)))) ((equal (car s) "--with-profile") (add-to-list (quote command-switch-alist) (cons (car args) (quote (lambda ...)))) (chemacs-load-profile (mapconcat (quote identity) (cdr s) "="))) (t (chemacs-check-command-line-args (cdr args))))) (chemacs-load-profile (chemacs-detect-default-profile)))
  chemacs-check-command-line-args(("emacs"))
  eval-buffer(#<buffer  *load*> nil "/home/micah/.emacs" nil t)  ; Reading at buffer position 6290
  load-with-code-conversion("/home/micah/.emacs" "/home/micah/.emacs" t t)
  load("~/.emacs" t t)
  #f(compiled-function () #<bytecode 0x1df989>)()
  command-line()
  normal-top-level()

[ckruse]

I tried out this branch this morning. There seems to be something wrong with package management: not all packages get loaded. For example I did the following:

~elisp
(package! move-text)
(use-package! move-text
:commands (move-text-up move-text-down)
:bind (([M-up] . move-text-up)
([M-down] . move-text-down)))
~

In the development branch it works like a charm; in the straight branch if have to eval-region the use-package! move-text to get it working. Same with for example this block:

~~~ elisp
(package! flycheck-mix)
(use-package! flycheck-mix
:commands flycheck-mix-setup
:init
(defvar elixir-enable-compilation-checking nil)

(add-to-list 'safe-local-variable-values
(cons 'elixir-enable-compilation-checking nil))
(add-to-list 'safe-local-variable-values
(cons 'elixir-enable-compilation-checking t))

(add-hook 'hack-local-variables-hook
(lambda ()
(when (and (or elixir-enable-compilation-checking) (string-equal major-mode "elixir-mode"))
(flycheck-mix-setup)
;; enable credo only if there are no compilation errors
(when (not (member '(warning . elixir-credo) (flycheck-checker-get 'elixir-mix 'next-checkers)))
(flycheck-add-next-checker 'elixir-mix '(warning . elixir-credo))))))

(add-hook 'flycheck-before-syntax-check-hook (lambda ()
(when (string-equal major-mode "elixir-mode")
(setenv "MIX_ENV" "test")))))
~~~

This was working when I was using the develop branch, but it doesn't with the straight branch: the flycheck-mix package doesn't get loaded and the elixir-enable-compilation-checking isn't marked as a safe directory-local variable.

Am I missing something?

Edit: it seems as if packages.el doesn't get loaded upon startup. A simple (message "called!") doesn't generate any messages in the *Messages* buffer when starting up?

[hlissner]

@ckruse I'm not sure if you combined them for brevity, but just to make sure:

• (package! move-text) should go in ~/.doom.d/packages.el
• And your use-package! block should go in~/.doom.d/config.el`

Doom's package management system is declarative. You do not put them in the same file. Your packages.el is not loaded at startup (for interactive sessions) on either branch (until you start using the package management API).

However, in both branches, bin/doom loads them immediately -- which is where the bulk of package management in Doom takes place.

[ckruse]

@hlissner with the changes you suggested everything works again, thanks.

Interestingly I've been using this setup on the develop branch for months :-) I believe you when you say that packages.el isn't read on startup on either branch, but something in my configuration must've caused Doom to load the file nonetheless.

I did this because I thought this is how it was meant to do. Sorry for causing unnecessary noise.

[hlissner]

@ckruse No sweat! It's possible you're eagerly loading package.el (which Doom spends some effort avoiding), either by requireing it directly or using it's autoloaded API (e.g. the package-installed-p or package-install functions), or you're eager loading a package that does (like magit, for instance).

Once package.el is loaded, Doom pulls in all its package management machinery (and reads all your module packages.el files). There is usually a better way of going about these things, but it's really easy to miss, what with how abysmal the documentation situation in Doom is at the moment. I'm working on it!

[hlissner]

The straight branch has now been merged into develop, so I'll consider this request fulfilled.

• I recommend current develop users do the following for the smoothest upgrade:

cd ~/.emacs.d
rm -rf .local
git pull --rebase
bin/doom refresh

• And current straight users will need to switch to the develop branch before runing doom upgrade:

cd ~/.emacs.d
git checkout develop
bin/doom upgrade

[ckruse]

Thank you for your work!