Doom-emacs: Doom CLI Rewrite

[hlissner]

I've been working on this CLI rewrite for a while. Over time, bug reports have trickled in about various problems that this rewrite directly or indirectly fixes. To reduce the noise I'm consolidating those and futures posts into this one, to represent the rewrite:

Primary goals of this rewrite

• [ ] (#4270) To make it transactional (failures should revert back to where we started rather than ignored).
• [ ] (#2141) To offer a doom rollback command.
• [ ] (#3490) Make doom sync smarter about :recipe changes and stale bytecode (so you don't need to do that doom sync -u and doom build dance).
• [ ] Improve the doctor API and checks, so we get more useful checks _and_ the warnings/errors they emit are less cryptic.
  
  
  
  • Check for misspelled modules and flags (#2048)
  
  
  • Warn users when they inadvertently use package.el #4157
  
  
  • Support for a private doctor.el in your DOOMDIR (#4190)
  
  
  • Perform more sanity checks on your private config (#2959)
  
  
• [ ] To fix our unit tests and add integration tests, powered by nix and buttercup.
• [ ] CI/CD, so bumping packages can be automated (and tested).
• [ ] To handle straight.el's prompts better -- or prevent them entirely. Re-cloning packages is better UX than these confusing prompts.
• [ ] Add a new doom package ... family of commands for per-package management (e.g. doom package build PACKAGE will spare you an expensive doom build to rebuild all packages).
• [ ] Add a new doom module ... family of commands for toggling modules (or installing third party ones) programmatically.
• [ ] Allow doom run to launch sandboxes and, if nix is available, do so from specific versions of Emacs.

This is not an exhaustive list.

Other bugs

• [ ] To fix whatever weirdness Windows is doing in #3844

---

I'll keep this post updated until the rewrite is done. There is no ETA on this.

[danilevy1212]

Hi! Any way we can help you with this rewrite or contribute to its features? Thanks for the awesome work.

[the-cobalt]

Add a new doom package ... family of commands for per-package management

I think it might be better to integrate this into existing commands. Using your example, instead of doom package build x, I think doom build x might be cleaner.

This is just a micro-aggression, but:

they use inadvertently use package.el

I'm pretty sure that's a typo

Also, I think it would be nice to put this in like a "cli" branch, considering how this could be a breaking change, but people might want to keep track of/try it on their own.

[innerout]

If possible, I would suggest adding in the doom upgrade command a message informing the user that init.el has been updated with new options. This would be useful when a new module is added or when a new flag is added in a module.

[hlissner]

If possible, I would suggest adding in the doom upgrade command a message informing the user that init.el has been updated with new options. This would be useful when a new module is added or when a new flag is added in a module.

@innerout I do have plans to address this:

1. A changelog to list module changes on our (as of yet unpublished) Discourse,
2. A doctor check that will complain about missing, renamed, or removed modules and flags.
3. A doom whatsup command that will summarize changes between generations (each time you run doom upgrade), including modules and packages.

1 and 2 are almost done, but 3 I haven't started on.

[hlissner]

I think it might be better to integrate this into existing commands. Using your example, instead of doom package build x, I think doom build x might be cleaner.

@the-cobalt I plan to improve the user story for doom sync; make it smarter about outdated byte-code. Perhaps have something like a --thorough switch to replace doom build, but I consider it a bug that users have had to depend on doom build so much.

Not to mention, build would be only one of doom package's prospective sub-commands. There'll be commands for resetting packages, byte-compiling, testing, snapshotting, bumping, and listing them -- among others. Because these are maintenance commands, I don't think they warrant a top-level command.

I'm pretty sure that's a typo

Good catch! Fixed.

Also, I think it would be nice to put this in like a "cli" branch, considering how this could be a breaking change, but people might want to keep track of/try it on their own.

I'm not certain this will happen, because it's a lot of work to support folks that decide to track it -- especially because Doom lacks adequate contributor documentation (something I plan to focus on after the rewrite). I'll let you know if that changes.

[hlissner]

Any way we can help you with this rewrite or contribute to its features?

@danilevy1212 Thank you! Unfortunately there is little code work here to share right now. Another reason for this rewrite was to resolve Doom's technical debt and make onboarding people easier. It's difficult to coordinate any development effort when we have no CI, tests, or contributor documentation to shed light on our black box. Especially so when I am everyone's central resource for information and support.

That's why I have a Discourse coming up soon. I hope it will lift some of that weight off my shoulders -- Discord has proven a lousy archive for problems and solutions, the larger Doom grows. Now that I'm moving my development roadmap and progress reports there, I hope Doom's development will be more transparent (and so, easier to contribute to).

In the meantime, I do need help with:

• Triage. Our issue tracker gets 1-6 new issues a day. 2 in 5 are non-issues (personal configuration or upstream problems). 1 in 6 are upstream or duplicates. I need help filtering those out. Otherwise, I need help prodding folks to do more testing, like:
  
  
  
  • Checking if their issue is reproducible in the sandbox.
  
  
  • Making sure they've tried the usual doom sync && doom build.
  
  
  • Producing backtraces from error messages.
  
  
  • Reminding them to include their M-x doom/info when they have ignored the issue template.
  
  
• There are a number of modules that still need documentation: https://github.com/hlissner/doom-emacs/issues/1166
• If you know your way around nix, adding a shell.nix to your favorite module(s) would be a big help. These will be the backbone of future CI. Plus, a minimum viable environment will help me debug issues for modules I don't use.

If those seem intimidating, it is also a great help stopping by our Discord or (when its up) our Discourse -- helping folks out or adding to the cheer.

[the-cobalt]

commands for resetting packages, byte-compiling, testing, snapshotting, bumping, and listing them

Ok, for all of that, I think it would be better for all second level command, unless you wanted to go full "I'm an apt clone!" style, but that's out of the scope here.

Good catch! Fixed.

Probably 76% of all of my PRs are just typo/grammar fixes. :wink:

I'm not certain this will happen, because it's a lot of work to support folks that decide to track it -- especially because Doom lacks adequate contributor documentation (something I plan to focus on after the rewrite). I'll let you know if that changes.

Yeah, probably working on that first before any kind of branching is a good idea.

[danilevy1212]

Any way we can help you with this rewrite or contribute to its features?

@danilevy1212 Thank you! Unfortunately there is little code work here to share right now. Another reason for this rewrite was to resolve Doom's technical debt and make onboarding people easier. It's difficult to coordinate any development effort when we have no CI, tests, or contributor documentation to shed light on our black box. Especially so when I am everyone's central resource for information and support.

That's why I have a Discourse coming up soon. I hope it will lift some of that weight off my shoulders -- Discord has proven a lousy archive for problems and solutions, the larger Doom grows. Now that I'm moving my development roadmap and progress reports there, I hope Doom's development will be more transparent (and so, easier to contribute to).

Yeah, a man can only do so much although you constantly defy what that expression means :laughing:. Doom is a great project, it's good to focus now on better organization to be able to delegate tasks before it becomes a giant spacemacs.

In the meantime, I do need help with:

* Triage. Our issue tracker gets 1-6 new issues a day. 2 in 5 are non-issues (personal configuration or upstream problems). 1 in 6 are upstream or duplicates. I need help filtering those out. Otherwise, I need help prodding folks to do more testing, like:

  * Checking if their issue is reproducible [in the sandbox](https://github.com/hlissner/doom-emacs/blob/develop/docs/getting_started.org#testing-in-dooms-sandbox).
  * Making sure they've tried the usual `doom sync && doom build`.
  * Producing backtraces from error messages.
  * Reminding them to include their `M-x doom/info` when they have ignored the issue template.

I've been popping into the discord a lot lately but not so much keeping an eye on issues. I'll see if can help out here too.

* There are a number of modules that still need documentation: #1166

Coincidentally, I am going through the modules one at the time configuring them to my liking, so I'll see if I can contribute to the documentation of those that had incomplete/missing READMEs. :+1:

* [If you know your way around nix, adding a `shell.nix` to your favorite module(s) would be a big help](https://github.com/hlissner/doom-emacs/issues/2552). These will be the backbone of future CI. Plus, a minimum viable environment will help me debug issues for modules I don't use.

Good thing I am looking into Nix at the moment, although it will be a while until I can help in this regard, total noob. :/

If those seem intimidating, it is also a great help stopping by our Discord or (when its up) our Discourse -- helping folks out or adding to the cheer.

Awesome, I will keep track of when discourse is up. Keep up the good work :smile: