Straight.el: Straight.el cannot install a working (recent) version of Eglot

[gagbo]

(only lightly) related to joaotavora/eglot#492

Straight uses its emacsmirror recipe to fetch project.el as far as I understand, which points to another repo https://github.com/emacsmirror/epkgs/blob/master/.gitmodules#L16449-L16451
And that repo is outdated : https://github.com/emacsmirror/project/blob/master/project.el#L4

@raxod502 @tarsius do you know if there's anything we can do to forcefully update that recipe ? Basically with the current state, installing eglot will fetch [email protected] anyway, and therefore trying to start eglot will trigger (void function project-root), as the API had a breaking change between 0.1.3 and 0.3

[gagbo]

It looks like gnu-elpa-mirror should be used to fetch project and there would be no problem. But for some reason, C-u M-x straight-get-recipe RET gnu-elpa-mirror RET doesn't list project in the candidates :(

[gagbo]

And apparently, the issue is that project gets filtered out of gnu-elpa-mirror because it's a "built-in package".

Since users might have the wrong version of the package "builtin", could we provide a user option to override this check if we want ? Speaking about those 2 functions :

(defun straight--package-built-in-p (package)
  "Given PACKAGE symbol, return non-nil if it's built in to Emacs.
The return value of this function might change between different
versions of Emacs for the same package.

If a package is built in, then the package won't be listed in GNU
ELPA (Mirror) and it won't be an error if no recipe can be found
for it."
  (unless straight--cached-built-in-packages
    (require 'finder-inf)
    (let ((table (make-hash-table)))
      (dolist (cell package--builtins)
        (puthash (car cell) t table))
      (setq straight--cached-built-in-packages table)))
  (gethash package straight--cached-built-in-packages))

(defun straight-recipes-gnu-elpa-mirror-retrieve (package)
  "Look up a PACKAGE recipe in the GNU ELPA mirror.
PACKAGE should be a symbol. If the package is maintained in GNU
ELPA (and should be retrieved from there, which isn't the case if
the package is built in to Emacs), return a MELPA-style recipe.
Otherwise, return nil."
  (unless (straight--package-built-in-p package)
    (when (file-exists-p (symbol-name package))
      `(,package :type git
                 :host github
                 :repo ,(format "emacs-straight/%S" package)
                 ;; Kinda weird, but in fact this is how package.el
                 ;; works. So if we want to replicate the build
                 ;; process, we should trust that the gnu-elpa-mirror
                 ;; put the correct files into the repository, and
                 ;; then just link *everything*. As an FYI, if we
                 ;; don't do this, then AUCTeX suffers problems with
                 ;; style files, see
                 ;; <https://github.com/raxod502/straight.el/issues/423>.
                 :files ("*" (:exclude ".git"))))))

[tarsius]

The Emacsmirror currently gets core packages from the latest release candidate not the master branch. I intend to change that soon.

[goranmoomin]

Oh, this looks like the culprit of what I thought as #529.

[raxod502]

I have updated straight.el so that it filters out an explicit blacklist of packages from GNU ELPA (and mirror), rather than all built-in packages. This should allow you to use packages from GNU ELPA that supersede builtins while still not using those packages that are obsolete backports of builtins for older Emacs versions. Please let me know if it fixes the issue for you.

[raxod502]

Since users might have the wrong version of the package "builtin", could we provide a user option to override this check if we want ?

The reason I implemented the above solution instead of what you suggested is that some packages from GNU ELPA should never be installed, such as cl-lib.

;; This is a forward compatibility package, which provides (a subset of) the
;; features of the cl-lib package introduced in Emacs-24.3, for use on
;; previous emacsen (it should work on Emacs≥21 as well as XEmacs).

;; Make sure this is installed *late* in your `load-path`, i.e. after Emacs's
;; built-in .../lisp/emacs-lisp directory, so that if/when you upgrade to
;; Emacs-24.3, the built-in version of the file will take precedence, otherwise
;; you could get into trouble (although we try to hack our way around the
;; problem in case it happens).

;; This code is largely copied from Emacs-24.3's cl.el, with the alias bindings
;; simply reversed.

[gagbo]

This looks like it should work, I'll try again when I have time :)

[gagbo]

It's perfect thanks ! (sorry for triggering the seq issue though :( )

[raxod502]

It seems to be an unfortunate side effect of making any changes to this kind of code. After all, the intended consequence of this change was a number of subtle changes in version resolution; the problem is just that not all of the subtle changes were the right subtle changes. I am not even sure how the problem could be avoided; the issues that come up are inevitably weird edge cases with specific packages or system configurations and I don't know how possible it is to cover them with tests.

The only real solution I see, at least for now, is for all users of straight.el to pin package versions. Perhaps however @hlissner will have thoughts, given that they are the one most affected by these bugs.

[gagbo]

I do think that pinning will prevent most of these issues. And if we want to do pinning, that means we should be able to access/use older versions ; therefore I think that moving a lot of these recipes out of "built-in" territory and into "fetch from repository on the network" is a good move.

It's more work for end-users, and maybe some utilities written in elisp might help people handle pin management (or even provide a safe hatch, as a unique boolean variable to revert entirely to the old behaviour), but it's probably for the best ?

[raxod502]

What do you mean? straight.el already provides a full solution, namely the lockfile system. The only issue is Doom doesn't use it currently.

[gagbo]

Didn't know there was a lockfile. I'll probably try to dig it a little bit later

[tarsius]

FWIW The Emacsmirror now mirrors "core" package (i.e. packages that are part of Emacs but are also available from GNU Elpa) from the master branch of Emacs.

Given the current update frequency, these packages are updated about every two weeks, but I intend to improve that.

Due to limitations/bugs of git filter-repo I had to discard old history. Only history since shortly after Emacs 27.1 is included for the time being. Including older history would take an hour and create a repository that is 16GB large for each of these 13 packages. Garbage collection would help but likely take another hour. I am guessing that as of right now a pull would take about 24h, even if non of these packages changed. (A pull of the shallow history, as I am currently doing, takes a few minutes at most.)

git filter-repo is supposed to support incremental filtering but that does not work for me. I am fairly confident this can be fixed but am still waiting for a response from the author of this tool.

[tarsius]

I haven't heard back on https://github.com/newren/git-filter-repo/issues/141 it.

@raxod502 Maybe you could take git filter-repo --state-branch to see if it is just me who is holding it wrong.

[raxod502]

Yeah, I don't know -- I've never used the tool and indeed there seems to be no documentation on that option. Seems like a bug to me!

[raxod502]

To be clear I get the same error as you when I run that command, though.