Straight.el: Straight.el won't build itself on latest emacs without native-comp

[garyo]

What's wrong

I built latest emacs from scratch on Centos 8 without native-comp, set up straight.el the usual way, and during startup straight.el fails while building itself.
The reason seems to be at https://github.com/raxod502/straight.el/blob/e1390a933b6f5a15079d6dec91eac97a17aad10c/straight.el#L4840:

(defun straight--build-native-compile (recipe)
  ...
  (when (and (fboundp 'native-compile-async)
             (member 'straight--build-compile straight--build-functions))

I had to add two cases to the when:

  (when (and (fboundp 'native-compile-async)
             (member 'straight--build-compile straight--build-functions)                                   
             (fboundp 'native-comp-available-p)                                                            
             (native-comp-available-p))

and then it worked normally.

Directions to reproduce

• Build emacs from scratch, without native-compilation.
• With this .emacs:

;;; Set up package system -- straight.el (better than built-in package.el)
(defvar bootstrap-version)
(let ((bootstrap-file
       (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
      (bootstrap-version 5))
  (unless (file-exists-p bootstrap-file)
    (with-current-buffer
        (url-retrieve-synchronously
         "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
         'silent 'inhibit-cookies)
      (goto-char (point-max))
      (eval-print-last-sexp)))
  (load bootstrap-file nil 'nomessage))

;;; Meta-package system: use-package. Auto-installs and configures packages.
(straight-use-package 'use-package)
(defvar straight-use-package-by-default)
(setq straight-use-package-by-default t) ; make use-package use straight
(defvar straight-check-for-modifications)
(setq straight-check-for-modifications
      '(check-on-save find-when-checking))

it should fail.

[progfolio]

Thank you for the detailed bug report. Could you include the output of M-x straight-version and M-x emacs-version please?
It would also be useful if you could include a stack trace for the failure (should be able to get it with emacs --debug-init) and/or the error message.

[maurelio1234]

I just had the same problem on a dev master from straight on top of a dev master from emacs.

The stacktrace I have is here:

```Debugger entered--Lisp error: (void-variable comp-deferred-compilation-deny-list)
straight--native-compile-file-p("/home/marcos/.emacs.d/straight/build/straight/stra...")
native-compile-async-skip-p("/home/marcos/.emacs.d/straight/build/straight/stra..." nil straight--native-compile-file-p)
native--compile-async("/home/marcos/.emacs.d/straight/build/straight/" recursively nil straight--native-compile-file-p)
native-compile-async("/home/marcos/.emacs.d/straight/build/straight/" recursively nil straight--native-compile-file-p)
straight--build-native-compile((:type git :host github :repo "raxod502/straight.el" :files ("straight.el") :branch "master" :package "straight" :local-repo "straight.el"))
run-hook-with-args(straight--build-native-compile (:type git :host github :repo "raxod502/straight.el" :files ("straight.el") :branch "master" :package "straight" :local-repo "straight.el"))
straight--build-package((:type git :host github :repo "raxod502/straight.el" :files ("straight.el") :branch "master" :package "straight" :local-repo "straight.el") nil)
#f(compiled-function () #)()
straight--transaction-exec(use-package-(:type\ git\ :host\ github\ :repo\ \"raxod502/straight.el\"\ :files\ (\"straight.el\")\ :branch\ \"master\"\ :package\ \"straight\"\ :local-repo\ \"straight.el\")-nil-nil :now #f(compiled-function () #))
straight-use-package((straight :type git :host github :repo "raxod502/straight.el" :files ("straight*.el") :branch "master"))
load-with-code-conversion("/home/marcos/.emacs.d/straight/repos/straight.el/b..." "/home/marcos/.emacs.d/straight/repos/straight.el/b..." nil t)
load("/home/marcos/.emacs.d/straight/repos/straight.el/b..." nil nomessage)
(let ((bootstrap-file (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory)) (bootstrap-version 5)) (if (file-exists-p bootstrap-file) nil (save-current-buffer (set-buffer (url-retrieve-synchronously "https://raw.githubusercontent.com/raxod502/straigh..." 'silent 'inhibit-cookies)) (goto-char (point-max)) (eval-print-last-sexp))) (load bootstrap-file nil 'nomessage))
load-with-code-conversion("/home/marcos/.emacs.d/bootstrap.el" "/home/marcos/.emacs.d/bootstrap.el" nil t)
require(bootstrap "/home/marcos/.emacs.d/bootstrap.el")
command-line-1(("--eval" "(require 'bootstrap \"/home/marcos/.emacs.d/bootstr..."))
command-line()
normal-top-level()

emacs version:

GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.22.30, cairo version 1.15.10) of 2021-05-12

I use this code to bootstrap straight. I am on a docker container, so everything is built fresh or almost, from both masters

;; Straight bootstrapper
(defvar bootstrap-version)
(let ((bootstrap-file
(expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
(bootstrap-version 5))
(unless (file-exists-p bootstrap-file)
(with-current-buffer
(url-retrieve-synchronously
"https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
'silent 'inhibit-cookies)
(goto-char (point-max))
(eval-print-last-sexp)))
(load bootstrap-file nil 'nomessage))
```

[garyo]

M-x straight-version:

prerelease (HEAD -> master, origin/master, origin/HEAD) e1390a9 2021-02-27

M-x emacs-version:

GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu) of 2021-05-11

(note: emacs built from master branch as of May 11 2021)

My stack trace is pretty much identical to @maurelio1234 's.

The error message during init: error: Emacs was not compiled with native compiler support (--with-native-compilation)

[maurelio1234]

@garyo However, your fix doesn't work for me.
What seems to work is adding a

(defvar comp-deferred-compilation-deny-list ())

before my bootstrap code

[progfolio]

@maurelio1234: Thanks for the additional info. Output of M-x straight-version and emacs-version please?

[maurelio1234]

straight-version:

prerelease (HEAD -> master, origin/master, origin/HEAD) e1390a9 %cs

emacs-version

GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.22.30, cairo version 1.15.10) of 2021-05-12
Mark set

[garyo]

@garyo However, your fix doesn't work for me.
What seems to work is adding a

(defvar comp-deferred-compilation-deny-list ())

before my bootstrap code

Interesting. It shouldn't be able to get into the body of straight--build-native-compile with the two conditions I added. Would you mind adding something like this just above that when (in straight.el)?

(message "straight--build-native-compile: native fbound? %s" (fboundp 'native-comp-available-p))
(message "straight--build-native-compile: native avail? %s" (native-comp-available-p))

It should print t then nil, and the nil should prevent it trying to native-compile things.

[progfolio]

You could, alternatively, (setq straight-disable-native-compile t) prior to the bootstrapping code and see if it will build properly.

[maurelio1234]

You could, alternatively, (setq straight-disable-native-compile t) prior to the bootstrapping code and see if it will build properly.

Yes, it seems to work

Interesting. It shouldn't be able to get into the body of straight--build-native-compile with the two conditions I added. Would you mind adding something like this just above that when (in straight.el)?

There it goes:

straight--build-native-compile: native fbound? t
straight--build-native-compile: native avail? t

[maurelio1234]

My stack trace is pretty much identical to @maurelio1234 's.

The error message during init: error: Emacs was not compiled with native compiler support (--with-native-compilation)

Oh I misread something! Just realized that my case is different from @garyo's.

I am indeed using the emacs' master with native compilation enabled and straight could not build itself on it,
unless I add (defvar comp-deferred-compilation-deny-list ()) before bootstrapping it.

In fact on a docker image built around 2weeks ago, when the nativecomp was merged into the master, I was able to build and use straight's master on emacs' master. Something may have changed on one of them during these two weeks.

[progfolio]

@maurelio1234: Yes, your case is a slightly different bug. Can you confirm that everything works ok if you toggle straight-disable-native-compile around the bootstrap code? e.g.

(setq straight-disable-native-compile t)
;;;bootstrap snippet here
(setq straight-disable-native-compile nil)

[progfolio]

Upstream change to the variable name:

https://github.com/emacs-mirror/emacs/commit/8bc91a6a3f6cc84f549d24467832f67b3130201d

 * Rename comp-deferred-compilation-deny-list

* lisp/emacs-lisp/comp.el (native-comp-deferred-compilation-deny-list)
(native-compile-async-skip-p): Rename
comp-deferred-compilation-deny-list ->
native-comp-deferred-compilation-deny-list.

We're chasing a moving target with native compilation, but I expect things will begin to settle now that it has been merged into Emacs.

[maurelio1234]

@maurelio1234: Yes, your case is a slightly different bug. Can you confirm that everything works ok if you unset/set straight-disable-native-compile around the bootstrap code? e.g.

No it doesn't work

[maurelio1234]

We're chasing a moving target with native compilation, but I expect things begin to settle now that it has been merged into Emacs.

Yes I noticed the name slightly different in straight--native-compile-file-p and in C-h v but I thought it was something internal to straight.

[progfolio]

@maurelio1234, @garyo:

I've pushed a couple of fixes to the "develop" branch which should address both of your issues. However, it looks like you are both on straight's "master" branch.
I would encourage you to both upgrade to the "develop" branch. You can do so by adding:

(setq straight-repository-branch "develop")

prior to the bootstrap snippet.
You can utilize straight-freeze-versions and straight-thaw-versions to define your own "stable" set of packages if you're uncomfortable remaining on the "develop" branch from there.

@raxod502: We'll probably want to update the master branch to incorporate the native-comp API change/bug fixes.

[maurelio1234]

Hi @progfolio I confirm it builds when using the develop branch with native compilation.
Thank you very much!

[rileyrg]

FYI, related : I just pulled from master (emacs) and emacs wouldn't start - deleted all elc, eln.cache and straight/build and no go.
I added this to my config and straight rebuilt itself properly and emacs started. After that build, I could remove this line,

• TODO error with straight in latest gcceamacs pull workaround
  #+begin_src emacs-lisp
  (setq comp-deferred-compilation-deny-list nil)
  #+end_src

Dropped this in here rather than a new issue. Hope it's helpful.

post edit : read the rest and this is indeed covered here : https://github.com/raxod502/straight.el/issues/757#issuecomment-839827576

[raxod502]

We'll probably want to update the master branch to incorporate the native-comp API change/bug fixes

Sounds good to me! Yeah, I think it's generally fine/desired to backport most bugfixes unless they seem alarmingly unsafe for whatever reason.

[jacksonludwig]

I still am getting this issue even after setting my branch to develop (M-x straight-version confirms it for me). Is there anything I'm missing? It still seems to be looking for comp-deferred-compilation-deny-list.

[progfolio]

Please link to or post the bootstrap snippet you are using as it appears in your init file. It would also be useful to see the output of straight-version.

[jacksonludwig]

@progfolio sure, this is what I am using:

(setq straight-repository-branch "develop")
(setq straight-use-package-by-default t)
(defvar bootstrap-version)
(let ((bootstrap-file
       (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
      (bootstrap-version 5))
  (unless (file-exists-p bootstrap-file)
    (with-current-buffer
    (url-retrieve-synchronously
     "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
     'silent 'inhibit-cookies)
      (goto-char (point-max))
      (eval-print-last-sexp)))
  (load bootstrap-file nil 'nomessage))
(straight-use-package 'use-package)

Also this is the straight version output

prerelease (HEAD -> develop, origin/develop) e46292d 2021-05-12

EDIT: Sorry, this was my fault. I did not realize but my emacs was not fully updated to the lastest master. After rebuilding and using the develop branch of straight, I get no error.

[rileyrg]

5541697 merge of dev into master and its not working without
(setq comp-deferred-compilation-deny-list nil)
still.
Heh, Ive entered org-mode build hell follwoing pulling straight, deleting build and starting up anew, I'll have a poke about but probably my fault at the moment. But the fix should be merged into master now?

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
straight-vc-git-local-repo-name((:includes (org) :package "org-plus-contrib" :type git))
apply(straight-vc-git-local-repo-name (:includes (org) :package "org-plus-contrib" :type git))
straight-vc(local-repo-name git (:includes (org) :package "org-plus-contrib" :type git))
straight-vc-local-repo-name((:includes (org) :package "org-plus-contrib" :type git))
straight--convert-recipe((org-plus-contrib :includes (org)) nil)
straight-use-package((org-plus-contrib :includes (org)))
eval-buffer(# nil "/home/rgr/.emacs.d/etc/elisp/rgr-org.el" nil t) ; Reading at buffer position 59
load-with-code-conversion("/home/rgr/.emacs.d/etc/elisp/rgr-org.el" "/home/rgr/.emacs.d/etc/elisp/rgr-org.el" t t)
#(rgr/org "rgr-org" NOERROR)
apply(# (rgr/org "rgr-org" NOERROR))
require(rgr/org "rgr-org" NOERROR)
eval-buffer(# nil "/home/rgr/.emacs.d/init.el" nil t) ; Reading at buffer position 2596
load-with-code-conversion("/home/rgr/.emacs.d/init.el" "/home/rgr/.emacs.d/init.el" t t)
load("/home/rgr/.emacs.d/init" noerror nomessage)
startup--load-user-init-file(#f(compiled-function () #) #f(compiled-function () #) t)
command-line()
normal-top-level()

[rileyrg]

Heh, Ive entered org-mode build hell follwoing pulling straight, deleting build and starting up anew, I'll have a poke about but probably my fault at the moment.

So, this line is causing my start to abort

(straight-use-package '(org-plus-contrib :includes (org)))

What's the official "how to use org and contrib" recipe now? I can find a few ;)

[progfolio]

@rileyrg:

Yes, Org has moved the org-contrib folder to its own repository.
I've accounted for that in the latest recipe, so:

(straight-use-package 'org-contrib)

Should work now.

[rileyrg]

@rileyrg:

Yes, Org has moved the org-contrib folder to its own repository.
I've accounted for that in the latest recipe, so:

(straight-use-package 'org-contrib)

Should work now

Sorry to be dim. I'm completely confused by org/org-contrib. Im a literate config guy and these errors completely break my editor :( Isnt org-contrib org with the contrib built in? So now org has moved it in, (use-package org) should be all I need?

post edit. I misread. Ignore for now.

[progfolio]

Contrib is just a bunch of add on packages for Org mode.
Bastien's goal is to eventually have all of these packages maintained elsewhere individually. In the mean time they are hosted in the org contrib repo.

So to install the development version of Org, you still need to use:

(straight-use-package 'org) 

Prior to loading Org (so the built in version doesn't shadow the development version) and then the above recipe for the contrib repo if you need any of those packages.