Straight.el: Upgrading/recompiling org leaves macros undefined somehow

[antifuchs]

What's wrong

Very often after I use straight-pull-package on org, I see errors about macros (defined in org-compat.el or org-macs.el) being undefined, as if they weren't present at byte-compile time. Here's an example from my current checkout (org 2eee3ca190520b516b86881dbf3dc4679b86f7fa):

Debugger entered--Lisp error: (invalid-function org-preserve-local-variables)
  org-preserve-local-variables(21)
  org-move-subtree-down(1)
  funcall-interactively(org-move-subtree-down 1)
  call-interactively(org-move-subtree-down)
  org-metadown(nil)
  funcall-interactively(org-metadown nil)
  call-interactively(org-metadown nil nil)
  command-execute(org-metadown)

Directions to reproduce

I haven't been able to reproduce this in a clean environment yet - I tried installing org cleanly, and that works, but I am pretty sure this has to do with upgrading from an older version and restarting emacs in the meantime. I do notice that every time I rm -rf ~/.emacs.d/straight/build/org, org often (but not always!) gets rebuilt with the same kinds of quirks.

Version information

• Emacs version: 26.3
• Operating system: macOS 10.15.2

[raxod502]

I don't understand how that error can happen. The function org-move-subtree-down is defined in the file org.el, which includes

(eval-and-compile (require 'org-macs))

which means that org-macs ought to be loaded at runtime (since it's eval-and-compile, not eval-when-compile), and there shouldn't be any way that macro isn't defined.

That said, this isn't the first time I've seen an "impossible" bug, so I guess we'll just have to see. Perhaps you could look at the org.elc file and see if there's anything peculiar, and see if the problem is solved by explicitly deleting org.elc from the build directory?

[antifuchs]

I wonder if there's something in my configuration that loads an "old" version of org before straight's version, maybe? I'm not sure what the org that ships with emacs defines, but if it differs substantially from the 9.x's that I've been installing, this might explain things. I'll try and move the declaration wayyy up so we can eliminate that cause.

Re. your suggestion that I delete org.elc from the build dir - there's one thing I noticed: that did work when I did it in one emacs session (after manually loading org-macs and others). However, after I restarted emacs, straight started recompiling all of org again, blowing away the good&fixed code and writing a .elc with the macros represented as function calls again. Is this related to the build-time tracking from #442? (I'm on macOS after all, it would be just seconds-based...)

[raxod502]

I wonder if there's something in my configuration that loads an "old" version of org before straight's version, maybe?

Yes, that must be the issue. It would explain everything -- requiring org-macs wouldn't do anything if some other version had already been loaded.

However, after I restarted emacs, straight started recompiling all of org again

That's a bug. Only the source directory should be checked for modifications, not the build directory. The only thing we check on the build directory is that it exists. If that's something you can reproduce, feel free to open an issue for it and I can look into it.

[kkhan01]

Quite an interesting issue. I've got another impossible bug in the same vein:

(use-package org
  :straight org-plus-contrib)

Seems to load up org-macs fine on a mac, but fails on my linux machine :pensive:

[raxod502]

Do you see anything unexpected in when packages are rebuilt? (This is all logged to *Messages*.) If so, then that would point to #442 as suggested by @antifuchs.

[kkhan01]

I didn't see anything to note in *Messages*.

I went down the Doom route and manually control the org package rather than the straight fix for it. It seems to have resolved my issue, though I'm not how helpful that is to you or trying to root cause the initial issue. Here is my fix, though it's not elegant, it gets the job done for now.

[raxod502]

This is a fine workaround. In fact, for some years I have thought that straight.el should provide a convenient facility for removing built-in packages from the load path.

[antifuchs]

@kkhan01 thanks for that - I applied your solution in a similar way, and it seems my org-mode is now better-behaved.

You have my +1 to deleting built-in packages; those kinds of incompatibilities only cause problems.

[progfolio]

657 Should have taken care of the case of a mixed byte-compilation environment.

That combined with :pre-build commands allows us to properly build Org.
Closing this, but feel free to comment/reopen if anyone is still experiencing issues.