Straight.el: Support arbitrary build commands

[raxod502]

Some packages (like Org) require you to make them before they can be run correctly. Others have nonstandard types of documentation, and so on.

[sooheon]

So to be clear one cannot currently have straight.el fully manage org?

[raxod502]

@sooheon No, it works just fine. You'll just get some warnings about the Org version not being defined, unless you add some hacks.

[dieggsy]

Can confirm it works just fine. I don't think I get any version warnings (?) but that hack is cool anyway.

[rgrinberg]

One package that doesn't work at all without a build step is proof-general. It's quite an important package as well so it would be nice to have support.

[dieggsy]

@rgrinberg the same may be true of pdf-tools

[vyp]

also structured-haskell-mode

[raxod502]

Indeed. You can probably get a pretty good idea of the list of such packages by just searching all the el-get recipes, since those specify build commands.

Are those packages not in MELPA? Because as I understand it, MELPA does not support build commands.

In any case, I am quite busy at the moment, but would gladly merge any pull requests that came along.

[dieggsy]

@raxod502 this is pretty late but I think MELPA does support build commands, see here:

https://github.com/melpa/melpa/blob/269acf093c3aa508479bf532a4e07c9c6d568c72/recipes/pdf-tools#L1

I believe pdf-tools gets compiled at package-install time.

[raxod502]

@dieggsy MELPA doesn't build pdf-tools. What you see in the recipe is just a specification to create a build/ directory in the built package and to link the named files into it:

drwxr-xr-x     - raxod502  2 Dec  7:29  .
drwxr-xr-x     - raxod502  2 Dec  7:29  ├── build
lrwxr-xr-x    58 raxod502  2 Dec  7:29  │  ├── Makefile -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/Makefile
drwxr-xr-x     - raxod502  2 Dec  7:29  │  └── server
lrwxr-xr-x    67 raxod502  2 Dec  7:29  │     ├── .gitignore -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/.gitignore
lrwxr-xr-x    63 raxod502  2 Dec  7:29  │     ├── ar-lib -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/ar-lib
lrwxr-xr-x    67 raxod502  2 Dec  7:29  │     ├── autogen.sh -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/autogen.sh
lrwxr-xr-x    64 raxod502  2 Dec  7:29  │     ├── compile -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/compile
lrwxr-xr-x    69 raxod502  2 Dec  7:29  │     ├── configure.ac -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/configure.ac
lrwxr-xr-x    67 raxod502  2 Dec  7:29  │     ├── epdfinfo.c -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/epdfinfo.c
lrwxr-xr-x    67 raxod502  2 Dec  7:29  │     ├── epdfinfo.h -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/epdfinfo.h
lrwxr-xr-x    68 raxod502  2 Dec  7:29  │     ├── Makefile.am -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/Makefile.am
lrwxr-xr-x    72 raxod502  2 Dec  7:29  │     ├── poppler-hack.cc -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/poppler-hack.cc
lrwxr-xr-x    73 raxod502  2 Dec  7:29  │     ├── poppler-versions -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/poppler-versions
lrwxr-xr-x    73 raxod502  2 Dec  7:29  │     ├── synctex_parser.c -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/synctex_parser.c
lrwxr-xr-x    73 raxod502  2 Dec  7:29  │     ├── synctex_parser.h -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/synctex_parser.h
lrwxr-xr-x    79 raxod502  2 Dec  7:29  │     ├── synctex_parser_local.h -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/synctex_parser_local.h
lrwxr-xr-x    82 raxod502  2 Dec  7:29  │     ├── synctex_parser_readme.txt -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/synctex_parser_readme.txt
lrwxr-xr-x    79 raxod502  2 Dec  7:29  │     ├── synctex_parser_utils.c -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/synctex_parser_utils.c
lrwxr-xr-x    79 raxod502  2 Dec  7:29  │     ├── synctex_parser_utils.h -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/synctex_parser_utils.h
lrwxr-xr-x    83 raxod502  2 Dec  7:29  │     └── synctex_parser_version.txt -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/server/synctex_parser_version.txt
lrwxr-xr-x    67 raxod502  2 Dec  7:29  ├── pdf-annot.el -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/lisp/pdf-annot.el
.rw-r--r--   59k raxod502  2 Dec  7:29  ├── pdf-annot.elc
lrwxr-xr-x    67 raxod502  2 Dec  7:29  ├── pdf-cache.el -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/lisp/pdf-cache.el
.rw-r--r--   15k raxod502  2 Dec  7:29  ├── pdf-cache.elc
lrwxr-xr-x    65 raxod502  2 Dec  7:29  ├── pdf-dev.el -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/lisp/pdf-dev.el
.rw-r--r--  3.1k raxod502  2 Dec  7:29  ├── pdf-dev.elc
lrwxr-xr-x    69 raxod502  2 Dec  7:29  ├── pdf-history.el -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/lisp/pdf-history.el
.rw-r--r--  5.4k raxod502  2 Dec  7:29  ├── pdf-history.elc
lrwxr-xr-x    66 raxod502  2 Dec  7:29  ├── pdf-info.el -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/lisp/pdf-info.el
.rw-r--r--   48k raxod502  2 Dec  7:29  ├── pdf-info.elc
lrwxr-xr-x    69 raxod502  2 Dec  7:29  ├── pdf-isearch.el -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/lisp/pdf-isearch.el
.rw-r--r--   24k raxod502  2 Dec  7:29  ├── pdf-isearch.elc
lrwxr-xr-x    67 raxod502  2 Dec  7:29  ├── pdf-links.el -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/lisp/pdf-links.el
.rw-r--r--   11k raxod502  2 Dec  7:29  ├── pdf-links.elc
lrwxr-xr-x    66 raxod502  2 Dec  7:29  ├── pdf-misc.el -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/lisp/pdf-misc.el
.rw-r--r--   13k raxod502  2 Dec  7:29  ├── pdf-misc.elc
lrwxr-xr-x    67 raxod502  2 Dec  7:29  ├── pdf-occur.el -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/lisp/pdf-occur.el
.rw-r--r--   29k raxod502  2 Dec  7:29  ├── pdf-occur.elc
lrwxr-xr-x    69 raxod502  2 Dec  7:29  ├── pdf-outline.el -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/lisp/pdf-outline.el
.rw-r--r--   21k raxod502  2 Dec  7:29  ├── pdf-outline.elc
lrwxr-xr-x    66 raxod502  2 Dec  7:29  ├── pdf-sync.el -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/lisp/pdf-sync.el
.rw-r--r--   24k raxod502  2 Dec  7:29  ├── pdf-sync.elc
.rw-r--r--   12k raxod502  2 Dec  7:29  ├── pdf-tools-autoloads.el
lrwxr-xr-x    67 raxod502  2 Dec  7:29  ├── pdf-tools.el -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/lisp/pdf-tools.el
.rw-r--r--   10k raxod502  2 Dec  7:29  ├── pdf-tools.elc
lrwxr-xr-x    66 raxod502  2 Dec  7:29  ├── pdf-util.el -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/lisp/pdf-util.el
.rw-r--r--   36k raxod502  2 Dec  7:29  ├── pdf-util.elc
lrwxr-xr-x    66 raxod502  2 Dec  7:29  ├── pdf-view.el -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/lisp/pdf-view.el
.rw-r--r--   51k raxod502  2 Dec  7:29  ├── pdf-view.elc
lrwxr-xr-x    69 raxod502  2 Dec  7:29  ├── pdf-virtual.el -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/lisp/pdf-virtual.el
.rw-r--r--   82k raxod502  2 Dec  7:29  ├── pdf-virtual.elc
lrwxr-xr-x    56 raxod502  2 Dec  7:29  └── README -> /Users/raxod502/.emacs.d/straight/repos/pdf-tools/README

So, you are correct in that any compilation happens on the client—which means it's orthogonal to straight.el, whose build step is analogous to what happens on the MELPA server.

The issue is how to handle packages which need to be built but, unlike pdf-tools, do not themselves handle it automatically.

[shackra]

is there any PR related to this issue? I was toying with https://github.com/ubolonton/emacs-tree-sitter and this feature would come handy

[raxod502]

No, there's no implementation, although you could easily hack something together using straight-use-package-pre-build-functions. See https://github.com/raxod502/straight.el/issues/389.

Personally, I think this is not something that should be taken care of by the package manager. Rather, each package that has custom build steps should offer a facility for running them (automatically, perhaps) when needed. That way, each package can decide for itself what needs to be run and when, reducing complexity in the package manager and also reducing the amount of time that users may need to wait for spurious rebuilds to complete. The problem of code duplication could be solved by a shared dependency that takes care of all the necessary user interface code, making the functionality only take a few lines of code to add to any package.

[YorkZ]

Not sure if it's related to this issue. While magit.info is built in ~/.emacs.d/straight/build/magit, the Info file for the package org-plus-contrib is not built in ~/.emacs.d/straight/build/org-plus-contrib.

[raxod502]

That's because Org, unlike Magit, does not provide its Texinfo manual in any kind of standard format, hence it's impossible for straight.el to identify and build it without a contribution such as #549 + #535.

[YorkZ]

Just noticed pymacs also cannot be built without supporting arbitrary build commands.

[progfolio]

merged #535 into develop. That should take care of this issue.

[raxod502]

Indeed! Wonderful!

[raxod502]

Is this the most requested feature in straight.el to date?

[shackra]

Is this the most requested feature in straight.el to date?

you bet