Lsp-mode: Feature: automatic server installation

[yyoncho]

We could use https://gitlab.com/jabranham/system-packages to automatically download and install language servers that have published their packages(npm/gem/pip).

NB: the servers that does not have installation package won't be handled by this feature.

(cc @jabranham)

[seagle0128]

Some comments form my experience. I tried this package with use-package a few months ago. It's unable to handle all scenarios on all platforms, especially for Windows and sudo with Linux. So it's a good feature but need more cautions.

[jabranham]

@seagle0128 What issues did you run into with sudo on linux? Would you mind opening a new issue for it?

[seagle0128]

@jabranham I remember that some commands need sudo privileges while others don't. During Emacs startup, there will be many prompts to interrupt, and really annoying! Especially on remote development environment which didn't install all packages :-(

[yyoncho]

We may use the vscode package for the installation, I am planning to do that for dap-mode.

[elxbarbosa]

@jabranham
Ok!

@seagle0128
Just tried system-packages today, it is simpler (no prompt at startup), and it correctly guess by checking the distro package if its needs sudo to install packages!

I had just a minor annoyance, if you have guix or nix installed in your distro it prefer those two package-manager rather than the distro default (eg: instead of apt it installs with guix)...

Opening a issue late!

[seagle0128]

@eubarbosa No password prompt for sudo on Linux?

I used system-packages for legacy lsp-mode to try installing all servers at startup. I think the current situation is different. If the servers are installed and started on-demand, the annoyances I met should not exist.

[tam5]

I'm all for auto installation of language servers, with the condition that a user could customize this to turn it off and specify the language server they want to use directly.

Also, I think a quicker/easier fix, should this take a little while to get done, would be to just add more detailed and exact instructions for installing each language server, so someone new to the idea of lsp would still be able to get up and running quickly

[alanz]

I think providing a pointer to install instructions makes more sense. Some language servers are fairly complex to install, and may involve interaction with the system package manager.

[yyoncho]

@alanz that's correct, I will update the feature description. In general I was referencing the following servers:

1. Servers that are available as a package like (pyls)
2. Servers that are packaged in VSCode extension(php).

[Pitometsu]

Lsp server may depend on per-project environment requirements (like version).
That's why system wide installation is a wrong way. There's nix environments for that and ad-hoc solutions like development in docker.

[yyoncho]

@Pitometsu I closed as wont fix https://github.com/emacs-lsp/lsp-mode/issues/440 which was intended to be used for testing purposes. Maybe we could have what you are suggesting as a separate issue?

In this PR I am looking for more lightweight solution, similar to what you have in vscode where you just download the extension and it works, no other configurations are needed.

[elxbarbosa]

@yyoncho

VSCode offers to "install" extensions just like lsp-java as we open a .xxx file!

That seems a good way to ask to install servers!

[jcs090218]

Any update for this feature?

[yyoncho]

Any update for this feature?

We have it for fsharp/java and ms python. We have automatic installation using vscode extensions for a lot of dap-mode adapters(implemented by @kiennq ) and we could potentially use that same approach for language servers (see https://github.com/emacs-lsp/dap-mode/issues/104). ATM we do not have the manpower to implement it for each server so we pretty much depend on contributions(like 99% of the FOSS projects).

[jcs090218]

I see, is there a method that you prefer to use? Either using system-packages or dap-utils???

[kiennq]

If I can put my opinion here, somethings that doesn't leave Windows user unusable would be the best.

[yyoncho]

I see, is there a method that you prefer to use? Either using system-packages or dap-utils???

IMO the easiest way to support that is to follow what VScode is doing which will address also @kiennq concern. Note that some of the servers cannot be just downloaded(e. g. the C++ one).

[jcs090218]

I'm pretty new to lsp. Do you know what do VSCode does when installing the language server? My first thought would just be an interface that controls all the installation flow to each server by just inputting the installation commands. Seconds, step will be the improvement for each language server's installation.

[seagle0128]

My two cents.

I developed the automatic server installation for lsp-python-ms, which has the similar behaviors as dap-mode. It looks good so far except one minor issue: there is no progress indicator for downloading or extracting. The users don't know what's happening when the Emacs is hanging if there is a connection issue. I believe the most of servers have the similar behaviors. So I am thinking it's better if we have a common utils to handle these situations and scenarios. And it's also easier to add this feature when the developers implement new plugins. dap-utils is a good start. IMO system-packages is good for user configurations while not for developers. I agree with @yyoncho , the easiest way is following VSCode.

[yyoncho]

Do you know what do VSCode does when installing the language server?

It depends - some of the language servers are packaged in the vsix extension that is downloaded from https://marketplace.visualstudio.com/ (e. g. JDT language server) others like ms python language server are downloaded from the extension using JS code. I think that RLS is downloaded using rustup. I think that the user is supposed to install clangd manually in a platform-specific way and there is no automation for it.

[yyoncho]

Few random thoughts based on what we currently have, follow up from https://github.com/emacs-lsp/lsp-mode/pull/1147

1. IMO we should extend the client definition with :download-function + :download-in-progress which accepts the current client to mark the download in progress.
2. We should change the startup process to detect that the server is missing and ask the user whether to download the server.
3. In case there are multiple installable servers we should ask the client to select one.
4. If there is no automatic installation we should redirect the user to instructions on how to download it (suggested by @Gastove )
5. There should be a flag to disable all these interactive actions.
6. We should make the server installation async. lsp startup must be delayed until the server is downloaded an then the server must be initialized in the corresponding buffers(or use lsp-deffered).
7. We should also provide a way to do the setup manually.
8. We should try to put these under CI.
9. To address the lack of progress we might implement scripts for utilizing wget/python/nodejs/(?) if they are available. (@seagle0128's concern).
10. (optionally) Provide a dashboard which shows what servers do we have configured.

And what we currently have

1. lsp-java has a pom.xml which is used to download the server automatically
2. dap-mode has facilities to download and extract vscode extensions
3. lsp-python-ms and fsharp synchronously download the server

[akirak]

I was working on an idea for installing servers using Nix: https://github.com/akirak/lsp-server.el

It parses the README of lsp-mode package and tries to discover an installation instruction for the language. It currently supports npm packages.

I am willing to contribute to lsp-mode package. Do you have any idea?

[yyoncho]

@akirak - using nix is a cool idea!

It parses the README of lsp-mode package and tries to discover an installation instruction for the language. It currently supports npm packages.

What do you think about having a field in the lsp--client structure that directly contains the name of the nix package?

[akirak]

What do you think about having a field in the lsp--client structure that directly contains the name of the nix package?

That would be a good idea and help develop my package. Note that some language servers are suggested for installation using a language-specific package manager. For example, the language server for Elixir can be installed using mix, but it does not seem directly available from nixpkgs. The same for Golang. I was thinking about how to handle such packages. For npm packages, I implemented a wrapper which is available as part of nix-env-install.

Nix is not (officially) supported on Windows, so any Nix-based solutions should probably not be considered the official way at present. Nix on WSL does work, and there seems to be attempts to use Nix on native/mingw Windows, but I am not sure if any of them are stable.

[yyoncho]

Nix is not (officially) supported on Windows,

I do not think that we should limit the download/install options to nix. Maybe it will be the preferable way if it is present but definitely we should support other ways(e. g. Download+Extract).

I think that we should change lsp-register-client to be macro generating variables e. g.

(lsp-register-client
   :server-id 'foo
   :binary-path '("/usr/bin/foo" "--stdio")
   ;;:nix-package
)

... will generate:

... 
(defcustom lsp-foo-binary-path '("/usr/bin/foo" "--stdio"))
...

What do you think?

[akirak]

@yyoncho
Thank you for the response.

I do not think that we should limit the download/install options to nix. Maybe it will be the preferable way if it is present but definitely we should support other ways(e. g. Download+Extract).

Yeah, I love Nix, but since it is not available for everyone, we should provide other options.

I think that we should change lsp-register-client to be macro generating variables e. g.

It would be a good idea. My lsp-server package has the following features, but it would be better if the information were explicitly specified in individual client definitions:

• It scans elisp source files in your repository and extract lsp-register-client forms (lsp-server--builtin-files and lsp-server--index-groups).
• It extracts executable names from the client definitions(lsp-server--extract-server-executable).
• It parses the README of lsp-mode and extract installation instructions, but also allows the user to override recipes (lsp-server-package-alist).
• Install packages in the background, if you use my nix-env-install package.

Language servers may not be available in the user's default exec-path, so it would be a good idea to make executable names customizable, and it can be generated in a macro.

The recipe overrides are defined as follows. It would be better if the default values were provided in client definitions.

(defcustom lsp-server-package-alist
  '((bash-ls . (npm "bash-language-server"))
    (fsac . (function lsp-fsharp--fsac-install))
    (elixir-ls . (error "Please install elixir-ls for yourself")))
  "Alist of package specifications for servers."
  :group 'lsp-server
  :type '(alist :key-type symbol
                :value-type sexp))

(defun lsp-server--install-with-server-spec (spec)
  "Install server packages from SPEC."
  (pcase spec
    (`(npm . ,packages)
     (if lsp-server-use-nix
         (progn
           (require 'nix-env-install)
           (nix-env-install-npm packages))
       (lsp-server--run-command (format "npm install -g %s"
                                        (mapconcat #'shell-quote-argument
                                                   packages ' ')))))
    (`(function ,func)
     (funcall func))
    (`(error ,msg)
     (message msg))
    (_
     (error "Unsupported spec %s" spec))))

My package currently lets the user install servers by manually running lsp-server-install command, but it should be part of the following step:

1. We should change the startup process to detect that the server is missing and ask the user whether to download the server.

As for the async installation, I would like you to take a look at my implementation in nix-env-install package. It runs an external process in the background and displays the progress in another window. If you wish, I will contribute the function to your package.

[yyoncho]

@akirak

If you wish, I will contribute the function to your package.

I like the idea. This will definitely go in but let's first lay down the foundation of download framework and then we will proceed with adding the concrete implementations. I will open a draft PR once I have something so you could provide your feedback.

[akirak]

I've changed how nix-env-install displays process buffers and process command output. Now it uses display-buffer-in-side-window to display buffers, which is a more sophisticated way to display a side window. If the user has xterm-color-filter package installed, command output is colorized. I've also fixed lint errors of the package, so hopefully it will be available on MELPA soon (https://github.com/melpa/melpa/pull/6452).

[yyoncho]

I have a lot of the code already in the https://github.com/yyoncho/lsp-mode/tree/downloads . I will post a PR soon.