Doom-emacs: lsp-python-ms-update-server "Keyword argument stdout not one of ..."

[steven-lang]

What did you expect to happen?
When running lsp-python-ms-update-server, the Microsoft Python Language Server is supposed to be downloaded and extracted in the background.

What actually happened?
When running lsp-python-ms-update-server, I get the following error:

Debugger entered--Lisp error: (error "Keyword argument :stdout not one of (:name :buffer...")
  signal(error ("Keyword argument :stdout not one of (:name :buffer..."))
  error("Keyword argument %s not one of (:name :buffer :com..." :stdout)
  make-process--with-editor-process-filter(#<subr make-process> :name "sh" :command ("sh" "-c" "mkdir -p /home/tak/.emacs.d/.cache/lsp/mspyls/ && ...") :sentinel #f(compiled-function (proc _) #<bytecode 0x155769235a01>) :stdout " *lsp-install*" :buffer " *lsp-install*" :noquery t)
  apply(make-process--with-editor-process-filter #<subr make-process> (:name "sh" :command ("sh" "-c" "mkdir -p /home/tak/.emacs.d/.cache/lsp/mspyls/ && ...") :sentinel #f(compiled-function (proc _) #<bytecode 0x155769235a01>) :stdout " *lsp-install*" :buffer " *lsp-install*" :noquery t))
  make-process(:name "sh" :command ("sh" "-c" "mkdir -p /home/tak/.emacs.d/.cache/lsp/mspyls/ && ...") :sentinel #f(compiled-function (proc _) #<bytecode 0x155769235a01>) :stdout " *lsp-install*" :buffer " *lsp-install*" :noquery t)
  lsp-async-start-process(#f(compiled-function () #<bytecode 0x155769235a41>) lsp--error "sh" "-c" "mkdir -p /home/tak/.emacs.d/.cache/lsp/mspyls/ && ...")
  #f(compiled-function (data) #<bytecode 0x155768fb96fd>)((:peer (:certificates ((:version 3 :serial-number "1c:00:18:3f:dd:7d:71:af:4a:86:94:e0:b8:00:00:00:18..." :issuer "C=US,ST=Washington,L=Redmond,O=Microsoft Corporati..." :valid-from "2020-04-22" :valid-to "2022-04-22" :subject "CN=*.blob.core.windows.net" :public-key-algorithm "RSA" :certificate-security-level "Medium" :signature-algorithm "RSA-SHA256" :public-key-id "sha1:2b:37:b4:6d:49:c1:65:c4:cc:0f:81:c4:76:9d:a9:..." :certificate-id "sha1:6b:a7:63:f9:7c:d6:cd:46:d0:94:de:75:64:c5:eb:..." :pem "-----BEGIN CERTIFICATE-----\nMIIONTCCDB2gAwIBAgITHA...") (:version 3 :serial-number "0f:2c:10:c9:5b:06:c0:93:7f:b8:d4:49:f8:3e:85:69" :issuer "C=IE,O=Baltimore,OU=CyberTrust,CN=Baltimore CyberT..." :valid-from "2016-05-20" :valid-to "2024-05-20" :subject "C=US,ST=Washington,L=Redmond,O=Microsoft Corporati..." :public-key-algorithm "RSA" :certificate-security-level "High" :signature-algorithm "RSA-SHA256" :public-key-id "sha1:aa:d9:58:e0:47:31:fa:48:58:a9:b5:20:0f:22:4b:..." :certificate-id "sha1:54:d9:d2:02:39:08:0c:32:31:6e:d9:ff:98:0a:48:..." :pem "-----BEGIN CERTIFICATE-----\nMIIFtDCCBJygAwIBAgIQDy...")) :certificate (:version 3 :serial-number "1c:00:18:3f:dd:7d:71:af:4a:86:94:e0:b8:00:00:00:18..." :issuer "C=US,ST=Washington,L=Redmond,O=Microsoft Corporati..." :valid-from "2020-04-22" :valid-to "2022-04-22" :subject "CN=*.blob.core.windows.net" :public-key-algorithm "RSA" :certificate-security-level "Medium" :signature-algorithm "RSA-SHA256" :public-key-id "sha1:2b:37:b4:6d:49:c1:65:c4:cc:0f:81:c4:76:9d:a9:..." :certificate-id "sha1:6b:a7:63:f9:7c:d6:cd:46:d0:94:de:75:64:c5:eb:..." :pem "-----BEGIN CERTIFICATE-----\nMIIONTCCDB2gAwIBAgITHA...") :key-exchange "ECDHE-RSA" :protocol "TLS1.2" :cipher "AES-256-GCM" :mac "AEAD" :encrypt-then-mac nil :safe-renegotiation t)))
  apply(#f(compiled-function (data) #<bytecode 0x155768fb96fd>) (:peer (:certificates ((:version 3 :serial-number "1c:00:18:3f:dd:7d:71:af:4a:86:94:e0:b8:00:00:00:18..." :issuer "C=US,ST=Washington,L=Redmond,O=Microsoft Corporati..." :valid-from "2020-04-22" :valid-to "2022-04-22" :subject "CN=*.blob.core.windows.net" :public-key-algorithm "RSA" :certificate-security-level "Medium" :signature-algorithm "RSA-SHA256" :public-key-id "sha1:2b:37:b4:6d:49:c1:65:c4:cc:0f:81:c4:76:9d:a9:..." :certificate-id "sha1:6b:a7:63:f9:7c:d6:cd:46:d0:94:de:75:64:c5:eb:..." :pem "-----BEGIN CERTIFICATE-----\nMIIONTCCDB2gAwIBAgITHA...") (:version 3 :serial-number "0f:2c:10:c9:5b:06:c0:93:7f:b8:d4:49:f8:3e:85:69" :issuer "C=IE,O=Baltimore,OU=CyberTrust,CN=Baltimore CyberT..." :valid-from "2016-05-20" :valid-to "2024-05-20" :subject "C=US,ST=Washington,L=Redmond,O=Microsoft Corporati..." :public-key-algorithm "RSA" :certificate-security-level "High" :signature-algorithm "RSA-SHA256" :public-key-id "sha1:aa:d9:58:e0:47:31:fa:48:58:a9:b5:20:0f:22:4b:..." :certificate-id "sha1:54:d9:d2:02:39:08:0c:32:31:6e:d9:ff:98:0a:48:..." :pem "-----BEGIN CERTIFICATE-----\nMIIFtDCCBJygAwIBAgIQDy...")) :certificate (:version 3 :serial-number "1c:00:18:3f:dd:7d:71:af:4a:86:94:e0:b8:00:00:00:18..." :issuer "C=US,ST=Washington,L=Redmond,O=Microsoft Corporati..." :valid-from "2020-04-22" :valid-to "2022-04-22" :subject "CN=*.blob.core.windows.net" :public-key-algorithm "RSA" :certificate-security-level "Medium" :signature-algorithm "RSA-SHA256" :public-key-id "sha1:2b:37:b4:6d:49:c1:65:c4:cc:0f:81:c4:76:9d:a9:..." :certificate-id "sha1:6b:a7:63:f9:7c:d6:cd:46:d0:94:de:75:64:c5:eb:..." :pem "-----BEGIN CERTIFICATE-----\nMIIONTCCDB2gAwIBAgITHA...") :key-exchange "ECDHE-RSA" :protocol "TLS1.2" :cipher "AES-256-GCM" :mac "AEAD" :encrypt-then-mac nil :safe-renegotiation t)))
  url-http-activate-callback()
  url-http-content-length-after-change-function(34017691 34021196 3505)
  url-http-generic-filter(#<process pvsc.blob.core.windows.net> "PK\1\2\24\0\24\0\0\0\10\0:\250\241P\300-6Y\36\0\0\0\34\0\0\0001\0\0\0\0\0\0\0\0\0\0\0\0\0Zi\4\2Type...")

I've checked this in the doom sandbox:

• vanilla Doom (only Doom core) -- no issue
• Doom + modules - your private config -- error

Additional details:

• Include a link to your private config: https://github.com/steven-lang/dotfiles/tree/master/configs/emacs/.doom.d

Steps to reproduce:

1. Start Emacs
2. (Optional) Enter Doom + modules - your private config sandbox mode via doom/sandbox
3. M-x lsp-python-ms-update-server
4. Wait a few seconds until the download is finished, then the error can be observed

System information:

emacs   version    27.0.91

        features   XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GLIB NOTIFY INOTIFY ACL GNUTLS LIBXML2 FREETYPE HARFBUZZ LIBOTF ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD JSON PDUMPER LCMS2 GMP

        build      Apr 23, 2020

        buildopts  (--prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games --with-sound=alsa --with-modules --without-gconf --without-gsettings --with-x-toolkit=gtk3 --without-xaw3d --without-m17n-flt --with-cairo --without-compress-install 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -g -flto -s -fuse-ld=gold' CPPFLAGS=-D_FORTIFY_SOURCE=2 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now)

        windowsys  x

        daemonp    server-running

doom    version    2.0.9

        build      HEAD -> develop 847bbf380 2020-06-13 17:12:23 -0400

        dir        ~/dotfiles/configs/emacs/.doom.d/

system  type       gnu/linux

        config     x86_64-pc-linux-gnu

        shell      /bin/bash

        uname      Linux 5.6.15-arch1-1 #1 SMP PREEMPT Wed, 27 May 2020 23:42:26 +0000 x86_64

        path       (~/.emacs.d/bin/../bin ~/bin ~/.gem/ruby/2.7.0/bin /usr/local/bin /usr/bin /bin /usr/local/sbin /usr/lib/jvm/default/bin /usr/bin/site_perl /usr/bin/vendor_perl /usr/bin/core_perl ~/.antigen/bundles/robbyrussell/oh-my-zsh/lib ~/.antigen/bundles/robbyrussell/oh-my-zsh/plugins/git ~/.antigen/bundles/robbyrussell/oh-my-zsh/plugins/git-flow-avh ~/.antigen/bundles/robbyrussell/oh-my-zsh/plugins/gradle ~/.antigen/bundles/robbyrussell/oh-my-zsh/plugins/z ~/.antigen/bundles/robbyrussell/oh-my-zsh/plugins/fzf ~/.antigen/bundles/zsh-users/zsh-autosuggestions ~/.antigen/bundles/zsh-users/zsh-syntax-highlighting ~/.antigen/bundles/mafredri/zsh-async ~/.antigen/bundles/sindresorhus/pure /usr/local/sbin /usr/local/bin /usr/bin /usr/lib/jvm/default/bin /usr/bin/site_perl /usr/bin/vendor_perl /usr/bin/core_perl ~/.local/bin ~/bin/ ~/.cargo/bin/ /opt/android-sdk/platform-tools/ /usr/bin/julia ~/.emacs.d/bin ~/.fzf/bin /usr/lib/emacs/27.0.91/x86_64-pc-linux-gnu/)

config  envfile    envvar-file

        elc-files  0

        modules    (:completion (company +childframe) (ivy +icons +prescient) :ui deft doom doom-dashboard hl-todo hydra modeline nav-flash (popup +all +defaults) vc-gutter vi-tilde-fringe window-select workspaces :editor (evil +everywhere) file-templates fold format snippets word-wrap :emacs dired electric vc :term term :checkers syntax spell grammar :tools debugger direnv (eval +overlay) (lookup +docsets) (lsp +peek) magit pdf :lang data emacs-lisp (latex +latexmk) markdown (org +dragndrop +pomodoro +roam +present) (python +lsp +pyenv) sh :email (mu4e +gmail) :config (default +bindings +smartparens))

        packages   ((mu4e) (winum) (edit-server) (blacken) (helm) (yasnippet-snippets) (lorem-ipsum) (ssh-agency) (aggressive-indent) (mu4e-alert) (mixed-pitch) (cuda-mode) (dockerfile-mode) (yaml-mode) (evil-snipe :disable t) (org-gcal) (org-super-agenda) (org-ql) (org-pomodoro) (org-fragtog) (org-noter) (org-pdftools) (org-noter-pdftools :recipe (:host github :repo fuxialexander/org-pdftools)) (org-roam :recipe (:host github :repo jethrokuan/org-roam)) (org-fragtog) (org-superstar))

        unpin      (n/a)

        elpa       (undo-tree shrink-path s popup memoize helm-core helm goto-chg f evil doom-modeline dash async all-the-icons)

[drchsl]

I get the same error, also tried a complete reinstall.

Something similar happens when i try to execute an emacs-jupyter
src block ("Keyword argument :coding-system not one of ...").

The really strange thing is, BOTH errors disappeared when i replaced
make-process--with-editor-process-filter with a version that takes
coding-system instead of coding.

EDIT: seeing this i tried overriding make-process-adapter-coding-a with... an unchanged version of itself in my config.el - and it works:

(defadvice! make-process-adapter-coding-a
  (fn &rest keys &key name buffer command coding noquery stop
      connection-type filter sentinel stderr file-handler)
  :override #'make-process--with-editor-process-filter
  (if (or (not file-handler) (not with-editor--envvar))
      (apply fn keys)
    (when (file-remote-p default-directory)
      (unless (equal (car command) "env")
        (push "env" command))
      (push (concat with-editor--envvar "=" with-editor-sleeping-editor)
            (cdr command)))
    (let* ((filter (if filter
                       (lambda (process output)
                         (funcall filter process output)
                         (with-editor-process-filter process output t))
                     #'with-editor-process-filter))
           (process (funcall fn
                             :name name
                             :buffer buffer
                             :command command
                             :coding coding
                             :noquery noquery
                             :stop stop
                             :connection-type connection-type
                             :filter filter
                             :sentinel sentinel
                             :stderr stderr
                             :file-handler file-handler)))
      (process-put process 'default-dir default-directory)
      process)))

Probably some kind of load order problem?

Stacktraces are here: https://pastebin.com/YHkS6jAE
I'm on Emacs 27.0.91, Ubuntu 20.04

[betaprior]

Confirming the bug. Workaround above works, but is ugly AF. The make-process--with-editor-process-filter is an advice defined by magit, so I'm guessing that's why doom core (without magit) doesn't show the issue. Reading the docs for this advice, it doesn't seem it should be running here but it is.

This breakage is new because the advice in question was added to magit very recently (see the with-editor repo, commit d79c32497)

[tarsius]

Should make-process really be called with a :stdout keyword argument? I looked in the doc-string, manual and source code (or Emacs 28); and did not see such an argument mentioned anywhere

[betaprior]

Should make-process really be called with a :stdout keyword argument? I looked in the doc-string, manual and source code (or Emacs 28); and did not see such an argument mentioned anywhere

I guess the answer is "no" and it was just fixed upsream (note to @hlissner -- LSP mode should be bumped to fix this). However, it is still quite concerning that advice from mode A makes a crucial Emacs API call fail in mode B, even if mode B was performing an out-of-spec invocation. I don't have deep enough understanding of the pertinent code to understand what the advice is doing, but if it is in fact the cause of this failure, could it be restricted only to magit contexts? @tarsius - can you shed some light on this if you don't mind?

[tarsius]

The only reason that my advice fails and the original function does not is that the latter implements the argument handling in C while the advice gets to use cl-defun &key. cl-defun checks that no invalid keyword arguments are used which the C implementation fails to do. And that could be considered a bug.

So basically my advice helped find a bug in lsp-python, which could otherwise have gone unnoticed for a long time. I don't intend to "fix" that. I consider it an unintended but nice side-effect.

[tarsius]

I don't have deep enough understanding of the pertinent code to understand what the advice is doing, but if it is in fact the cause of this failure, could it be restricted only to magit contexts?

Magit doesn't even use this.

This advice only changes make-process's behavior if it was called like so:

(with-editor
  (make-process ...))

That and, as we have seen, if the arguments are invalid.

[tarsius]

Well I suppose since the doc-string says

This is similar to start-process, but arguments are specified as
keyword/argument pairs. The following arguments are defined:

(emphasis mine) one could argue that one is allowed to pass invalid arguments. And I also don't want to have to deal with this again, so I am changing my mind and add &allow-other-keys (see https://www.gnu.org/software/emacs/manual/html_node/cl/Argument-Lists.html).

[visini]

https://github.com/magit/with-editor/commit/7c512887c6d69864fb600d32fb92857c51babcff

Can confirm that this fixed this error for me (doom upgrade). Thank you!

[hlissner]

Between 042cc781b (bumped magit) and 15cad3038 (bumped lsp), this should be resolved. Let me know if that isn't the case and I'll reopen this. Thanks for bringing it to my attention!