Lsp-mode: lsp-prefer-capf doesnt work well

[glepnir]

Describe the bug
Set the lsp-prefer-capf t, It performs exact matching and does not perform filtering well.
Bug 1: I can get completion items , but when input p cant get any completion items.

input p

Bug 2: doesnt filter for gopls.

To Reproduce

1. open a go file
2. use this lsp config and company config

(use-package lsp-mode
  :commands (lsp-install-server lsp lsp-deferred)
  :hook (prog-mode . (lambda ()
                          (unless (derived-mode-p 'emacs-lisp-mode 'lisp-mode)
                            (lsp-deferred))))
  :init
  (setq  lsp-auto-guess-root t
        lsp-prefer-capf  t
        lsp-print-io t
        lsp-keep-workspace-alive nil)
 :config
 (when lsp-auto-configure
    (mapc (lambda (package) (require package nil t))
          lsp-client-packages)))
(use-package company
  :commands company-complete-common company-manual-begin company-grab-line
  :after-call pre-command-hook after-find-file
  :init
  (setq company-tooltip-align-annotations t
        company-tooltip-limit 14
        company-idle-delay 0
        company-echo-delay (if (display-graphic-p) nil 0)
        company-minimum-prefix-length 2
        company-require-match 'never
        company-dabbrev-ignore-case nil
        company-dabbrev-downcase nil
        company-global-modes '(not erc-mode message-mode help-mode gud-mode eshell-mode shell-mode)
        company-backends '(company-capf)
        company-frontends '(company-pseudo-tooltip-frontend
                            company-echo-metadata-frontend))
  :config
  (global-company-mode +1))

Expected behavior

1. doesn't exact match
2. filter the response.

Which Language Server did you use
lsp-go

OS
macos 10.15.3

Error callstack

[kiennq]

Can you provide the language server log according to this?
I cannot repro bug 1:

For bug 2: I can kind of repro it, the problem is that the gopls return candidates with no prefix (the request.position == response.textEdit.range.start).
Thus lsp-capf cannot detect the prefix and cannot filter the completion candidates.

[Trace - 09:08:40 PM] Sending request 'textDocument/completion - (1303)'.
Params: {
  "textDocument": {
    "uri": "file:///home/kienn/projects/go/src/github.com/kiennq/test/a.go"
  },
  "position": {
    "line": 20,
    "character": 4
  },
  "context": {
    "triggerKind": 1
  }
}


[Trace - 09:08:40 PM] Received response 'textDocument/completion - (1303)' in 1ms.
Result: {
  "items": [
    {
      "textEdit": {
        "newText": "bufio",
        "range": {
          "end": {
            "character": 4,
            "line": 20
          },
          "start": {
            "character": 4,
            "line": 20
          }
        }
      },
      "insertTextFormat": 2,
      "filterText": "bufio",
      "sortText": "00000",
      "preselect": true,
      "detail": "\"bufio\"",
      "kind": 9,
      "label": "bufio"
    },...
  ]
}  

What's the behavior of VsCode in this case?

[yyoncho]

For bug 2: I can kind of repro it, the problem is that the gopls return candidates with no prefix (the request.position == response.textEdit.range.start).

Why having empty prefix breaks us?

[kiennq]

Why having empty prefix breaks us?

No prefix means we do no filtering. We just display whatever language server returns.

[yyoncho]

It returns empty prefix even if you type after .?

[kiennq]

No, it seems return empty prefix only when you define new function name.

[kiennq]

@taigacute Not that, but the communication log

set lsp-log-io to t to inspect communication between client and the server. Use lsp-workspace-show-log to switch to the corresponding log buffer.

Also, what's the value of your completion-styles?

[glepnir]

@kiennq i upload the log .please check it .the completion values

completion-styles is a variable defined in minibuffer.el.gz.

Value
(basic partial-completion emacs22)

Original Value
(basic partial-completion emacs22)

log.txt

[kiennq]

Thanks, you can set the completion-styles to the following to archive fuzzy candidate filtering for Emacs 27.

(setq completion-styles `(basic partial-completion emacs22 initials
                                ,(if (version<= emacs-version "27.0") 'helm-flex 'flex)))

[kiennq]

The returns value from your gopls server all have filterText set to Errorf, that's why you see no candidate after typing p. Since the filtering is done against filterText.
What's version of your gopls?
I've tried with latest gopls (v0.3.2) and it seems return correct filterText.

[glepnir]

Dont know why it doesnt work ,I just updated 0.3.2.

completion-styles is a variable defined in minibuffer.el.gz.

Value
(basic partial-completion emacs22 initials flex)

gopls version

golang.org/x/tools/gopls v0.3.2
    golang.org/x/tools/[email protected] h1:eP1aj1AvT6ynElQH6KP0mmOT2gnWa1gYclHL4wGUbMo=

[kiennq]

Could you run M-x lsp-shutdown-workspace and M-x lsp in the main.go.
Then take a trace again?

[glepnir]

this is log ,i found something ..
when input .

but when i input p

log2.txt

[kiennq]

Hmm, the trace still showing the same filter Text for all of returned completion.
Can you check if you're actually using the correct gopls from Emacs?
You can take a look at *lsp-log* buffer and looking for something similar to

2020/02/25 00:45:02 Build info
----------
golang.org/x/tools/gopls v0.3.2
    golang.org/x/tools/[email protected] h1:eP1aj1AvT6ynElQH6KP0mmOT2gnWa1gYclHL4wGUbMo=
    github.com/BurntSushi/[email protected] h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
    github.com/sergi/[email protected] h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
    golang.org/x/[email protected] h1:WG0RUwxtNT4qqaXX3DPA8zHFNm/D9xaBpxzHt1WcA/E=
    golang.org/x/[email protected] h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=
    golang.org/x/[email protected] h1:D2X+P0Z6ychko7xn2jvd38yxQfdU0eksO4AHfd8AWFI=
    golang.org/x/[email protected] h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA=
    honnef.co/go/[email protected] h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=
    mvdan.cc/xurls/[email protected] h1:KaMb5GLhlcSX+e+qhbRJODnUUBvlw01jt4yrjFIHAuA=

[glepnir]

I can reproduce use this min config

(setq
      straight-cache-autoloads nil
      straight-check-for-modifications nil
      straight-enable-package-integration nil
      straight-vc-git-default-clone-depth 1
      autoload-compute-prefixes nil)
;; Install and load straight.el
;; https://github.com/raxod502/straight.el#getting-started
(defvar bootstrap-version)
(let ((bootstrap-file
       (expand-file-name "straight/repos/straight.el/bootstrap.el"
                         user-emacs-directory))
      (bootstrap-version 5))
  (unless (file-exists-p bootstrap-file)
    (with-current-buffer
        (url-retrieve-synchronously
         "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
         'silent 'inhibit-cookies)
      (goto-char (point-max))
      (eval-print-last-sexp)))
  ;; (benchmark 1 `(load ,bootstrap-file nil 'nomessage))
  (load bootstrap-file nil 'nomessage))

(straight-use-package 'use-package)

(require 'use-package)

(setq use-package-always-defer t)

(straight-use-package 'company)
(use-package company)

(use-package company
  :init
  (setq completion-styles `(basic partial-completion emacs22 initials
                                ,(if (version<= emacs-version "27.0") 'helm-flex 'flex)))
  (setq company-tooltip-align-annotations t
        company-tooltip-limit 14
        company-idle-delay 0
        company-echo-delay (if (display-graphic-p) nil 0)
        company-minimum-prefix-length 2
        company-require-match 'never
        company-dabbrev-ignore-case nil
        company-dabbrev-downcase nil
        company-global-modes '(not erc-mode message-mode help-mode gud-mode eshell-mode shell-mode)
        company-backends '(company-capf)
        company-frontends '(company-pseudo-tooltip-frontend
                            company-echo-metadata-frontend))
  :config
  (global-company-mode +1))

(straight-use-package 'lsp-mode)
(use-package lsp-mode
  :commands (lsp-install-server lsp lsp-deferred)
  :hook (prog-mode . (lambda ()
                          (unless (derived-mode-p 'emacs-lisp-mode 'lisp-mode)
                            (lsp-deferred))))
  :init
  (setq
        lsp-prefer-flymake nil
        lsp-prefer-capf  t
        lsp-log-io  t
        lsp-keep-workspace-alive nil)
         ;; For `lsp-clients'
 :config
 (when lsp-auto-configure
    (mapc (lambda (package) (require package nil t))
          lsp-client-packages)))

(straight-use-package 'go-mode)
(add-hook 'go-mode #'lsp-deferred)
(straight-use-package 'exec-path-from-shell)

(when (memq window-system '(mac ns x))
  (exec-path-from-shell-initialize))

[glepnir]

i found this

2020/02/24 23:59:01 Build info
----------
golang.org/x/tools/gopls v0.3.2
    golang.org/x/tools/[email protected] h1:eP1aj1AvT6ynElQH6KP0mmOT2gnWa1gYclHL4wGUbMo=
    github.com/BurntSushi/[email protected] h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
    github.com/sergi/[email protected] h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
    golang.org/x/[email protected] h1:WG0RUwxtNT4qqaXX3DPA8zHFNm/D9xaBpxzHt1WcA/E=
    golang.org/x/[email protected] h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=
    golang.org/x/[email protected] h1:D2X+P0Z6ychko7xn2jvd38yxQfdU0eksO4AHfd8AWFI=
    golang.org/x/[email protected] h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA=
    honnef.co/go/[email protected] h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=
    mvdan.cc/xurls/[email protected] h1:KaMb5GLhlcSX+e+qhbRJODnUUBvlw01jt4yrjFIHAuA=

[sebastiansturm]

at the risk of muddying the waters, I've seen the following issue today (may not be related to the issue described above, but at least it's related to the capf backend):
when trying to complete the following line in a TypeScript buffer,

let some_set : Set<string> = new Set();
some_set.|

the language server decides I might be interested in a completion item with "filterText": ".Symbol":

[Trace - 06:19:08 PM] Received response 'textDocument/completion - (41)' in 7ms.
[
  {
    "insertTextFormat": 2,
    "data": {
      "entryNames": [
        "add"
      ],
      <offset/line/file removed>
    },
    "commitCharacters": [
      ".",
      ",",
      "("
    ],
    "sortText": "0",
    "kind": 2,
    "label": "add"
  },
  ... some more items of interest here ...
  {
    "textEdit": {
      "newText": "[Symbol]",
      "range": {
        ...
      }
    },
    "filterText": ".Symbol", <-- entirely uninteresting to me, but unfortunately starts with '.'
    "data": {
      "entryNames": [
        "Symbol"
      ],
      ...
    },
    "commitCharacters": [
      ".",
      ",",
      "("
    ],
    "sortText": "0",
    "kind": 6,
    "label": "Symbol"
  },
  ... more interesting items ...
]

Since lsp-completion-at-point thinks (quite reasonably, IMO) that its current prefix is '.', it disregards all the interesting items and offers me a single-item menu with [Symbol] as the only candidate to choose. Not sure what's the right way to fix this

[sebastiansturm]

should trigger characters generally be removed from prefix strings?

[kiennq]

Since lsp-completion-at-point thinks (quite reasonably, IMO) that its current prefix is '.'

lsp-completion-at-point will try to find the prefix based on response from language server.
Specifically, it will look at items[0].textEdit.range.start or (car (bounds-of-thing-at-point 'symbol)) or (point) as prefix start position.

Can you try this

(advice-add #'completion-all-completions :around
            (lambda (orig string table pred point &optional metadata)
              (let* ((result (funcall orig string table pred point metadata))
                     (last (last result))
                     (base-size (if (consp last) (cdr last))))
                (if (consp last) (setcdr last nil))
                (message "completion-all-completions string=%s table-len=%s length=%s"
                         string
                         (if (and (not (functionp table)) (listp table)) (length table))
                         (length result))
                (append result base-size)))
            '((name . --a1)))

Repro the problem and see what did message spill?

[kiennq]

@taigacute I think I found the problem why you don't get any completions.
Can you try to type fmt.P instead of fmt.p, and see if you get any completion?
The filtering is done case-sensitive depends on completion-ignore-case.

[sebastiansturm]

right after the some_set.|, I'm always getting those two lines of output:

completion-all-completions string=. table-len=10 length=1
completion-all-completions string= table-len=nil length=1

[yyoncho]

@kiennq is this caused by the double filtering? If yes, we may try to fix it the way I described in the PR - by declaring custom category and have it bound only while we are filtering.

[kiennq]

@yyoncho No, it's not due to double filtering. The second filtering in the double filtering is basically noop if completion-styles include basic.
The problem here is that an completion item returned from language server contains a different prefix than the rest, its prefix start point (decided by textEdit.range.start) is one character before others. I'm thinking of having to go through all completion items and heuristically getting prefix start point as most right side prefix start point of all items.
With that we can still get other completion items after filtering

[yyoncho]

@kiennq wouldn't that mean that we might lose the first item?

I am looking at the eclipse lsp implementation:

https://git.eclipse.org/c/lsp4e/lsp4e.git/tree/org.eclipse.lsp4e/src/org/eclipse/lsp4e/operations/completion/LSIncompleteCompletionProposal.java#n378

It seems like the completion prefix per item and the filtering is performed per item. If we do what you are suggesting we will lose the items with most left prefix start point when we are doing basic filtering.

We may also consider fixing this upstream.

[s-kostyaev]

With tips from this thread capf works better, but still there is some issues. For example with this simple file:

package main

import (
    "fmt"
)

func main() {
    fmt.Println("Results:")
}

func DoSome(a interface{}, b interface{}) {}

snippet expansion works well with company-lsp but not with capf:

[glepnir]

@kiennq yes . P is worked. use this solved

(setq completion-styles `(basic partial-completion emacs22 initials
                                ,(if (version<= emacs-version "27.0") 'helm-flex 'flex))
      completion-ignore-case t)

by the way i can get snippet..

the filter doesnt work on gopls 0.3.2. What do we need to do to make the filter work for gopls

[kiennq]

the filter doesnt work on gopls 0.3.2. What do we need to do to make the filter work for gopls

I think you should create an issue upstream in gopls repo, even if we tweak for the filter to work, when you insert, you will get text being inserted doubly.

The problem is that in the mentioned case, gopls returns current cursor position as textEdit range.
So, for example if you type int| (| is cursor position), and the filter shows [int int8 int16 int32] as completion results. Now if you select int16, what will be inserted will be intint16, since basically that's what gopls told LSP client to do

[glepnir]

but it works fine on vim (coc.nvim).I don't know if there is a good way in emacs

[yyoncho]

but it works fine on vim (coc.nvim).I don't know if there is a good way in emacs

The fact that it works somewhere else does not mean that the server is implemented correctly.

Is 0.3.2 the latest version? I am testing with master and it works fine. Can you provide a test file which does not work with gopls master?

[glepnir]

@yyoncho hmm master gopls ? mean you get gopls bygo get golang.org/x/tools/gopls@master? a simple main.go .

[yyoncho]

Yes. When I start typing outside of the main method it works fine.

[glepnir]

doesnt work for me . i tried the gopls@master and gopls@latest ..

[yyoncho]

Please include the test file, not screenshots hiding the content of the file.

[glepnir]

sry . a simple file, a go.mod project

package main

import "fmt"

func main() {
    fmt.Println("test")
}
//method here

[yyoncho]

@muirdm @stamblerre willing to take a look? I looks like the gopls breaks when the completion is invoked outside of the method like that:

package main

import "fmt"

func main() {
    fmt.Println("test")
}
main|

(cursor at | ) - it seems like gopls is disregarding the prefix and it is returning everything that is in the global scope with insert textEdit with no replace range, like this:

{
      "command": {
        "command": "",
        "title": ""
      },
      "textEdit": {
        "newText": "fmt",
        "range": {
          "end": {
            "character": 0,
            "line": 16
          },
          "start": {
            "character": 0,
            "line": 16
          }
        }
      },
      "insertTextFormat": 2,
      "filterText": "fmt",
      "sortText": "00000",
      "preselect": true,
      "detail": "\"fmt\"",
      "kind": 9,
      "label": "fmt"
    }

[muirdm]

Completion at file level should be fixed on gopls@master. The only completions offered now are the keywords "const", "func", "import", "type", and "var". Writing arbitrary code at the file level is a syntax error so completion is not supported.

As for completion-at-point, gopls by default performs server side fuzzy matching and always returns completion results with "isIncomplete: true", intending to disable client-side filtering. gopls offers up to 3 "deep completion" candidates which we want to update as you continue to type, so client side filtering is not possible. gopls also gives all candidates an identical "filterText" as a workaround to prevent VSCode from re-ordering the candidates.

An example of deep completion:

package main
import "context"
func main() {
    var _ context.Context = cb // completes straight to "context.Background()"
}

company-capf doesn't seem to work in above case.

If you want to use company-capf with only client side filtering, you should set "gopls.matcher" to "default", which will disable server side fuzzy matching, causing gopls to return full results. Note that deep completion candidates won't be able to update as you type since the filtering is client side, so disabling "gopls.deepCompletion" also probably makes sense. However, I feel the default gopls config paired with company mode gives the best completion experience ATM.

(lsp-register-custom-settings
 '(("gopls.matcher" "default")
   ("gopls.deepCompletion" nil t)))

[yyoncho]

Thank you @muirdm .

Completion at file level should be fixed on gopls@master.

AFAICS this is not the case(although I might be wrong).

As a side note company-capf works fine if you switch to flex matching:

@kiennq - I think that @muirdm's suggestion to do not filter client-side makes sense when server has returned isIncomplete = t. If the server has returned partial result this automatically means that we should not filter client-side because the server has already filtered the data. WDYT?

[kiennq]

If the server has returned partial result this automatically means that we should not filter client-side because the server has already filtered the data. WDYT?

It makes sense to do that.

[glepnir]

@kiennq Update the lsp-mode .still not work.

[yyoncho]

@taigacute refer to @muirdm's comment about completion on top level. ATM the original issue report is solved from lsp-mode point of view. The only outstanding work is on @sebastiansturm report about typescript language server.

[glepnir]

@yyoncho aha ok.

[yyoncho]

There is one more issue when completing imports: see https://github.com/emacs-lsp/lsp-mode/issues/1459

When the server no filter text, no textEdit, no insertText we should not filter those items.

  {
    "data": {
      "entryNames": [
        "@angular-devkit/build-angular"
      ],
      "offset": 43,
      "line": 2,
      "file": "/home/kyoncho/Sources/my-first-app/src/main.ts"
    },
    "sortText": "0",
    "kind": 9,
    "label": "@angular-devkit/build-angular"
  },

Also, it seems like it will behave better if we use the algorithm from https://git.eclipse.org/c/lsp4e/lsp4e.git/tree/org.eclipse.lsp4e/src/org/eclipse/lsp4e/operations/completion/LSIncompleteCompletionProposal.java#n378 instead calling back to thing-at-point for the cases when we do not have textEdit and when we have to guess the prefix.

The alternative is to try to fix typescript server but I believe it won't be easy since the data comes from the tsserver.

[s-kostyaev]

There is another problem: if completion triggered during snippet expansion of other completion, snippet expansion will be stopped, so I can't expand snippet of first completion:

• with capf:
  
• with company-lsp:
  

[glepnir]

@s-kostyaev cant reproduce. it works fine on my emacs.

[rrudakov]

@s-kostyaev try (setq yas-inhibit-overlay-modification-protection t)

[glepnir]

maybe dont set (setq yas-inhibit-overlay-modification-protection t)

[s-kostyaev]

@rrudakov, @taigacute thanks! (setq yas-inhibit-overlay-modification-protection t) fixed it for me.

I think we should collect tips from this topic to default configuration and/or docs.

[glepnir]

It ’s weird. I didn’t set it, but it still works.

[yyoncho]

@taigacute based on the screenshots you are using company-box which does not use overlays but posframe - thus there is no problem.

[glepnir]

@yyoncho thanks for your reply. got it.

[kiennq]

When the server no filter text, no textEdit, no insertText we should not filter those items.

Indeed, The algorithm you said may provide a better heuristic to determine the prefix when there's no textEdit.
JFYI, I'm working on a change to divide the completion items into groups based on prefix offset position and later combine all filtered items into a final list. Just that we need a ranking algorithm. For now I will rely on emacs built-in one and heuristically combine the first item of each group first then the rest, going from the left most prefix offset group.

And thought on how we should rank the filtered candidate is welcome.

[sebastiansturm]

another data point, though not very helpful yet: when editing C++ buffers with the capf backend, I sometimes get the impression that lsp-mode stops requesting completions. Just happened again so I could enable lsp-log-io and indeed no requests were being set out. One call to (lsp--capf-clear-cache) resolved the issue (though without that call, no candidates were shown at all so apparently the cache either contained no matches or wasn't used properly). I'll try to find a reproducible way to trigger this issue or, failling that, will have a look at the current capf implementation and see if I can spot anything

[kiennq]

another data point, though not very helpful yet: when editing C++ buffers with the capf backend, I sometimes get the impression that lsp-mode stops requesting completions. Just happened again so I could enable lsp-log-io and indeed no requests were being set out. One call to (lsp--capf-clear-cache) resolved the issue (though without that call, no candidates were shown at all so apparently the cache either contained no matches or wasn't used properly). I'll try to find a reproducible way to trigger this issue or, failling that, will have a look at the current capf implementation and see if I can spot anything

The lsp-mode can stop requesting completion when the server said that it has return all the completion item.
However, there's may be cases that the lsp--capf-cache is not cleared on needed, for now it's automatically cleared when company selection is aborted or completed. Let me know if you think there should be other case it should be cleared

[yyoncho]

JFYI, I'm working on a change to divide the completion items into groups based on prefix offset position and later combine all filtered items into a final list. Just that we need a ranking algorithm. For now I will rely on emacs built-in one and heuristically combine the first item of each group first then the rest, going from the left most prefix offset group.

I was thinking about the sample implementation but I wonder whether we should follow what the others do. I know that the other option is to structure the code in a way that we reuse as much as possible from the fast C implementation but seems like we will have the problems with merging.

WDYT about:

1. Request a function from the core to provide the match weight so we could use it independently from current completion related functionality.
2. Until we have that implement a match function on our side.
3. Also, we could create a dynamic module written in C/Rust to have that function for older versions of emacs.

This will allow us to do proper filtering and sorting. For example, if two completion items have the same score we should sort by sortText. ATM this does not happen since the completion in emacs does not have the notion of sortText.

PS: we should investigate if 2) is sufficient to cover our performance needs.

[s-kostyaev]

we could create a dynamic module written in C/Rust to have that function for older versions of emacs

like this one: https://github.com/rustify-emacs/fuz.el ?

[yyoncho]

@s-kostyaev yes. I think that there is even a Haskell version.

[s-kostyaev]

Request a function from the core to provide the match weight so we could use it independently from current completion related functionality.

For now (in emacs 27) score for icomplete calculates in completion-pcm--hilit-commonality function. It's written on elisp. For flex completion it use pretransformation of matching pattern by completion-flex--make-flex-pattern

[kiennq]

• Request a function from the core to provide the match weight so we could use it independently from current completion related functionality.
• Until we have that implement a match function on our side.

When I saw the results from flex, there's some kind of match scores inside them.
Note that the flex score is also calculated using elisp, not C so it would be the same if we provide our own match function.

3. Also, we could create a dynamic module written in C/Rust to have that function for older versions of emacs.

This would be nice, we should considering providing a release binary for the load module instead of asking user to built it themselves too.
Just that since we're using different prefix (pattern) to compare against different insert string, I'm not sure it's easy to just compare the score in that case. We still have to use some combination algorithm in our end to make the score comparable.

[yyoncho]

Request a function from the core to provide the match weight so we could use it independently from current completion related functionality.

For now (in emacs 27) score for icomplete calculates in completion-pcm--hilit-commonality function. It's written on elisp. For flex completion it use pretransformation of matching pattern by completion-flex--make-flex-pattern

And it seems to be dead slow - https://github.com/emacs-lsp/lsp-mode/pull/1336#issuecomment-576580140

[s-kostyaev]

thank you for explanation.

[yyoncho]

When I saw the results from flex, there's some kind of match scores inside them.
Note that the flex score is also calculated using elisp, not C so it would be the same if we provide our own match function.

If we can pass custom match function I guess we do not need to split and then merge?

[kiennq]

Also, it seems like it will behave better if we use the algorithm from https://git.eclipse.org/c/lsp4e/lsp4e.git/tree/org.eclipse.lsp4e/src/org/eclipse/lsp4e/operations/completion/LSIncompleteCompletionProposal.java#n378 instead calling back to thing-at-point for the cases when we do not have textEdit and when we have to guess the prefix.

@yyoncho I just notice this may not work with flex matching, since the prefix will be different. We can still try to use fuzzy match to guess prefix, but I doubt we can get a good perf from it

[yyoncho]

@yyoncho I just notice this may not work with flex matching, since the prefix will be different. We can still try to use fuzzy match to guess prefix, but I doubt we can get a good perf from it

Can you give an example? Also, can we do prefix matching against different prefixes?

[kiennq]

@yyoncho I just notice this may not work with flex matching, since the prefix will be different. We can still try to use fuzzy match to guess prefix, but I doubt we can get a good perf from it

Can you give an example? Also, can we do prefix matching against different prefixes?

For example, user can type "cb" to have it match to "context.Background", with the algorithm described in LSIncompleteCompletionProposal (if I read it correctly), it will fail to find "cb" as prefix for "context.Background" since it doesn't match as exact prefix.

[yyoncho]

For example, user can type "cb" to have it match to "context.Background", with the algorithm described

That is true - but for the cb case we have textEdit. If we implement the logic from lsp4e the result will be that we won't support fuzzy when we do not have textEdit which IMO is fine. Still, we will have to experiment to find out what is the best way - I am open for suggestions

[sebastiansturm]

another data point, though not very helpful yet: when editing C++ buffers with the capf backend, I sometimes get the impression that lsp-mode stops requesting completions. Just happened again so I could enable lsp-log-io and indeed no requests were being set out. One call to (lsp--capf-clear-cache) resolved the issue (though without that call, no candidates were shown at all so apparently the cache either contained no matches or wasn't used properly). I'll try to find a reproducible way to trigger this issue or, failling that, will have a look at the current capf implementation and see if I can spot anything

The lsp-mode can stop requesting completion when the server said that it has return all the completion item.
However, there's may be cases that the lsp--capf-cache is not cleared on needed, for now it's automatically cleared when company selection is aborted or completed. Let me know if you think there should be other case it should be cleared

I'm not sure yet. I just tried to reproduce the issue and could do so two times by randomly jumping around in the buffer and requesting completions, but I didn't yet find a sequence of actions that works every time. When I saw the issue, the last completion returned by clangd always hat isIncomplete set to true, but there's also plenty of times when incomplete completion responses cause no issue at all.

[yyoncho]

@sebastiansturm if we suspect stale cache we might add lsp--capf-clear-cache to company-completion-started-hook as well.

[sebastiansturm]

thanks, I can try and see if that helps but I guess I should first try to figure out why it doesn't work properly as is. Since no one else has reported this issue so far, maybe it's something specific to my setup

[yyoncho]

it doesn't work properly as is. Since no one else has reported this issue so far, maybe it's something specific to my setup

My guess is that this may happen if for some reason the finish/cancelled hooks are not called. I think I have seen the same in the past but I also don't a reproducer.

[sebastiansturm]

thanks for the pointer!

[sebastiansturm]

are we sure that company-prefix is always non-nil when we want the cache to be cleared? It seems to me (see company-continue) that company-prefix might not get set if there's a unique completion candidate, in which case none of the hooks would be executed?
Also, do we need the (setq-local lsp-inhibit-lsp-hooks nil) to precede the (company-call-backend 'post-completion result) within company-cancel? Because otherwise we could replace the two hooks by one bound to company-after-completion-hook I guess?

[yyoncho]

are we sure that company-prefix is always non-nil when we want the cache to be cleared? It seems to me (see company-continue) that company-prefix might not get set if there's a unique completion candidate, in which case none of the hooks would be executed?

Can you elaborate? I fail to see the connection to company-prefix - it seems to me that we do not use this variable.

Also, do we need the (setq-local lsp-inhibit-lsp-hooks nil) to precede the (company-call-backend 'post-completion result) within company-cancel? Because otherwise we could replace the two hooks by one bound to company-after-completion-hook I guess?

We have to inhibit right after the completion has started - this will prevent the calls to highlights/lenses/links, etc while completion is active. Otherwise, the completion might slow down and be less responsive.

[sebastiansturm]

are we sure that company-prefix is always non-nil when we want the cache to be cleared? It seems to me (see company-continue) that company-prefix might not get set if there's a unique completion candidate, in which case none of the hooks would be executed?

Can you elaborate? I fail to see the connection to company-prefix - it seems to me that we do not use this variable.

as far as I can see, company-completion-finished-hook and company-completion-cancelled-hook will only be called (within company-cancel) if company-prefix is non-nil

Also, do we need the (setq-local lsp-inhibit-lsp-hooks nil) to precede the (company-call-backend 'post-completion result) within company-cancel? Because otherwise we could replace the two hooks by one bound to company-after-completion-hook I guess?

We have to inhibit right after the completion has started - this will prevent the calls to highlights/lenses/links, etc while completion is active. Otherwise, the completion might slow down and be less responsive.

yes, I'm talking about the reactivation once completion is done. Seems to me that instead of adding to two hooks company-completion-finished-hook and company-completion-cancelled-hook, we might just as well use company-after-completion-hook. Unless we want the reactivation to run before company-cancel calls (company-call-backend 'post-completion result) that is

[yyoncho]

@sebastiansturm AFAICS the company-after-completion-hook is also called only if company-prefix is non nil?

@dgutov - we want a code to run a piece of code on our side when the completion in company mode has started and pair that with a cleanup function after the completion has finished(no matter if it is successful or not). ATM we are using company-completion-cancelled-hook and company-completion-finished-hook but it seems like they are called only if there is company-prefix. My question is is it possible begin hook to be executed without corresponding cancelled/finished hook calls? And if yes, can we either change after-completion to be called unconditionally (even if company-prefix is nil) or introduce a new hook for that purpose?

[sebastiansturm]

@sebastiansturm AFAICS the company-after-completion-hook is also called only if company-prefix is non nil?

yes, company-after-completion-hook would be a more convenient place to clear lsp-inhibit-lsp-hooks (and being restricted to non-nil prefixes is probably ok because company-completion-started-hook is called in a place where company-prefix should be non-nil), but I think it won't help with capf

[sebastiansturm]

for now I guess I could remove (lsp--clear-capf-cache) from both hooks and instead add it as advice to company-cancel, just to check whether I still keep running into the stale-cache issue

[kiennq]

The issue of variable prefix length will be fixed with #1466

Also, just a tip, using company-statistics is quite a good way to have some most used completion candidates to be placed on top of company-mode popup.
Here is my configure for that so we can still preserver the order of candidates returned from language server while still have the most used candidates on top

(use-package company-statistics
  :ensure t
  :after company
  :diminish
  :config
  (company-statistics-mode)
  (advice-add #'company-sort-by-statistics :override
              (lambda (candidates)
                (if (> (length candidates) 2)
                    (let* ((max-by-score
                            (lambda (list)
                              (--reduce (if (> (funcall company-statistics-score-calc it)
                                               (funcall company-statistics-score-calc acc))
                                            it acc)
                                        list)))
                           (max-cand (funcall max-by-score candidates))
                           (candidates (delq max-cand candidates))
                           (max-cand-2 (funcall max-by-score candidates))
                           (candidates (delq max-cand-2 candidates)))
                      (-concat `(,max-cand ,max-cand-2) candidates))
                  candidates))
              '((name . --boost-2))))

[dgutov]

@yyoncho The -started-hook is run in company--begin-new, and only in the case when completion is non-unique, meaning the process isn't going to conclude right away, and a UI will be shown.

So it's not called in other cases. Would it help if it was also called in the company--unique-match-p case (but not when prefix is nil)?

Could you describe your use case in more detail?

[yyoncho]

@dgutov

Could you describe your use case in more detail?

We want to inhibit a certain lsp-mode functionality while the company is active to make lsp-mode more responsive while completing. So we are looking for begin/end events. I think that you answered my question. If the start/end hooks are not called only when there is a single candidate, we are fine since we won't need to suppress anything because the candidate will be inserted right away.

[yyoncho]

We are done with that - thanks to @kiennq for the hard work! If there is something missing please open a new bug following the bug template.

[glepnir]

thansk for great work. it works fine.