Lsp-mode: Performance issue when server registers many watchers

[michaelpj]

Describe the bug
In https://github.com/haskell/ghcide/issues/776 we discovered that some versions of ghcide (a Haskell LSP implementation) create a large number of file watchers: one for each file it wants to watch, which can be thousands (I observed ~3k).

This seems to cause performance issues in lsp-mode: we observed emacs slowdowns or hangs when

1. Adding all the watchers (typically when the session starts up).
2. Dispatching events to workspaces.

Now, certainly ghcide should be smarter and not submit so many small watchers (and apparently new versions produce many fewer watchers). Still, it would be nice if lsp-mode didn't choke either.

@mrBliss observed that 2 might be caused by the way lsp-mode scans over the watchers when processing events. I'm not sure if this is the same or a related problem that causes 1.

Expected behavior
lsp-mode should never hang emacs.

Which Language Server did you use
haskell-language-server 0.3.0.0 (which uses ghcide under the hood).

OS
Linux (NixOS).

Error callstack
I tried to get a callstack during the hang, but I couldn't get anything that wasn't emacs display functions. So possibly it's happening in IO.

lsp-log shows nothing out of the ordinary, and the server log just shows 3k entries like this:

[Trace - 06:06:22 PM] Received request 'client/registerCapability - (3305).
Params: {
  "registrations": [
    {
      "id": "/home/michael/projects/iohk/plutus/dist-newstyle/build/x86_64-linux/ghc-8.8.3/plutus-core-0.1.0.0/build/autogen/Control/Arrow.hs",
      "method": "workspace/didChangeWatchedFiles",
      "registerOptions": {
        "watchers": [
          {
            "globPattern": "/home/michael/projects/iohk/plutus/dist-newstyle/build/x86_64-linux/ghc-8.8.3/plutus-core-0.1.0.0/build/autogen/Control/Arrow.hs",
            "kind": 5
          }
        ]
      }
    }
  ]
}

[yyoncho]

No matter how many watches ghcide is sending, lsp-mode will watch the whole directory. In general, we have a solution for both issues - using Emacs threads. It was discussed in the past, just someone has to implement it.

[michaelpj]

It does seem that the slowdown comes from processing all the messages, though, not the file watching. Tailing the the server log, I can see it sending requests in bursts, after which emacs slows to a crawl for a bit.

[yyoncho]

Yes. The plan is you do that in bkg thread and yield to allow the main thread to process the user actions.

[leungbk]

No matter how many watches ghcide is sending, lsp-mode will watch the whole directory. In general, we have a solution for both issues - using Emacs threads. It was discussed in the past, just someone has to implement it.

@skeeto mentioned in a blog post written when Emacs 26.1 came out that Emacs' threading implementation was not reliable. Have things improved since then?