Selectrum: Tramp vs Selectrum 3.1 - /ssh: and hang, opening tonnes of IPs instead of listing them?

[skeezix]

Firszt - Selectrum 3.1 is _amazing_; prior versions were wonderful, but this latest update a few days ago really improved many things, but notably performance; M-x was taking a couple seconds to open before, now nearly instantly (and thats with the extra help annotations added by other packages.)

Problem - Just doing open file (C-x C-f) and entering /ssh: (goal is to put in somethign like /ssh:foo@hostname... for Tramp obviously), and as soon as the : is listed Emacs essentially hangs for quiter awhile; C-g to break out sometimes works, and I end up spamming it for awhile.

Enabling tramp verbose logging suggests that selectrum is returning a pile of stuff there (?), and Tramp or selecturm is trying to open everything; like, it seems Tramp is being tried against all IPs I've opened redcently (perhaps Selectrum or Tramp is reviewing recentf's cache? or bookmarks? not sure where its getting the list, its quite a pile of IPs its finding, but they're all valid ones I've used recently.)

Not sure if this is a Tramp or Selectrum issue, as both have had updates recently.

[clemera]

Thank, I also just also discovered that Tramp is slow again, too. It doesn't seem to happen with 3.1 release candidate or does it? Are you on current master?

[clemera]

Should be fixed by 46f79d17f0f5fe6d66f1880c09efa1da1c4803e4?

[clemera]

@minad We should work out something else but to I just reverted for now as I'm short on time.

[minad]

Ah I see. It seems the optimization is useful because the following cl-position call uses selectrum--get-full which is too slow. Does it make sense to use selectrum--get-full? Is this really necessary? Is it possible to optimize this function? The problem I see here is that the shortcut works nicely, but we could get a slowdown again if move-default=nil. Maybe a better fix would check first if the default candidate is non-nil and only search via cl-position in that case? Secondly I think selectrum--get-full must be optimized to ensure that slowdown never happens!

So in other words - your fix is incomplete and the slowdown issue will bite us again in other scenarios since the fix does not to resolve the root cause.

[clemera]

I also think that selectrum--get-full should get optimized. Another thing I noticed is that for file completions we should generally only check for the default when the prompt equals the initial prompt.

[clemera]

Just realized that getting the position like we currently do for file completions is broken anyway, I will fix that later. Regarding the shortcut I think we can leave it there as this is how the UI should work currently, if we change that we can look at it again.

[minad]

Does it make sense to change the short cut like this?

~
(or (not selectrum--default-candidate) selectrum-move-default-candidate)
~

[clemera]

Also should the selection of the default only happen when selectrum--is-initializing is non-nil?

[minad]

I don't know. You tell me!

I would appreciate if you add a comment after you decided on how the logic should be.

[clemera]

I think it makes more sense because the selection is only a help to be able to press RET at the initial prompt. Trying to keep it selected after you provide input doesn't feel right to me.

[minad]

Yes, but if the case does not match it will fall through to the default with the cl-position and you may end up with 0 again? Please carefully check each case and add comments why the cases to compute the index are how they are.

[clemera]

See 12c241a84898b7d3b5b3e260672d43475f00d221

[clemera]

For explaining comments see a17f3c9f4ed14c3d2e3b5ef300a14a39f49f14b1. @skeezix I hope your issue is fixed?

[clemera]

@minad On another note I noticed there are slowdown issues when using marginalia with tramp even though no annotations are shown (using marginalia-annotators-heavy).

[minad]

Yes, that could happen when the file annotator accesses the slow tramp fs. The file annotator should maybe filter tramp. Or did you also observe it for other annotators?

[clemera]

Only for file completion, skipping for tramp might be good, the annotations also aren't shown is that a Selectrum issue or are they also not shown with default completion?

[minad]

I don't know. Just try it :smile:

[clemera]

Right, I shouldn't be so lazy :laughing: I just tested and they aren't shown there, too.

[minad]

I cannot confirm this. For me the file annotator works with tramp. And it does not feel particularly slow.

[clemera]

Oh, okay I will try with emacs -Q.

[clemera]

Hm, still slow for me, I notice it mostly when scrolling (keeping "C-n" pressed) it starts to stutter in this case for me.

[minad]

Yes, it is not super fast, but that is expected. But even buffer switching and other things become slower. Over a slow connection Emacs with tramp is utterly unusable. I never use tramp with slow connections. It works well in a local network, but that's about it. I somehow think tramp is a misguided idea :shrug:

[clemera]

The stuttering case I described was with sudo, I don't use the heavy annotations so I'm not affected usually.

[minad]

Okay, it is probably simply slow due to the overhead of tramp+sudo. We could filter it, but that's about all we can do. If someone interested creates a PR I would merge it.

[clemera]

Okay, I wasn't sure what the state with tramp and marginalia is, I searched the issue list and did not found anything, hence I asked you here. I guess if no one complaints it is fine :+1:

[skeezix]

Regarding https://github.com/raxod502/selectrum/commit/a17f3c9f4ed14c3d2e3b5ef300a14a39f49f14b1 I'll try it tonight after chores are done, thanks!

I haven't looked over the selectrum code nor do I understand the completion mechanism, but I am curious as to what was causing the slo down; from the Emacs messages and tramp log, it sure looked like it was actually trying to Tramp _open_ all the IPs I'd recently visited; so questions are.. i) how does Selectrum figure out what IPs to be listing there? (thats such a neat idea, btw, hadn't conceived of that as something Selectrum could do at all :) .. and 2) why was Tramp seemingly trying to actually connect to/open all of them?

I live in Tramp mode (company internal network, from home vpn'd to the network of course due to covid), and Selectrum is one of my favorite things (esp with heavby marginalia, oh baby!) .. but just having Selectrum matching the text against recentf and bookmarks (say) would be a treat.

ie: The fact Emacs was hanging at the : part .. /ssh:,m hanging and generating lists of IPs for the selection list and trying to connect to them, blew my mind; I expect a slow down _once connected_, like trying to get the directory listing etc. (And recent Selectrum 3.1 made that _enormously_ faster, it was unusable with Selectrum previously, and yet now Selectrum + Tramp is fantastic); but this was not even at the point it shoudl be lookign for dir list or connecting, this is just trying to put in the username and hostname part of the file name, wherein Tramp shoudl reach out.

Anyway, if you guys have fixed it, thats super awesome, and I'll try it out soon.

The fact you guys pounced on this is remarkable! If theres a tip jar somewhere, let me know :)

[clemera]

Thanks, it is nice to hear that you like it! Regarding your question: In file completions when you insert a candidate with TAB or select it with RET we use substitute-in-file-name and completion--sifn-requote (see selectrum--get-full) to get the candidate path which is composed of some part of the minibuffer contents and the selected candidate. The problem was that this path computation was accidentally done on each completion candidate (see here). In your case the path computation on each candidate might have caused tramp to open connections for each of them (I'm actually not sure but I hope this was the case otherwise you might have hit another bug we aren't aware of, yet).

[skeezix]

OKay, in a spare moment I just curl pulled selectrum.el over top of my current one (I'm using melpa 3.1, and like to stay on melpa for the easy upgrades); restarted Emacs to be safe, and still the problem.

C-x C-f /ssh:
^ as soon as the : is hit, Emacs is 'hung'

If I hit C-g a few times Selectrum will list:
ssh:foo@bar
ssh:10.1.2.3
ssh:10.1.2.4
...etc (various valid looking targets I've been to)

.. but contionues being 'hung'

Hit C-g spam a few more times to get right out and its okay back to buffer to work.

I effectively can't C-x C-f and put in /ssh: anymore, to even get to the part of entering /ssh:foo@somewhere .. soon as the colon is entered, it goes into Tramp-hell.

So this is at : time; is that Tramp grabbing that event and going off to check if ssh is present or something? is it Selectrum waking up? the two fighting?

But at a random guess, is /ssh: waking up Tramp or Selectrum, and Selectrum producing a list of IPs or something from , and then Tramp tries ot open them all, rather than just listing something?

As a workaround, I've been adding servers to bookmarks (manually editing the bookmarks file) and then I can use bookmarks to get dired on a server, open a file, and then good to go.. opening another file, from that first file, Selectrum and Tramp cooperate on the dir listing nice and fast

But that /ssh: bit is brutal :)

[skeezix]

(yes, I removed selectrum.elc when I pulled in the selectrum.el .. which I tried both the master current, and the one specifically mentioned above ...3c9) .. still hang at /ssh:
Mac OS Catalina fwiw .. I shall have to clone my .emacs/.emacs.d onto another box and see as well, if thats helpfull

What kind of debugging aid can I offer?

[clemera]

What Tramp/Emacs version are you using? And can you also post your full minibuffer completion setup and Tramp setup? For debugging you could eval the selectrum.el buffer and do M-x toggle-debug-on-quit, then trigger the bug and hit C-g which should give you backtrace.

[skeezix]

Tramp setup is not much:

(when skeez/feature_docker_tramp
;; MELPA docker-tramp
;;
;; WOA
;; Tramp multihop: C-x C-f /ssh:you@remotehost|sudo:remotehost:/path/to/file RET
;; Tramp multihop through a docker at second stage: /plink:[email protected]|docker:root@ranokia_19.10.0-6b_0:/path/to/file
(use-package docker-tramp
:ensure t)

(push
(cons
"docker"
'((tramp-login-program "sudo docker")
(tramp-login-args (("exec" "-it") ("%h") ("/bin/bash")))
(tramp-remote-shell "/bin/sh")
(tramp-remote-shell-args ("-i") ("-c"))))
tramp-methods)

(defadvice tramp-completion-handle-file-name-all-completions
(around dotemacs-completion-docker activate)
"(tramp-completion-handle-file-name-all-completions \"\" \"/docker:\" returns
a list of active Docker container names, followed by colons."
(if (equal (ad-get-arg 1) "/docker:")
(let* ((dockernames-raw (shell-command-to-string "docker ps | awk '$NF != \"NAMES\" { print $NF \":\" }'"))
(dockernames (cl-remove-if-not
#'(lambda (dockerline) (string-match ":$" dockerline))
(split-string dockernames-raw "\n"))))
(setq ad-return-value dockernames))
ad-do-it))
)

;; try to speed up tramp..
(setq tramp-completion-reread-directory-timeout nil)
;; tramp: ignore version control on remotes
(setq vc-ignore-dir-regexp
(format "\(%s\)\|\(%s\)"
vc-ignore-dir-regexp
tramp-file-name-regexp))

[skeezix]

Selectrum:

(selectrum-mode +1)
(add-hook 'comint-mode-hook
(defun my-comint-use-default-completion ()
(setq-local completion-in-region-function #'completion--in-region)))
;; set selectrum to use orderless
(setq selectrum-refine-candidates-function #'orderless-filter)
(setq selectrum-highlight-candidates-function #'orderless-highlight-matches)

(add-hook 'buffer-list-update-hook
(lambda ()
(unless (eq major-mode 'minibuffer-inactive-mode)
(setq completion-styles (if (derived-mode-p 'comint-mode) '(basic) '(orderless))))))

[skeezix]

Sometimes when spamming C-g to break the hang, it'll show 'Tramp connecting to..." one of the IPs; so the hang is almost certainly somethign Selectrum is doing, that causes Tramp to be trying to open the connection(s)

[jdm204]

@clemera I've had the same issue as @skeezix - I think the issue is selectrum + marginalia's heavy annotations, maybe it tries to retrieve some information about the remotes to display as annotations that is causing this hang.

If I emacs -Q and load just selectrum, or selectrum and marginalia without heavy annotations, it works okay, (setq marginalia-annotators '(marginalia-annotators-heavy nil)) and I get the hang.

[clemera]

@jdm204 Thanks, I opened a PR on the marginalia side here

[clemera]

I will close this as it the remaining problems are a Marginalia issue, see the followup PR here

[skeezix]

Ah, very interesting!

I pulled latest Selectrum and Marginalia manually, and everything is working a treat again, plus the new bits. Whew!

Super bug-tracking, and super support.

Thanks so much guys, and looove Selectrum. Honestly, tinkering with my now-enormous .emacs has helped me stay sane during this last years global apocalypse :)