Helm: Add "real" fuzzy search behavior

[Silex]

Hello,

I think this great plugin would be MUCH more used if there was "ido-like" fuziness searching. I know about the match plugin, but it's not the same (except if I misconfigured it).

Having to type a space between the sub expressions ruins what fuzzy is about.

This fuzzy behavior is present in Textmate, vim's FuzzyFinder, emac's ido and SublimeText2. People DO expect a good plugin to offer this kind of Fuzzyness.

That said, I hope you'll show me that this behavior is already present and that I'm an idiot for not finding it :)

If it's not present, what would it require to implement it? Maybe simply by defering to ido's fuzzy function...

[lewang]

I'd much prefer the current behaviour that ido's "fuzzy", which essentially inserts ".*" between letters. Although Sublime's fuzzy is sublimely good. It's hard to do it that well though. Not sure if Emacs-lisp is fast enough.

Anyway you're not the first one to request this, and @thierryvolpiatto has said he's not interested in developing it.

[Silex]

What would it require to develop it anyway? I'm sure a lot of people would be interested. If it's as simple as adding .* between letters it sounds easy.

[lewang]

As I've said, adding ".*" is not the right way to go. It's okay as a filtering mechanism, but other heuristics are needed to make useful matches sort to the top. This is what Sublime is very good at and what make ido's "fuzzy" worthless.

Maybe @thierryvolpiatto can give you some help on implementing that if that's what you want.

Vim's current state of the art for this is: https://wincent.com/products/command-t

If we can get a fuzzy filtering and sorting that passes most of its test suite:

https://github.com/wincent/Command-T/blob/master/spec/command-t/match_spec.rb

That'd be a good start.

[Silex]

I disagree that ido's fuzzy is worthless. I agree it could be better, but in it's current state it's already very useful. I use it along with textmate-find-file-in-project and it's fine.

What I have a problem with ido is how it presents results. Helm presents results in a much better way (like FuzzyFinder does in vim), but helm's current way of searching stuffs is very annoying for me. Having to press space to separate regexes, then still having to select the first entry requires too much intervention from me IMHO.

So, I'm trying to get the best of both world: a helm presentation with ido's fuzzy search. Right now I'm using a hack that displays ido's results vertically, but it's still a PITA when having to cycle results.

Thanks for digging the tests of Command-T, it'd indeed be a good start.

[thierryvolpiatto]

FYI helm-find-files is already doing this.(more or less)
However, helm-match-plugin is an horror that need to be rewrited, and should have such feature once done to extend this to all helm.

[reactormonk]

I tried implement that, but it's too slow in elisp. Code: http://sprunge.us/GHYE Tests: http://sprunge.us/bUYg Data: http://sprunge.us/NQiY
I tried gisting it, but it kept failing.

[Silex]

Thanks @Tass. I know vim's Command-T uses Ruby because Vimscript is too slow too, maybe we could use a similar idea?

[reactormonk]

The basic matcher uses C, because ruby would be too slow. I already use a bunch of regexp to speed things up, but it's not enough. And FFI in emacs is a politically loaded topic.

[Silex]

Maybe it's possible to leverage org-babel which can run C code? To be honest I don't think FFI being a politically loaded topic is a reason not to do it if we need it, we'd care about features not politics/philosophy imho.

[reactormonk]

I suppose they use subprocesses. But that communication tends to be slow, you'd need to do the whole searching externally, otherwise get a delay while emacs is shoveling data to the subprocess. But that remains to be tested.

Another idea worth testing would be SLIME with an external clisp which should be faster than the elisp.

[lewang]

On Sat, Nov 17, 2012 at 4:07 AM, Simon Hafner [email protected]:

I suppose they use subprocesses. But that communication tends to be slow,
you'd need to do the whole searching externally, otherwise get a delay
while emacs is shoveling data to the subprocess. But that remains to be
tested.

This matching needs to be a part of Emacs.. Features like autocomplete,
dabbrev, etc would also benefit (PS, sublime does "fuzzy completion"
everywhere). I think we need to eloquently state why this will be useful
with lots of examples. This is the hard part. It's very easy for people
who havent' played around with something Sublime text or Textmate to
dismiss this feature just from hearing how it roughly works, they need to
see concrete examples of how intuitive and useful it is.

The rough api should be something like

getBestMatches(collection, count, &optional strategy, syntax-table)

Le

[thierryvolpiatto]

I did some improvements in helm-find-files, please have a try.

[Silex]

@lewang: today I was told that you can already customize the heuristic in ido, see ido-make-buffer-list-hook, ido-make-dir-list-hook or ido-make-file-list-hook.

I didn't play with it but it looks interesting.

[Silex]

By the way, I just found out https://github.com/auto-complete/fuzzy-el/blob/master/fuzzy.el

Is it any good?

[thierryvolpiatto]

It is working but I prefer the actual behavior.
Here a simple example against buffer list:

(defvar helm-c-source-test
  '((name . "test")
    (candidates . (lambda ()
                    (with-helm-window
                      (fuzzy-all-completions
                       helm-pattern (mapcar 'buffer-name (buffer-list))))))
    (match-strict . identity)
    (volatile)))
;; (helm :sources 'helm-c-source-test :buffer "*helm test*")

[thierryvolpiatto]

I am closing this, feel free to continue discussing this on emacs-helm mailing list.

[Silex]

Nice, thanks for the snippet.

[Silex]

Just for curious people, I found this: https://github.com/d11wtq/fiplr/blob/master/fiplr.el
It uses this library for the Fuzzy matching: https://github.com/d11wtq/grizzl

[lewang]

@Silex since you're still on the prowl, I've implemented the kind of smart matching I described earlier here:

https://github.com/lewang/flx

It's only for ido so far. I haven't had the time to integrate into helm yet.

[Silex]

@lewang: nice! I'll give it a try, did you know about grizzl? It looks nice too.

[lewang]

I had a chat with grizzl's author about implementation ideas. Have not actually tried using it yet.

flx is focused on scoring and sorting intelligently. It seems grizzl is more focused on speed.

[jgeerds]

@lewang It is possible that you work together with grizzl's author and perhaps merge these two projects?

Apart from this: Do you find time to implement flx into helm? That would be really great! :-)

Thanks for your amazing work!

[lewang]

I'm not sure what grizzl is doing at all actually. It seems to do the exact same thing as ido+flex but with a slightly different interface. Am I correct @geerds ?

Anyway I have not had time to work on flx Helm integration for a while. Don't think that's going to happen in the next few months. Sorry.

[egbulmer]

Just curious, have there been any updates on this style of behaviour since the conversation 7 months ago?

[thierryvolpiatto]

Elliot Bulmer [email protected] writes:

Just curious, have there been any updates on this style of behaviour since the conversation 7 months ago?

No, only improvements to helm-find-files.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[egbulmer]

Thanks for the update @thierryvolpiatto :grin:

[bling]

any possibility of making fuzzy search available for all helm buffers? fuzzy search is immensely useful for many users, especially those migrating away from vim or sublime.

i took a look at the code and so far it seems that a slight change in helm-buffers to call helm-ff-mapconcat-candidate will do the trick.

diff --git a/helm-buffers.el b/helm-buffers.el
index cad5a4b..aa490f0 100644
--- a/helm-buffers.el
+++ b/helm-buffers.el
@@ -359,9 +359,10

 (defun helm-buffer--match-pattern (pattern candidate)
-  (if (string-match "\\`!" pattern)
-      (not (string-match (substring pattern 1) candidate))
-    (string-match pattern candidate)))
+  (let ((ptn (helm-ff-mapconcat-candidate pattern)))
+    (if (string-match "\\`!" ptn)
+        (not (string-match (substring ptn 1) candidate))
+      (string-match ptn candidate))))

[thierryvolpiatto]

Bailey Ling [email protected] writes:

any possibility of making fuzzy search available for all helm buffers?
fuzzy search is immensely useful for many users, especially those
migrating away from vim or sublime.

i took a look at the code and so far it seems that a slight change in
helm-buffers to call helm-ff-mapconcat-candidate will do the trick.

What you want to modify is helm-buffer--match-pattern' and not helm-buffers-match-inside'.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

I have intalled a change allowing this when helm-buffers-fuzzy-matching is non--nil.
See aae877e190b672a87677270b937a565b604ca526

[bling]

awesome! thanks. i'm not sure why, but virtual buffers seem to be going through a different matching algorithm, as they are not being fuzzy searched.

[thierryvolpiatto]

Bailey Ling [email protected] writes:

awesome! thanks. i'm not sure why, but virtual buffers seem to be
going through a different matching algorithm, as they are not being
fuzzy searched.

Yes, virtual buffers are not using the same match function, the reason
is the virtual buffers are non--existing buffers and major-mode checking
cannot be used.

I will make soon a generic fuzzy match plugin that can be used in such
places.

Thanks.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[dgellow]

any news on this issue ?

[unbalancedparentheses]

I like helm, but I prefer smex rather than Helm-M-x or fiplr/projectile rather than helm-projectile because helm does not have a good fuzzy finder. For example you need to type dash for example if what you are looking for has a dash, and that is not the case with fiplr, projectile or smex for commands.

[unbalancedparentheses]

Oh if you replace the dash by space it does work fine, but still it does not have proper fuzzy finding.

[tuhdo]

@unbalancedparentheses it does have other things to make fup for it. For example, you can open multiple files with helm-projectile; this is much faster than fuzzy searching file by file, and Helm provides you out of order matching: that is, regardless of the pattern order of candidates you want to find, you end up with the same thing. You can open Dired/Magit/Eshell on a project. You can also perform incremental grep at project root with helm-projectile. And more. I'm going to write a guide for helm-projectile soon once my changes are merged.

[codecoll]

@tuhdo but what is the technical problem holding back fuzzy matching in Helm? If the algorithm is already implemented in elisp then what prevents it to add fuzzy matching too as an alternative algorithm to use (configurable by the user, so the current default matching mechanism should also be kept of course, because it has its own advantages, but the user could turn on fuzzy matching either for all sources, or only for individual sources with an attribute)

[tuhdo]

@nerton Probably because of the highlighting. Le Wang said he tried, but he could not finish it because he didn't know how to integrate with Helm's highlighting: https://github.com/lewang/flx/issues/59

[codecoll]

@tuhdo then I guess it should be added without highlighting first. It could be useful for people who like fuzzy matching even without highlighting if they find fuzzy matching more effective, and highlighting could be added later as an improvement

[tuhdo]

You could ask @lewang how to integrate Helm with flx, since Le Wang said he did it.

[Pitometsu]

As for me, fuzzy matching pretty useful, but just as an addition: e.g. I'll type some_string and I expect to see things in that order:

1. the same string (if exist)
2. strings that start with that string (if exist)
3. strings that contain my string (if exist)
4. string that strongly matches my string except character order (. . .)
5. string that start with string that match my string with any character order
6. string that just contain similar one
7. strings that contain some of characters somewhere in some order from string. Better matched -- first.

And about speed -- string may check dynamically, on idle. So better matches calculate faster.
Additionally it may remember choices and use it with better priority according to:

1. how recently
2. frequency of using it
3. count of using item

Regards!

[thierryvolpiatto]

Actually, helm-find-files and helm-buffers-list are using fuzzy matching (if enable for buffers).

In exp branch (should be merged soon)
one can define source like this to have fuzzy-matching:

(helm :sources (helm-build-sync-source "test"
                 :candidates '("foo" "fox" "box" "bar" "baz")
                 :fuzzy-match t)
      :buffer "*helm test*")

(helm :sources (helm-build-in-buffer-source "test"
                 :data '("foo" "fox" "box" "bar" "baz")
                 :fuzzy-match t)
      :buffer "*helm test*")

[tuhdo]

Great news!

[tuhdo]

@thierryvolpiatto Now we have fuzzy matching, what Helm commands are going to make it default? Some commands I think suitable are:

• helm-M-x: would be nice if we have faster way to execute command.
• helm-semantic-or-imenu and helm-imenu: I tried it and it worked really well, and I think the number of candidates in a file is usually small anyway (I don't think anyone is going to put hundreds of functions in a file anyway).
• helm-recentf: by adding this, we can make helm-mini consistent as a whole. So, when people fuzzy search a buffer that is not available, they won't have to delete the whole pattern and enter again to search in helm-recentf.
• helm-apropos: the same reason with helm-M-x.
• Virtual Ido buffer list in helm-buffers-list for the same reason as above.

[bling]

can you please provide an option to make fuzzy default globally? i think for most users they'll either want to use fuzzy all the time or not at all.

[thierryvolpiatto]

Bailey Ling [email protected] writes:

can you please provide an option to make fuzzy default globally? i
think for most users they'll either want to use fuzzy all the time or
not at all.

No, fuzzy is good in some places, but not everywhere.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

So, what about an option to use fuzzy matching for stock commands that are "hemified"? If user prefers fuzzy matching, they can turn in on for every command that has no Helm source. Of course, the default is nil.

[dgellow]

@thierryvolpiatto

can you please provide an option to make fuzzy default globally? i think for most users they'll either want to use fuzzy all the time or not at all.

No, fuzzy is good in some places, but not everywhere.

So, make it nil by default and let people who like fuzzy everywhere activate it if they want to ?

[Pitometsu]

And what about option to enable fuzzy everywhere if count of candidate less than some_number. And do it for greatest number of candidates on idle? According to ergonomic reason, it's more comfortable for user if everywhere editor will use same behavior.

[tuhdo]

it seems like there's still a problem with accuracy. For example:

(helm :sources (helm-build-in-buffer-source "test"
                 :data (split-string (shell-command-to-string "git ls-files") "\n")
                 :fuzzy-match t)
      :buffer "*helm test*")

I tried it on Helm repo and type "escreen" (short for helm-elscreen.el). However, helm-elscreen is the 3rd match. It is worse for larger source tree, i.e. you could try with the sufficiently large source tree with about thousands of files.

[thierryvolpiatto]

Tu Do [email protected] writes:

I tried it on Helm repo and type "escreen" (short for
helm-elscreen.el). However, helm-elscreen is the 3rd match.

The two other are not matches, we need to double check if the
current-line match.
This should work now.

Thanks.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

That's better but there's still a problem. Here is the content of all my files in my .emacs.d, you can use this as sample: http://pastebin.com/tihkAz1s

When I tried elpacejump to match elpa/ace-jump, it returns an empty list.

[tuhdo]

Or when I enter elpjump, it also returns an empty list whenever the word "jump" is in it.

[thierryvolpiatto]

Tu Do [email protected] writes:

That's better but there's still a problem. Here is the content of all
my files in my .emacs.d, you can use this as sample:
http://pastebin.com/tihkAz1s

When I tried elpacejump to match elpa/ace-jump, it returns an empty list.

Thanks for the file for testing, I will look into this next week.
I have a lot of things to rewrite in candidates-in-buffer
implementation.
The sync version is working as expected though with correct
performances.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

It seems like the sync version is having problem with large collection as well. You can use my sample file list to test it. It has the same problem I described with in buffer version above.

[thierryvolpiatto]

Tu Do [email protected] writes:

It seems like the sync version is having problem with large collection
as well.

No it is working fine here with this code:

(with-current-buffer "my_fille_list.txt"
(helm :sources (helm-build-sync-source "test"
:candidates (save-match-data
(split-string (buffer-string) "n" t))
:fuzzy-match t))
:buffer "_helm test_")

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

Tu Do [email protected] writes:

You can use my sample file list to test it.

Ok it is working now if I remove all the crazy yasnippet filenames
containing space, parenthesis, dots etc...

Slowly though.

Can you try with helm-ls-git ?

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

(with-current-buffer "my_fille_list.txt" 
   (helm :sources (helm-build-sync-source "test" 
                    :candidates (save-match-data (split-string (buffer-string) "\n" t)) 
                    :fuzzy-match t)) 
         :buffer "*helm test*")

The function save-match-data changes things. It works as expected now.

I tried the exp branch. It's indeed better. I tried it with the Helm repo as before, as now helm-elscreen is correctly narrowed. But for my file list, it still has problem. For example, the example elpjump, or oryas+` still does not work. Here is how I tried it:

• Clone Helm and switch to fuzzy branch.
• Run emacs-helm.sh.
• Install helm-ls-git from MELPA.
• Tested with helm-ls-git.

[thierryvolpiatto]

Tu Do [email protected] writes:

(with-current-buffer "my_fille_list.txt"
(helm :sources (helm-build-sync-source "test"
:candidates (save-match-data (split-string (buffer-string) "n" t))
:fuzzy-match t))
:buffer "_helm test_")

The function save-match-data changes things. It works as expected now.

I tried the exp branch. It's indeed better. I tried it with the Helm
repo as before, as not helm-elscreen is correctly narrowed. But for my
file list, it still has problem. For example, the example elpjump, or
oryas+` still does not work.

Your file list contain invalid filenames from yasnippet try to remove
them, otherwise you have to write something much more complex (like
helm-find-files with handles such crap strings).

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

But we cannot assume users will have to avoid such cases? Is it possible to reuse the matching function for helm-find-files explicitly rather than using the generic one? For example, you could use the fuzzy match function from helm-find-files for helm-ls-git.

[thierryvolpiatto]

Tu Do [email protected] writes:

For example, the example elpjump, or oryas+` still does not work.

It is working now, but it is very slow, you have to wait several seconds
after entering "jump" before seeing the result, there is a bottleneck
somewhere, I am trying to fix it actually.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[PythonNut]

Does helm take advantage of flx caching? It's pretty easy. I believe you can feed flx-strings-cache to flx-score for regular string matching, and flx-file-cache for file matching.

... or does it use flx at all?

[thierryvolpiatto]

@tuhdo Can you try again ?

(with-current-buffer (find-file-noselect "~/tmp/my_fille_list.txt")
  (helm :sources (helm-build-in-buffer-source "test"
                   :data (save-match-data
                           (split-string (buffer-string) "\n" t))
                   :fuzzy-match t)
        :buffer "*helm test*"))

BTW, FYI the typo in "my_fille_list.txt" mean in french "my_girl_list.txt" :-)

[tuhdo]

@thierryvolpiatto It works really good now. But there are a few cases like this:

elpa/yasnippet-20141102.1554/snippets/c++-mode/const_[]
elpa/yasnippet-20141102.1554/snippets/c++-mode/d_operator[]
elpa/yasnippet-20141102.1554/snippets/c++-mode/d_operator[]_const
elpa/yasnippet-20141102.1554/snippets/c++-mode/delete[]
elpa/yasnippet-20141102.1554/snippets/c++-mode/operator[]

When I enter yas], those candidates are displayed but when I enter yas[], I got no candidate. But it's definitely much faster and more accurate.

BTW, FYI the typo in "my_fille_list.txt" mean in french "my_girl_list.txt" :-)

Woot I've just looked up that word and I know its "more accurate" meaning :). Just on character for such coincidence, I'm amazed.

[PythonNut]

@tuhdo Are you sure that yas[] isn't being parsed as a regex?

[tuhdo]

@PythonNut Yes, it works in helm-find-files. Of course, you have to get into that directory which contains [] pattern. But such pattern is not treated as regex.

[PythonNut]

Hmm.. that's very strange. flx apparently parses symbols just fine.

[tuhdo]

@PythonNut It's getting much better. Once this is fix and highlighting is added, nothing we can miss from Ido and flx.

[thierryvolpiatto]

Tu Do [email protected] writes:

When I enter yas], those candidates are displayed but when I enter
yas[], I got no candidate.

Yes I have noticed this, fortunately it is not due to the fuzzy search
function but to matchplugin function (mp-3) that try a regexp matching.
Just disabling it with ":matchplugin nil" fix the problem, but I am
working on this actually to allow the both methods working together.

But it's definitely much faster and more accurate.

Good,

thanks for feedback.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

Tu Do [email protected] writes:

When I enter yas], those candidates are displayed but when I enter
yas[], I got no candidate.

This is now working without disabling match-plugin
(i.e :matchplugin nil)

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

This is now working without disabling match-plugin (i.e :matchplugin nil)

Thanks. I've tried and it works. Is there any plan to add highlighting?

[thierryvolpiatto]

Tu Do [email protected] writes:

This is now working without disabling match-plugin (i.e :matchplugin nil)

Thanks. I've tried and it works. Is there any plan to add highlighting?

No, highlighting is now disabled when fuzzy matching, because
highlighting text that match a fuzzy regexp would highlight most part of
buffer (i.e most part of each lines) which would be an anoyance.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[Pitometsu]

You may highlight it like in regexp-builder: not whole matched expression, but strongly matched parts.

[thierryvolpiatto]

Netsu [email protected] writes:

You may highlight it like in regexp-builder: not whole matched
expression, but strongly matched parts.

You can try in regexp-builder a fuzzy expression, you will see the whole
buffer (or nearly) is highlighted.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

It seems like it is having problem with larger file list like the Linux kernel. I tried fuzzy searching for a candidate like init/main.c with initmain pattern, but it returns nothing. Occasionally there are results but most of the time none appears. Helm without fuzzy and ido + flx works fine with it. You can download the file list here.

[PythonNut]

@Pitometsu The problem is that matching h.*?f.*?m would match almost all of helm-flx-mode (if such a symbol were to exist), and fail to yield much information to the user (it would appear as helm-flx-m⁣​ode). _Real_ flex matching (like that is done in flx-ido: h​elm-f​lx-m⁣​ode) is much harder.

[Pitometsu]

@PythonNut I have in ming regex groups, similar to \\(h\\)\\(\\(.*\\)?\\)\\(f\\)\\(\\(.*\\)?\\)\\(m\\)\\(\\(.*\\)?\\). So _helm_ may highlight only some subgroups.

[thierryvolpiatto]

Current state with following code:

(with-current-buffer (find-file-noselect "/home/thierry/download/file_list.txt")
  (helm :sources (helm-build-sync-source "test"
                   :candidates (save-match-data
                                 (split-string (buffer-string) "\n" t))
                   :fuzzy-match t)
        :input "initmain"
        :buffer "*helm test*"))

(with-current-buffer (find-file-noselect "/home/thierry/download/file_list.txt")
  (helm :sources (helm-build-in-buffer-source "test"
                   :data (save-match-data
                           (split-string (buffer-string) "\n" t))
                   :fuzzy-match t)
        :input "initmain"
        :buffer "*helm test*"))

• Sorting: done.
• Highlighting: done.
• Performance are now quite good:

(with-helm-time-after-update
    (with-current-buffer (find-file-noselect "/home/thierry/download/file_list.txt")
         (helm :sources (helm-build-sync-source "test"
                          :candidates (save-match-data
                                        (split-string (buffer-string) "\n" t))
                          :fuzzy-match t)
               :input "initmain"
               :buffer "*helm test*")))
;; =>1.205902099609375


(with-helm-time-after-update
    (with-current-buffer (find-file-noselect "/home/thierry/download/file_list.txt")
         (helm :sources (helm-build-in-buffer-source "test"
                          :data (save-match-data
                                  (split-string (buffer-string) "\n" t))
                          :fuzzy-match t)
               :input "initmain"
               :buffer "*helm test*")))
;; =>0.9823942184448242

• I still need to fix confusion around negation with/without fuzzy.
• Need now to use cache for match-part also.

[tuhdo]

Great. Thank you Thierry. Work nicely for most cases. The only thing left is this case (again):

elpa/yasnippet-20141102.1554/snippets/c++-mode/const_[]
elpa/yasnippet-20141102.1554/snippets/c++-mode/d_operator[]
elpa/yasnippet-20141102.1554/snippets/c++-mode/d_operator[]_const
elpa/yasnippet-20141102.1554/snippets/c++-mode/delete[]
elpa/yasnippet-20141102.1554/snippets/c++-mode/operator[]

When I enter yas], those candidates are displayed but when I enter yas[], I got no candidate.

[thierryvolpiatto]

Tu Do [email protected] writes:

Great. Thank you Thierry. Work nicely for most cases. The only thing left is this case (again):

elpa/yasnippet-20141102.1554/snippets/c++-mode/const_[]
elpa/yasnippet-20141102.1554/snippets/c++-mode/d_operator[]
elpa/yasnippet-20141102.1554/snippets/c++-mode/d_operator[]_const
elpa/yasnippet-20141102.1554/snippets/c++-mode/delete[]
elpa/yasnippet-20141102.1554/snippets/c++-mode/operator[]

When I enter yas], those candidates are displayed but when I enter yas[], I got no candidate.

Argh! forget about this one, thanks fixed.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

Tu Do [email protected] writes:

Great. Thank you Thierry. Work nicely for most cases. The only thing left is this case (again):

I need to fix several things though.

Especially:

• negation.
• match-part.

In this both cases searching is deriving to the search/match-part
functions which are not optimized (performances) and handle the negation
wrongly.

When this will be done, we should have something correct.

Thanks for testing.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

@thierryvolpiatto I mentioned this before but I will remind again. We have properly fuzzy matching with highlighting now, what Helm commands are going to make it default? Some commands I think suitable are:

• helm-M-x: would be nice if we have faster way to execute command.
• helm-semantic-or-imenu and helm-imenu: I tried it and it worked really well, and I think the number of candidates in a file is usually small anyway (I don't think anyone is going to put hundreds of functions in a file anyway).
• helm-recentf: by adding this, we can make helm-mini consistent as a whole. So, when people fuzzy search a buffer that is not available, they won't have to delete the whole pattern and enter again to search in helm-recentf.
• helm-apropos: the same reason with helm-M-x.
• Virtual Ido buffer list in helm-buffers-list for the same reason as above in helm-recentf.
• helm-lisp-complete-at-point
• helm-dabbrev.

[tuhdo]

If it is ok, I can help you add fuzzy matching for those sources.

[thierryvolpiatto]

Tu Do [email protected] writes:

@thierryvolpiatto I mentioned this before but I will remind again. We
have properly fuzzy matching with highlighting now, what Helm commands
are going to make it default?

I don't want to go too fast, and want to test command by command first,
to see if fuzzy is suitable for this or that.
We already use it for files and buffers, and yes its nice for those, so
the file/buffer source could use it by default.

Anyway my work is not finished, I need to fix some inconsistencies,
document, polish the code...
Some behaviors may change so I don't want to throw fuzzy everywhere
right now, I need more time to finish.

Some commands I think suitable are:

• helm-M-x: would be nice if we have faster way to execute command.

Not sure, the multi matching is more appropriate for this one IMO i.e
"file find" or "com shell" will make find-file or shell-command go on
top of list.
Multi matching is also faster.

• helm-semantic-or-imenu and helm-imenu: I tried it and it worked
  really well, and I think the number of candidates in a file is
  usually small anyway (I don't think anyone is going to put
  hundreds of functions in a file anyway).

Maybe, need to try.

• helm-recentf: by adding this, we can make helm-mini consistent as
  a whole. So, when people fuzzy search a buffer that is not
  available, they won't have to delete the whole pattern and enter
  again to search in helm-recentf.

Yes.

• helm-apropos: the same reason with helm-M-x.

Same as M-x.

• Virtual Ido buffer list in helm-buffers-list for the same reason
  as above in helm-recentf.

Yes.

• helm-lisp-complete-at-point

Same as M-x.

• helm-dabbrev.

Need to try, we really need speed for helm-dabbrev, depending the
settings, the search maybe huge.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

Sure. I just want to know what you think and start playing with those sources, so it will be ready when fuzzy matching in Helm is complete.

Not sure, the multi matching is more appropriate for this one IMO i.e "file find" or "com shell" will make find-file or shell-command go on top of list. Multi matching is also faster.

Consider how popular package like smex is, we can safely assume that people love fuzzy matching in general and find it useful. Given current fuzzy matching can work with 30k files, I don't think this is a problem for helm-M-x. I have over 100 packages installed and I doubt I have over 10k commands. I will test to see how helm-M-x behaves with fuzzy matching. Probably we could create a variable to enable fuzzy matching for helm-M-x, similar to helm-buffers-fuzzy-matching.

The same can be applied for code completion like helm-lisp-complete-at-point. In general, people care less about the dash; they just want to retrieve as fast as possible, for example, hff instead of hel fin fil. It is useful if we have something like this for Emacs Lisp, similar to slime-fuzzy-complete-symbol. And it will be unique because there's no fuzzy-matching completion package for Emacs Lisp yet.

[priyadarshan]

If I may add a note as a user and a fan, if fuzzy matching for helm-M-x is not meant to be as default, could we have it as an option? I would like very much to simplify my toolchain, avoiding to load smex as well. Thank you so much for helm!

[thierryvolpiatto]

Tu Do [email protected] writes:

Sure. I just want to know what you think and start playing with those
sources, so it will be ready when fuzzy matching in Helm is complete.

Not sure, the multi matching is more appropriate for this one IMO
i.e "file find" or "com shell" will make find-file or
shell-command go on top of list. Multi matching is also faster.

Consider how popular package like smex is, we can safely assume that
people love fuzzy matching in general and find it useful. Given
current fuzzy matching can work with 30k files, I don't think this is
a problem for helm-M-x. I have over 100 packages installed and I doubt
I have over 10k commands. I will test to see how helm-M-x behaves with
fuzzy matching. Probably we could create a variable to enable fuzzy
matching for helm-M-x, similar to helm-buffers-fuzzy-matching.

Ok we can start trying, I must provide a new keyword to helm-comp-read
though before being able to do this.

Now with fuzzy matching enabled, we can do both multi regexp matching
and fuzzy-matching:
If a space is in pattern fuzzy is disabled and job is delegated to
match-plugin.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[f0086]

It is awesome to see that fuzzy matching is comming soon (before cristmas? :D) I've lurking this ticket for over a year now because I want to replace my ido/smex setup with the superior helm. The only reason I can't switch to helm is that I need fuzzy matching to work. So, keep on going :+1:

[thierryvolpiatto]

Tu Do [email protected] writes:

• helm-recentf: by adding this, we can make helm-mini consistent as
  a whole. So, when people fuzzy search a buffer that is not
  available, they won't have to delete the whole pattern and enter
  again to search in helm-recentf.

Done for this one.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

I don't know if it has something to do with caching but when I enable fuzzy matching for helm-projectile-find-file (basically it is the same as helm-ls-git-ls, lists all files in the project), it took a while to open the file list, even though the list is cached inside a hash table. Disable fuzzy matching and I get the list instantly.

[thierryvolpiatto]

Tu Do [email protected] writes:

I don't know if it has something to do with caching but when I enable
fuzzy matching for helm-projectile-find-file (basically it is the same
as helm-ls-git-ls, lists all files in the project), it took a while to
open the file list, even though the list is cached inside a hash
table. Disable fuzzy matching and I get the list instantly.

Don't know for projectile, but with helm-ls-git, with status+lst
sources, update is done in 0.39s which seem correct:

(with-helm-time-after-update
(let ((default-directory "~/labo/emacs"))
(helm :sources '(helm-source-ls-git-status
helm-source-ls-git)
;; When helm-ls-git-ls' is called from lisp ;;default-directory' is normally let-bounded,
;; to some other value;
;; we now set this new let-bounded value local
;; to `helm-default-directory'.
:default-directory default-directory
:buffer "_helm lsgit_")))
=>0.3983945846557617

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

Tu Do [email protected] writes:

• helm-M-x: would be nice if we have faster way to execute command.

Done for this one too.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

I will check it later. It seems to happen through a period of time, not instantly. And it's awesome to see helm-M-x to have fuzzy matching now. Thanks.

[tuhdo]

After updating, helm-M-x and helm-recentf has exactly the delay (quite long) as I described when enabled fuzzy matching in Helm Projectile. When I invoked helm-M-x, I has to wait for a while but as soon as I entered something during the delay, it response immediately.

[thierryvolpiatto]

Tu Do [email protected] writes:

After updating, helm-M-x and helm-recentf has exactly the delay (quite
long) as I described when enabled fuzzy matching in Helm
Projectile. When I invoked helm-M-x, I has to wait for a while but as
soon as I entered something during the delay, it response immediately.

That strange, as at start of M-x no sorting is done at all and fuzzy
matching is not started.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

It works fine now. Probably MELPA was retrieving the older version or the old byte compiled files were there. It works fine for Helm Projectile now. Thanks.

[thierryvolpiatto]

Tu Do [email protected] writes:

When I invoked helm-M-x, I has to wait for a while but as soon as I
entered something during the delay, it response immediately.

Did you modify global default value of `helm-candidate-number-limit' ?

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

We could add a special behaviour for fuzzy-matching in helm-M-x to improve it: when I tried smex and , I notice that it always match the first characters of words in a command separated by a dash. For example, if want to search for helm-find-files, I will only need to enter hff. If only it fails to find anything, then the normal fuzzy match kicks in. We could optimize it like that.

[tuhdo]

Did you modify global default value of `helm-candidate-number-limit' ?

My default value is 500. It is still small and should not cause any problem. It works fine now.

[tuhdo]

Another source that could be used with fuzzy matching is helm-locate-library. I use it all the time :)

[thierryvolpiatto]

Tu Do [email protected] writes:

We could add a special behaviour for fuzzy-matching in helm-M-x to
improve it: when I tried smex and , I notice that it always match the
first characters of words in a command separated by a dash. For
example, if want to search for helm-find-files, I will only need to
enter hff. If only it fails to find anything, then the normal fuzzy
match kicks in. We could optimize it like that.

Not sure to understand this, can you develop ?

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

Tu Do [email protected] writes:

My default value is 500.

Bigger this value is, slower the display will be, especially at startup,
I use 200 as global value which is reasonable IMO.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

Not sure to understand this, can you develop ?

For example, when you enter hff, it selects like this: [h]elm-[f]ind-[f]iles; characters in the box are prioritized, and they are the first characters of each word separated by the hyphen -. Well, just a suggestion for improvement if it is possible. The current is fine by me now.

[thierryvolpiatto]

Tu Do [email protected] writes:

Not sure to understand this, can you develop ?

For example, when you enter hff, it selects like this:
[h]elm-[f]ind-[f]iles; characters in the box are prioritized, and they
are the first characters of each word separated by the hyphen -. Well,
just a suggestion for improvement if it is possible. The current is
fine by me now.

My sorting function actually don't take this in account, it would reduce
performances.
I tried many different things for sorting and finally ended up to this one,
which is enough smart to do good sorting but not too smart to keep good
performances.

Note that when fuzzy is enabled, you can switch immediately to a multi
regexp searching by entering a space, e.g with M-x, writing at prompt
"file find" allow matching find-file, which is not possible with fuzzy.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

Tu Do [email protected] writes:

Another source that could be used with fuzzy matching is helm-locate-library. I use it all the time :)

Ok, will do, don't remember where it is and how the code is but it
should be easy to enable.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[env0der]

+1 for this feature. It works really well in Sublime Text for example.

On Thu, Dec 4, 2014 at 6:46 AM, Tu Do [email protected] wrote:

Not sure to understand this, can you develop ?

For example, when you enter hff, it selects like this:
[h]elm-[f]ind-[f]iles; characters in the box are prioritized, and they
are the first characters of each word separated by the hyphen -. Well,
just a suggestion for improvement if it is possible. The current is fine by
me now.

—
Reply to this email directly or view it on GitHub
https://github.com/emacs-helm/helm/issues/145#issuecomment-65540833.

Best Regards,
Anton Onyshchenko

[PythonNut]

Is there a possibility of adding more intelligent flex _sorting_ in the future?

While, the current helm fuzzy is very nice, (and quite fast) it's harder to narrow down on what you want, as @tuhdo pointed out. @lewang is also a proponent of smart fuzzy sorting (obviously). flx is indeed slower, but it's actually much faster than you might expect (it can deal with thousands of candidates with ease). One strategy that I suspect would work very well is to use the inbuilt helm fuzzy to filter candidates, and after that have flx sort them. Then we can have flx kick in only when the number of filtered candidates is small (say <5000).

With this method, you can start with a massive dataset (say a list of files in projectile), and start typing, say pmedtf which would match a few hundred candidates, and those would then be sorted to bring packages/modules/emacs/dumb_test_file.el to the top. flx would only sort those hundred candidates (to save time). But in the end, you'd get better results.

This is the strategy Sublime uses (but with their own intelligent fuzzy algorithm), IIRC.

[Pitometsu]

+1 for sorting.
For example in *helm buffers* when I type "pr", I have first line with "Helper" and second with "Process".
Surely I expect to see "Process" on first line. And fuzzy work just fine, thank you a lot for amazing feature!

[thierryvolpiatto]

Netsu [email protected] writes:

For example in _helm buffers_ when I type "pr", I have first line with
"Helper" and second with "Process". Surely I expect to see "Process"
on first line.

The sort function for fuzzy haven't been optimized yet for buffers.
(Fuzzy have been implemented slighly differently here and need some reworking)

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[lewang]

On Thu, Dec 4, 2014 at 3:20 PM, Thierry Volpiatto [email protected]
wrote:

The sort function for fuzzy haven't been optimized yet for buffers.
(Fuzzy have been implemented slighly differently here and need some
reworking)

What did you have in mind for the sort function? Is it already implemented
elsewhere? From experience to make a robust sorter is not trivial. I
initially started with regexp based solutions and soon bumped into stack
overflow errors.

If you can implement it as a filter that takes a list and returns a sorted
list then I can add flx feature detection, and sort through flx if
available.

So the work flow could be:

1. Helm does a flex search basically converting a search term - "query"
   into "q._u._e._r._y"
2. The number of matches is counted until it's below a certain threshold.
3. After the threshold is crossed, helm starts filtering search results
   through a function.

Le

[syohex]

Would you tell me how to disable fuzzy search ?

[thierryvolpiatto]

Le Wang [email protected] writes:

What did you have in mind for the sort function?

You come after the battle, the job is done now ;-).

Is it already implemented elsewhere?

Yes.
Any sort function can be used anyway, it is configurable.

From experience to make a robust sorter is not trivial.

Indeed.

I initially started with regexp based solutions

I use contiguous matches, enough smart but not too much to keep decent
performances and not regexp based.
I probably will improve it in the next months.

and soon bumped into stack overflow errors.

Yes of course.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

Syohei YOSHIDA [email protected] writes:

Would you tell me how to disable fuzzy search ?

For helm-M-x set helm-M-x-fuzzy-match to nil.
In other places where it have been enabled recently, I haven't already
added the user vars to turn it off, will come soon.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[syohex]

@thierryvolpiatto Thanks for information.

[meganeuker]

Just my two cents: this is good stuff, especially for files and buffers, but unless things are sped up, I think it should be turned off by default for M-x. I have helm-candidate-number-limit set to 200, yet it is still unbearably slow.

[PythonNut]

@thierryvolpiatto, well, if the sort function is configurable, can I do something like this:

(defun flx-isearch-sort (str symbols &optional cache)
  "Sort the strings in `symbols' according to their `flx-score'
with respect to `str'"
  (sort symbols
    (lambda (a b)
      (>
        (car (flx-score (car a) str cache))
        (car (flx-score (car b) str cache))))))

Or at least make it such that I can configure helm to use it.

[thierryvolpiatto]

Farasha Euker [email protected] writes:

Just my two cents: this is good stuff, especially for files and
buffers, but unless things are sped up, I think it should be turned
off by default for M-x.

It will be nil by default, I just set it to t to have some feed back.
Same for helm-apropos.

I have helm-candidate-number-limit set to 200, yet it is still
unbearably slow.

I wouldn't say unbearably, but yes it is slow though it is usable.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[meganeuker]

Good to hear. I wouldn't say it is Helm's fault for being slow, but my computer is very old. Aside from that, it takes some getting used to since the same few letters typed before often lead to very different results. For instance, without fuzzy, od would give me org-drill, which I want, but fuzzy would ironically give me ido-mode, which I certainly don't want since I am using Helm. I will keep testing, but at least for my use case, I think fuzzy on everywhere but M-x works well.

[thierryvolpiatto]

Farasha Euker [email protected] writes:

Good to hear. I wouldn't say it is Helm's fault for being slow, but my
computer is very old.

Ok I understand, with a modern i5 processor, fuzzy is not fast, but
really usable, at least for M-x.
Probably people typing very fast and not able to wait 0.5/1s will find
it also unusable, but well...

Aside from that, it takes some getting used to since the same few
letters typed before often lead to very different results. For
instance, without fuzzy, od would give me org-drill, which I want, but
fuzzy would ironically give me ido-mode, which I certainly don't want
since I am using Helm. I will keep testing, but at least for my use
case, I think fuzzy on everywhere but M-x works well.

It is now disabled by default everywhere except in files/buffers
completion where it is reasonably fast.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[Pitometsu]

Symbol's function definition is void: helm-fuzzy-matching-default-sort-fn-1

When I try set Helm M X Fuzzy Match to t.

[thierryvolpiatto]

Netsu [email protected] writes:

Symbol's function definition is void: helm-fuzzy-matching-default-sort-fn-1
When I try set Helm M X Fuzzy Match to t.

Can't reproduce.
Typically a bad install problem.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[Pitometsu]

thank you, after reinstalling it work.

[tuhdo]

helm-locate-library lost highlighting after the latest update.

[tuhdo]

We should enable helm-semantic-fuzzy-match by default because usually the number of candidates is small. Currently, toggling helm-semantic-fuzzy-match has no effect because the source is already defined in a varible, not dynamically created when invoking helm-semantic command..

[tuhdo]

I'm having degrading performance after using fuzzy matching for a while. For example, if I use helm-M-x, after using it a few times, it starts getting long to open helm-M-x buffer again. I had to disable fuzzy matching in it. Similar for helm-mini with helm-recentf inside it.

To make it easier to reproduce, you can try it with my repo:

git clone https://[email protected]/hoangtu/.emacs.d.git

Just clone it and run. Every package is already in there. I know I should be using emacs-helm.sh, but in my case I have many packages installed and as a result, have a lot of commands. Probably for such a large amount of commands, it slows Helm down.

[tuhdo]

Someone is also [having problem with helm-recentf](http://emacs.stackexchange.com/questions/4243/how-to-fix-problem-with-fuzzy-matching-in-helm-recentf). I think we should also provide an option forhelm-recentf`.

[tuhdo]

The problem with fuzzy matching exhibits when workgroups2 is enabled and I switch workspace. I never had this problem with non-fuzzy Helm before.

You could try comment/uncomment workgroups2 configuration code in my init.el, at the bottom of the file.

[thierryvolpiatto]

Tu Do [email protected] writes:

Someone is also [having problem with `helm-recentf. I think we should also provide an option for helm-recentf.

For the given example, user can:

1) Fuzzy match against basename to match "ini" by either entering " -b"
after "ini" or by switching to basename with `C-]'.

2) Adding a space after "ini" to make a regexp match (fall back to
matchplugin as soon a space is detected).

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

Tu Do [email protected] writes:

I'm having degrading performance after using fuzzy matching for a
while. For example, if I use helm-M-x, after using it a few times, it
starts getting long to open helm-M-x buffer again. I had to disable
fuzzy matching in it. Similar for helm-mini with helm-recentf inside
it.

Start by setting candidate-number-limit to a lower value, really 100
(default) is enough, especially for M-x, the only place I need a bigger
value is for buffers, but this can be set individually.

About performances degrading after some time I don't see why, as helm is
not storing lot of data in a cache or something.

To make it easier to reproduce, you can try it with my repo:

git clone https://[email protected]/hoangtu/.emacs.d.git

Just clone it and run. Every package is already in there. I know I
should be using emacs-helm.sh, but in my case I have many packages
installed and as a result, have a lot of commands. Probably for such a
large amount of commands, it slows Helm down.

I don't think the slowness is related to your many commands.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

I already disabled workgroup2 and it is fine now. Helm starts getting delay when I switch to another workspace. It gets worse the more I use Helm commands with fuzzy-matching.

I don't think it's about candidate-number-limit, because even though setting to 500 or higher, it should still be usable. I've been using that number for a year and found no problem. After disabling workgroup2, Helm commands with fuzzy matching are fast again.

[thierryvolpiatto]

Tu Do [email protected] writes:

I already disabled workgroup2 and it is fine now. Helm starts getting
delay when I switch to another workspace. It gets worse the more I use
Helm commands with fuzzy-matching.

I don't think it's about candidate-number-limit, because even though
setting to 500 or higher, it should still be usable. I've been using
that number for a year and found no problem. After disabling
workgroup2, Helm commands with fuzzy matching are fast again.

So isn't the problem coming from workgroup2 ?
I.e something is taking resources, I think you see it with fuzzy in helm
but you could see the same slowness with any other costly emacs command
in this same context.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[scaramouche1]

I believe it would be useful to have an easy way to disable fuzzy matching in specific helm functions. For instance, fuzzy matching in helm-recentf does not produce good matches: it discards the recency information collected by recentf (i.e., matches are not sorted according to recency; see an example here).

[thierryvolpiatto]

scaramouche1 [email protected] writes:

I believe it would be useful to have an easy way to disable fuzzy
matching in specific helm functions. For instance, fuzzy matching in
helm-recentf does not produce good matches: it discards the recency
information collected by recentf (i.e., matches are not sorted
according to recency; see an example here).

You will have this (i.e disabling fuzzy) in version 1.6.5, it is coming
soon.

BTW arguing the behavior is un-emacs is true also for helm generally
that try to be more clever than emacs vanilla, so probably an effort can
be made to integrate these more advanced behaviors.

Also sorting by recency after entering something in pattern is
non sense.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[scaramouche1]

Thank you for the fast response and change to the code. Now the default behavior of helm-recentf is perfect. BTW, my comment on "un-emacs" was not about Helm (which is a wonderful addition to Emacs), it was just about helm-recentf not behaving in a "do what I mean" manner. Many thanks!

[tuhdo]

After the latest commit, enable helm-recentf-fuzzy-matching doesn't activate fuzzy matching for helm-recentf.

[thierryvolpiatto]

Tu Do [email protected] writes:

After the latest commit, enable helm-recentf-fuzzy-matching doesn't
activate fuzzy matching for helm-recentf.

Use either customize interface or

(custom-set-variables '(helm-recentf-fuzzy-match t)
'(helm-imenu-fuzzy-match t)
...etc)

setq will have no effect unless you rebuild the source yourself.

This is not true for the sources that are build at runtime, e.g *M-x or
*apropos.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[Pitometsu]

When I call _describe-function_ via helm, it can't match setq-local not with setl nor with setql, only with se and se l it found that.

[thierryvolpiatto]

Netsu [email protected] writes:

When call describe-function via helm, I can't match setq-local not
with setl nor with setql, only with se and se l it found that.

All the emacs commands helmized by helm-mode are not using fuzzy match
(and will not).

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[Pitometsu]

Is there any way to enable fuzzy for all commands? My laptop pretty fast, so probably it may be usable.

[Pitometsu]

UX rule 'same behavior everywhere' important for me.

[killdash9]

Thanks for implementing fuzzy matching. With the optimization I made in #773, the following went from 19 seconds to 1.5 seconds. It is working on a list of about 5000 files.

(with-helm-time-after-update
 (with-current-buffer (find-file-noselect "~/filelist.txt")
   (helm :sources (helm-build-sync-source "test"
                    :candidates (save-match-data
                                  (split-string (buffer-string) "\n" t))
                    :fuzzy-match t)
         :input "e"
         :buffer "*helm test*")))
=> 1.5118329524993896 (was 19 seconds before optimization)

[tuhdo]

@thierryvolpiatto There's something that can be improved: Helm should rank candidates that match the whole pattern first. Consider this example:

• First, download the file list from here: http://pastebin.com/F3cXcQFs
• Assume you already open the file list in Helm with fuzzy matching enabled, enter the pattern "const".
• The candidates that match the whole "const" are not at the top, but at the 3rd place and downward.

I think we can improve it by add another check if a candidate matches a whole pattern, then collect that candidate into a list that later be appended at the top. It should not have any performance problem thank to @killdash9 's performance improvement after it is merged..

[killdash9]

I agree it can be improved. Instead of creating a second list for whole matches, what about working it in to the scoring method and just giving it a +1 boost in score when it matches a whole word? That seems cleaner to me.

[tuhdo]

Here is another problem:

• Load the file list as above.
• Enter the pattern "langtool".

Now the pattern is scattered all over the matched candidates, even though the candidates have full match of "langtool" in it. We should check for whole match first before fuzzy match.

@killdash9 Yes, that's better but we must make sure that full matched candidates always have higher score than candidates with scatter match.

[thierryvolpiatto]

Tu Do [email protected] writes:

@thierryvolpiatto There's something that can be improved: Helm should
rank candidates that match the whole pattern first. Consider this
example:

• First, download the file list from here:
  http://pastebin.com/F3cXcQFs * Assume you already open the file list
  in Helm with fuzzy matching enabled, enter the pattern "const". *
  The candidates that match the whole "const" are not at the top, but
  at the 3rd place and downward.

I think we can improve it by add another check if a candidate matches
a whole pattern, then collect that candidate into a list that later be
appended at the top. It should not have any performance problem thank
to @killdash9 's performance improvement after it is merged..

Take in account that for matching, whole pattern is tried first by
the first matchplugin fn.
But yes for the sort fn which run when all candidates are fetched
we should give the highest score.

Thanks @tuhdo and @killdash9 for your work on this and test file sample.
I look into this as soon as possible, really no time to code these next
days.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

BTW you can test also performances on big directory list by running
C-x C-f => navigate to a big directory not under VC (or home/you) =>
C-u C-x C-d.

You have to enable fuzzy-match yourself in source code in helm-files.el
for now.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

No problem. Merry Christmas and happy new year!

[thierryvolpiatto]

Tu Do [email protected] writes:

@thierryvolpiatto There's something that can be improved: Helm should
rank candidates that match the whole pattern first.

Done.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

The pattern that contains [ breaks again. To reproduce:

• Load my file list above.
• Enter the pattern yas[ and no candidate is displayed.

[thierryvolpiatto]

Tu Do [email protected] writes:

The pattern that contains [ breaks again.

Oops, yes for the in-buffer implementation it break again indeed.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

Tu Do [email protected] writes:

The pattern that contains [ breaks again.

Should be fixed now, thanks.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

The fuzzy matching in helm-find-files should be updated. The problem is described in this thread. src is the exact match to src directory, but other candidates: .newsrc, .newsrc.eld, .newsrc-old, .newsrc-old-dribble and .newsrc-old.eld are ranked above src.

[thierryvolpiatto]

Tu Do [email protected] writes:

The fuzzy matching in helm-find-files should be updated. The problem
is described in this thread. src is the exact match to src directory,
but other candidates: .newsrc, .newsrc.eld, .newsrc-old,
.newsrc-old-dribble and .newsrc-old.eld are ranked above src.

Well the link is broken, so I haven't the details, but IIUC when you
have in a directory .newsrc.eld and friends and a directory named src/,
the newsrc* come first in the list, right ?

But it is not happening here, after creating a src/ in my /home/user and
I type src, the src/ directory is immediately expanded (So it comes
first).

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

Sorry for the link. Here is the full link: http://emacs.stackexchange.com/questions/5929/how-to-sort-file-list-in-helm-find-files

Probably you set helm-find-files to auto expand, so it may behave differently. I will look into it more tomorrow.

[tuhdo]

@thierryvolpiatto I tested this again. With helm-ff-auto-update-initial-value set to t, src/ directory is automatically expanded as the first result, but if you set it to nil, the ranking is exactly like this screenshot: src is at the bottom.

[thierryvolpiatto]

Tu Do [email protected] writes:

@thierryvolpiatto I tested this again. With
helm-ff-auto-update-initial-value set to t, src/ directory is
automatically expanded as the first result, but if you set it to nil,
the ranking is exactly like this screenshot: src is at the bottom.

Yes see #794.
I wrote helm-find-files with helm-ff-auto-update-initial-value == t in
mind, perhaps I will change this in the future, but is is not a priority
for now.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

Tu Do [email protected] writes:

@thierryvolpiatto I tested this again. With
helm-ff-auto-update-initial-value set to t, src/ directory is
automatically expanded as the first result, but if you set it to nil,
the ranking is exactly like this screenshot: src is at the bottom.

Finally fixed (see #794).

Thanks.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[ticktricktrack]

I love then new fuzzy matching, great work guys. My switch from Sublime is complete now.

[bling]

kudos for all the great work recently! :+1:

i've noticed that whenever i'm searching for something, and the one that i want is _not_ the best match, it can almost always become the best match if the list was sorted by minimum length. are others seeing this as well? if so, perhaps we can add this into the default sorting algorithm.

[thierryvolpiatto]

Bailey Ling [email protected] writes:

kudos for all the great work recently! :+1

i've noticed that whenever i'm searching for something, and the one
that i want is not the best match, it can almost always become the
best match if the list was sorted by minimum length. are others seeing
this as well? if so, perhaps we can add this into the default sorting
algorithm.

The default sort fn is already sorting by length when the score of the
two compared candidates are equal.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

Another improvement to improve best match:

• First, download file_list2.txt.
• Load into Helm with fuzzy matching enabled.
• Type boot
• Candidates are matched like this: "lab1/boot/boot.

Such candidate has exact match in its name for the given pattern. I think, in addition to check length, it should verify if the candidate also contains exact match. If so, give it a bonus +1 score since it is closer to our query than other scatter matched candidates.

[thierryvolpiatto]

Tu Do [email protected] writes:

Another improvement to improve best match:

• First, download file_list2.txt.
• Load into Helm with fuzzy matching enabled.
• Type boot
• Candidates are matched like this: "lab1/boot/boot.

Such candidate has exact match in its name for the given pattern. I
think, in addition to check length, it should verify if the candidate
also contains exact match. If so, give it a bonus +1 score since it
is closer to our query than other scatter matched candidates.

We are already doing this, what you want to match is the basename part
of the file, so you have to use match-part slot for this, e.g

(helm :sources (helm-build-in-buffer-source "test"
:data (with-current-buffer (find-file-noselect "/home/thierry/Téléchargements/file_list2.txt.txt")
(buffer-string))
:match-part (lambda (candidate) (helm-basename candidate)))
:buffer "_helm test_")

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

No I mean it should appear like this: "lab1/boot/boot", similar to non-fuzzy matching in Helm

[thierryvolpiatto]

Tu Do [email protected] writes:

No I mean it should appear like this: "lab1/boot/boot", similar to non-fuzzy matching in Helm

Sorry but I don't understand.

Appear where ?

(helm :sources (helm-build-in-buffer-source "test"
:data (with-current-buffer (find-file-noselect "/home/thierry/Téléchargements/file_list2.txt.txt")
(buffer-string))
:match-part (lambda (candidate) (helm-basename candidate)))
:buffer "_helm test_")

with this code, taping "boot" in minibuffer show "boot/boot.S"

What do you expect ?

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

I mean is it possible to have such behaviour with :fuzzy-match t? That is, without modifying matching behaviour.

[thierryvolpiatto]

Tu Do [email protected] writes:

I mean is it possible to have such behaviour with :fuzzy-match t? That
is, without modifying matching behaviour.

No because fuzzy don't know you are completing on fnames.
i.e basename have no meaning on e.g symbols.

So it is up to the developer to adapt his source to what he want to
match against.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[bling]

@thierryvolpiatto

here's what i'm seeing:

the 3 results at the bottom should be ranked higher if things are sorted by length, unless there's another match that's being given higher priority. the source of these files are from recentf.

a habit i developed when i used to use vim was to append the extension of the file as a way to rank it higher. often times you might have things like "awesome-thing.html" and "awesome-thing.css", so you can search for "awecss" to rank that higher. that trick doesn't seem to work all the time with the current helm implementation.

[thierryvolpiatto]

Bailey Ling [email protected] writes:

a habit i developed when i used to use vim was to append the extension
of the file as a way to rank it higher. often times you might have
things like "awesome-thing.html" and "awesome-thing.css", so you can
search for "awecss" to rank that higher.

Because awesome-thing.css and *.html are the basenames and helm for
recentf is matching against the whole path to allow matching also
directories.
If you want to match basename add "-b" at end of pattern, e.g
"awecss -b"
You can also achieve this by toggling basename with `C-]', that way no
need to add the "-b" at end.

that trick doesn't seem to work all the time with the current helm
implementation.

It may work if awecss is not matched in the directories names.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[PythonNut]

Sources that would benefit from fuzzy matching include helm-source-imenu and helm-source-semantic.

[thierryvolpiatto]

PythonNut [email protected] writes:

Sources that would benefit from fuzzy matching also include helm-source-imenu and helm-source-semantic.

They already have fuzzy.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

@PythonNut All fuzzy sources are not activated by default. See the list of sources here.

[PythonNut]

Hey everyone!

If you have a moment (and at least helm-20150110/ 2151577 ), try this config:

;; an emacs 24.4 macro. You know what to do if you have 24.3
(with-eval-after-load 'helm
  ;; make sure you have flx installed
  (require 'flx)
  ;; this is a bit hackish, ATM, redefining functions I don't own
  (defvar helm-flx-cache (flx-make-string-cache #'flx-get-heatmap-str))

  (defun helm-score-candidate-for-pattern (candidate pattern)
    (car (flx-score candidate pattern helm-flx-cache)))

  (defun helm-fuzzy-default-highlight-match (candidate)
    (let* ((pair (and (consp candidate) candidate))
            (display (if pair (car pair) candidate))
            (real (cdr pair)))
      (with-temp-buffer
        (insert display)
        (goto-char (point-min))
        (if (string-match-p " " helm-pattern)
          (cl-loop with pattern = (split-string helm-pattern)
            for p in pattern
            do (when (search-forward p nil t)
                 (add-text-properties
                   (match-beginning 0) (match-end 0) '(face helm-match))))
          (cl-loop with pattern = (cdr (flx-score display
                                         helm-pattern helm-flx-cache))
            for index in pattern
            do (add-text-properties
                 (1+ index) (+ 2 index) '(face helm-match))))
        (setq display (buffer-string)))
      (if real (cons display real) display))))

This replaces helm's fuzzy matching with @lewang's flx library, which has much more sophisticated (and magical) heuristics. I'm seeing very promising results on my lame Dell D620, Core2Duo w/ 2GB of RAM and a HDD.

In particular, I'm interested in performance. Is this config ever too slow? When?

Flx should, in theory, solve any complaints you guys have regarding the quality of flex matching. If it's fast enough, then maybe it can worked into the mainline (switching back to simple matching under known-slow cases).

[tuhdo]

@PythonNut Thank you. Really nice. It's fast, even on large source tree. You may want to download this file list for testing (it's 36k files). I tested with that list and had no problem, but you may want to have a sample to test against. Here is a simple code to test:

(with-current-buffer (find-file-noselect "file_list")
  (helm :sources (helm-build-in-buffer-source "test"
                   :data (save-match-data
                           (split-string (buffer-string) "\n" t))
                   :fuzzy-match t)
        :buffer "*helm test*"))

[tuhdo]

@PythonNut Also, it would be great if you could make Helmizied commands have fuzzy matching by default, sort of like Ido with its ubiquitous effect. Is it possible?

[thierryvolpiatto]

PythonNut [email protected] writes:

In particular, I'm interested in performance.

Bound the code sent by @tuhdo into with-helm-time-after-update to
benchmark your code.

Also you can use helm-fuzzy-matching-highlight-fn to bound your
helm-fuzzy-default-highlight-match fn.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[lewang]

Just a heads up that I'm working on flx based replacement for both the sort
and highlight functions. It will work similar to how flx works with ido
right now with better highlighter. The cache addition will improve
performance significantly once warmed.

It should be up in the next two days.

On Sun, Jan 11, 2015 at 12:46 AM, Thierry Volpiatto <
[email protected]> wrote:

PythonNut [email protected] writes:

In particular, I'm interested in performance.

Bound the code sent by @tuhdo into with-helm-time-after-update to
benchmark your code.

Also you can use helm-fuzzy-matching-highlight-fn to bound your
helm-fuzzy-default-highlight-match fn.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

—
Reply to this email directly or view it on GitHub
https://github.com/emacs-helm/helm/issues/145#issuecomment-69484569.

Le

[PythonNut]

@lewang Good news! I'd love to see a more optimized code that the hacks I posted.

@thierryvolpiatto The benchmarks are in! This was done with @tuhdo's file list and the following code:

(with-helm-time-after-update
  (with-current-buffer (find-file-noselect "~/Downloads/file_list.txt")
    (helm :sources (helm-build-in-buffer-source "test"
                     :data (save-match-data
                             (split-string (buffer-string) "\n" t))
                     :fuzzy-match t)
      :input "doabtesys"
      :buffer "*helm test*")))

Without flx
First run: 1.712 seconds
Average of next ten runs: 1.722 seconds

With flx
First run: 1.929 seconds
Average of next ten runs: 1.740 seconds

Remember that this is completely optimized. I dare say these numbers very competitive, and sorting with flx by default seems quite usable.

[thierryvolpiatto]

PythonNut [email protected] writes:

@lewang Good news! I'd love to see a more optimized code that the hacks I posted.

@thierryvolpiatto The benchmarks are in! This was done with @tuhdo's file list and the following code:

Without flx
First run: 1.712 seconds
Average of next ten runs: 1.722 seconds

Hmm, same code and same file, I am always under the second here, last
call was 0.8750095367431641.

With flx
First run: 1.929 seconds
Average of next ten runs: 1.740 seconds

What the purpose of flx if it isn't faster that this ?

Remember that this is completely optimized.

What do you mean ?

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[tuhdo]

What the purpose of flx if it isn't faster that this ?

Well, I must say, it's more accurate, so it is slower. I've tested using my older laptop (core i3 with 5400 rpm), and I must say it's quite slow with the file list above. The first time I did the test was on my MacBook Air with an SSD, so it was much faster and saw no difference.

Let's see how Le Wang's change can handle this. However, I think it's still good to incorporate flx to make it handle smaller source better. For large source, we still use Helm's default. There should be a variable that defines a threshold when to use which one.

What do you mean ?

I think he meant "isn't". Probably a typo.

[PythonNut]

What do you mean ?

I think he meant "isn't". Probably a typo.

Yes yes, sorry! I was tired. _Unoptimized_ is what I meant.

@tuhdo Did you test it without flx fuzzy? It only really makes sense to test flx relative to the current implementation. For example, on my Raspberry Pi, the slowest computer I could find that can run Emacs:

Average of 20 runs Without flx: 16.87 seconds
Average of 20 runs With flx: 17.239

flx may be slow here, but it's only 2% slower that vanilla helm. (On my laptop, it's 1% slower). Surely that's a negligible difference?

I'm curious to see how much faster Le's solution will be. My guess is that there are other bottlenecks besides the raw scoring of candidates.

Also you can use helm-fuzzy-matching-highlight-fn to bound your helm-fuzzy-default-highlight-match fn.

@thierryvolpiatto thanks for the heads up. Would it be possible to do something similar for helm-score-candidate-for-pattern?

[tuhdo]

@PythonNut To run the benchmark single time, yes, it's almost similar to stock Helm's fuzzy matching, although my fuzzy matching result is sometimes under 1 second. The problem, if you try the code without the benchmark call and enter actual input, then flx started lagging for that file list (it is usable for my more powerful MacBook Air though) and took a while to give result. Try inserting a pattern and delete a few characters and inserting some other characters. You will notice the performance degrade.

[tuhdo]

@thierryvolpiatto This could be improved: currently, when a pattern is matched, it could possibly have another substring matched the pattern. For example:

• Load this file list (again).
• Type "elhelm".
• These are the candidate list (truncated):

elpa/helm-20150107.2203/helm.el
elpa/helm-20150107.2203/helm.elc
elpa/helm-20150107.2203/helm-apt.el
elpa/helm-20150107.2203/helm-man.el
elpa/helm-20150107.2203/helm-net.el
..............truncated....................

The filename can also be matched and highlighted, but it isn't. The addition substrings could be anywhere, so using the basename won't help. Does having addition highlighting affect performance much?

[PythonNut]

@tuhdo I'm pretty sure that giving the basename priority is already done in flx. In regular fuzzy matching you can always type "helm" again. (Or, more intelligently "elhhelm")

[thierryvolpiatto]

PythonNut [email protected] writes:

@tuhdo I'm pretty sure that giving the basename priority is already
done in flx. In regular matching you can always type "helm"
again. (Or, more intelligently "elhhelm")

Please guys, send always the piece of code you are using for testing, we
don't now what we are speaking about, do we compare with using a
:match-part slot using base name or not ?

For flx, for now I have to say I am not convinced:

• it seems performances are not better and even slower that helm default.
• it seems according to what tuhdo said, there is a degradation of
  performances as long as you write in minibuffer.

Probably it is smarter than helm default for sorting, but this is not
the difficult part (i.e beeing smart), the difficult part is having
fuzzy matching with decent performances, so each second we win where
it's possible is good to take, so we have to find a good balance
"performance/smart_sorting".

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[PythonNut]

@thierryvolpiatto You can be both smart and fast. Just enable flx storing only when the number of candidates is, say, <5,000 (configurable number) and you're good to go. Helm will be smart when it can afford to be so, and fast when it cannot.

[f0086]

I have a little optimization for the fuzzy matching:
Most of the time, the inserted character should match the filename (or some part of it) for efficenty. The current fuzzy matching matches from the left. In my case, I guess something like /^.*a.*p.*p.*c.*o.*n.*t.*r.*o.*/.

It would be more accurate for the most finds if the matching will try to match at the end of the path. flx works this way and it feels much more intuitive.

[lewang]

@thierryvolpiatto I just want to be clear that what flx is about is not speed. It's about sorting smartly. I'm writing the plugin as a replacement for the default sorter/highlighter, so people can opt in.

In any case, I'm having problems implementing the highlighter portion. In my mind it works in 3 phases:

1. Match query against collection of candidates, and filter out non-matches.
2. helm-fuzzy-sort-fn sorts matching candidates based on some algorithm. At this point we don't propertize the display part, because we don't yet know how many results helm wants to display.
3. As helm displays each result it passes the candidate to helm-fuzzy-matching-highlight-fn to do the highlighting.

I'm actually seeing helm-fuzzy-matching-highlight-fn called before helm-fuzzy-sort-fn. Can you tell me where I'm misunderstanding how things work?

The specific problem I'm having is that the flx highlighter cannot function independently of the sorter, because given a query there may be multiple sets of matching characters, and only the sorter knows which set should be highlighted. So the sorter needs to run first and somehow pass the match data to the highlighter.

[lewang]

To elaborate on the problem, in @PythonNut 's implementation flx-score (where the hard work is done) is called twice for each candidate, first by the highlighter then by the sorter. In my workflow above, it would be called only by the sorter, and the match data is reused by the highlighter.

[thierryvolpiatto]

Le Wang [email protected] writes:

In any case, I'm having problems implementing the highlighter
portion. In my mind it works in 3 phases:

1. Match query against collection of candidates, and filter out
   non-matches.

Candidates are highlighted at this time.

1. helm-fuzzy-sort-fn sorts matching candidates based on some
   algorithm.

Yes, that's all, only two phases.

At this point we don't propertize the display part, because we don't
yet know how many results helm wants to display.

Why? the candidates list is the same, it is just sorted.

1. As helm displays each result it passes the candidate to
   helm-fuzzy-matching-highlight-fn to do the highlighting.

No, this would run another loop just to highlight candidates.

I'm actually seeing helm-fuzzy-matching-highlight-fn called before
helm-fuzzy-sort-fn. Can you tell me where I'm misunderstanding how
things work?

When helm is matching and collecting candidates it highlight each
candidate as soon as it is collected, when the list is complete, it sort
the list, this save one loop against candidates.

The specific problem I'm having is that the flx highlighter cannot
function independently of the sorter, because given a query there may
be multiple sets of matching characters, and only the sorter knows
which set should be highlighted. So the sorter needs to run first and
somehow pass the match data to the highlighter.

Hmm, so you will have to run an extra loop to highlight your candidates.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[PythonNut]

When helm is matching and collecting candidates it highlight each
candidate as soon as it is collected, when the list is complete, it sort
the list, this save one loop against candidates.

Could you not save time by highlighting only the candidates the user sees? (In the spirit of helm-quick-update) If you have a lot (in some cases >30,000) candidates, highlighting them all is almost certainly wasted effort. The user will never see them all.

This may save a loop, but you can save even more by not looping at all.

[lewang]

@thierryvolpiatto In addition to what @PythonNut is saying, say we are matching "xyz" against "--x-a-y-b-z--xy--xyz--". The current workflow highlights the first occurrence of the pattern yes? Does it also score based on the first occurrence?

Doing the "right thing" (i.e. one contiguous group match) here is what amplifies the power of fuzzy search, and really the main thrust of flx. I say main thrust because flx also takes other factors (distance between matches, whether match occurs at word break, in basename, etc, etc) into account which once you start offering "fuzzy" users will inevitably demand like @f0086 did above.

[thierryvolpiatto]

Le Wang [email protected] writes:

@thierryvolpiatto In addition to what @PythonNut is saying, say we are
matching "xyz" against "--x-a-y-b-z--xy--xyz--". The current workflow
highlights the first occurrence of the pattern yes? Does it also score
based on the first occurrence?

Scoring and highlighting are two different things, please provide small
recipe describing what you have and what you expect.

Doing the "right thing" (i.e. one contiguous group match) here is what
amplifies the power of fuzzy search,

We are actually scoring mainly on contiguous matches, + a bonus if the
contiguous matches occur on start of candidate, + a bonus if pattern is
matched exactly somewhere in candidate.

and really the main thrust of flx. I say main thrust because flx also
takes other factors (distance between matches, whether match occurs at
word break, in basename, etc, etc)

For now, I don't take in account distance and word break, I wanted
something not too smart at first to keep helm fast enough.

For basename, which is useful only on filenames, I made helm able to
configure this easily when needed, both with sync and in-buffer methods.

into account which once you start offering "fuzzy" users will
inevitably demand like @f0086 did above.

Users always demand even more at each time you implement something, so I
keep cool on this, I will make fuzzy more accurate slowly as the time
pass, I think what I did is a good start for now and really usable.

PS: About this thread, please everybody, provide example of what you are
talking about, refering to what @tuhdo, @pythonnut or whoever said mean nothing
as now the thread is very long.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

@tuhdo Issue #794 is now fully fixed and enabled by default with no loss of performances.

[tuhdo]

@thierryvolpiatto Great to hear that. Thanks.

[lewang]

On Thu, Jan 15, 2015 at 1:00 AM, Thierry Volpiatto <[email protected]

wrote:

I removed all the red herrings about the match algorithm. Let's address
one problem at a time:

Scoring and highlighting are two different things, please provide small

recipe describing what you have and what you expect.

Let's look at some code:

(momentary-string-display
(let ((strs '("--bxlyazh-x-a-y-b-z-xy-xyz-1--"))
(helm-pattern "xyz"))
(mapconcat (lambda (s)
(apply 'format "(%s %s)"
(list (helm-score-candidate-for-pattern s
helm-pattern)
(helm-fuzzy-default-highlight-match s))))
strs
" "))
(point))

What I see: first occurrence of "xyz" is highlighted.

What I expect: the portion of the string contributing the score (i.e. last
"xyz") is highlighted.

• The score function should output not just a numerical score, but the
  indices of matches that caused the score.
• The highlighter should do no searching on its own, but use the match data
  from the score function.

Le

[lewang]

No immediate changes are necessary on your end, Thierry. I'm setting
helm-fuzzy-matching-highlight-fn to nil and handling everything
from helm-fuzzy-sort-fn.

On Thu, Jan 15, 2015 at 10:17 AM, Le Wang [email protected] wrote:

On Thu, Jan 15, 2015 at 1:00 AM, Thierry Volpiatto <
[email protected]> wrote:

I removed all the red herrings about the match algorithm. Let's address
one problem at a time:

Scoring and highlighting are two different things, please provide small

recipe describing what you have and what you expect.

Let's look at some code:

(momentary-string-display
(let ((strs '("--bxlyazh-x-a-y-b-z-xy-xyz-1--"))
(helm-pattern "xyz"))
(mapconcat (lambda (s)
(apply 'format "(%s %s)"
(list (helm-score-candidate-for-pattern s
helm-pattern)
(helm-fuzzy-default-highlight-match s))))
strs
" "))
(point))

What I see: first occurrence of "xyz" is highlighted.

What I expect: the portion of the string contributing the score (i.e. last
"xyz") is highlighted.

• The score function should output not just a numerical score, but the
  indices of matches that caused the score.
• The highlighter should do no searching on its own, but use the match
  data from the score function.

Le

Le

[thierryvolpiatto]

Le Wang [email protected] writes:

No immediate changes are necessary on your end, Thierry. I'm setting
helm-fuzzy-matching-highlight-fn to nil and handling everything
from helm-fuzzy-sort-fn.

Ok good, thanks.

Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[lewang]

The first pass is done. Please don't publicize this until I have it's up as a project with documentation.

https://gist.github.com/lewang/34013fdaa2739a028eb8

You need to install the "heap" package from here: http://www.dr-qubit.org/emacs.php .

[PythonNut]

@lewang I believe heap can also be found on ELPA.

[tuhdo]

@lewang Not sure if I'm doing this right but I saw the same output before and after activate helm-flx-mode. Here is what I did:

• Install heap.
• Copy the code you provided into *scratch* buffer and eval it.
• Enable helm-flx-mode.

[lewang]

Don't use helm-M-x to enable the mode. It dynamically binds the sorter so
changes don't take effect. I'm working on a solution.
On 17 Jan 2015 00:58, "Tu Do" [email protected] wrote:

@lewang https://github.com/lewang Not sure if I'm doing this right but
I saw the same output before and after activate helm-flx-mode. Here is
what I did:

• Install heap.
• Copy the code you provided into _scratch_ buffer and eval it.
• Enable hem-flx-mode.

—
Reply to this email directly or view it on GitHub
https://github.com/emacs-helm/helm/issues/145#issuecomment-70356148.

[tuhdo]

@lewang I was able to use it. helm-M-x shows score now. However, it's only available for helm-M-x. Another thing is that, if I insert a space, an error occurs:

Error running timer: (wrong-type-argument number-or-marker-p nil)

[lewang]

On Sat, Jan 17, 2015 at 9:31 AM, Tu Do [email protected] wrote:

@lewang https://github.com/lewang I was able to use it. helm-M-x shows
score now. However, it's only available for helm-M-x.

The sources are cached, which prevents them from reflecting the updated
sorter value. @thierry will have to have a look at what to do about this.

helm-M-x is the only source that builds dynamically, but it's too dynamic
... :) It breaks turning on the mode due to dynamically binding to
helm-fuzzy-sort-fn

Another thing is that, if I insert a space, an error occurs:

Error running timer: (wrong-type-argument number-or-marker-p nil) [13
times]

Fixed here: https://github.com/lewang/helm-flx/blob/master/helm-flx.el

Le

[lewang]

I've created a branch with a temporary workaround for the cached sources -
https://github.com/lewang/helm-flx/blob/cached-source-workaround/helm-flx.el
.

I propose this bug can be closed at this point. I'll open pull-requests
and bugs for specific issues as I find them.

On Sat, Jan 17, 2015 at 4:32 PM, Le Wang [email protected] wrote:

On Sat, Jan 17, 2015 at 9:31 AM, Tu Do [email protected] wrote:

@lewang https://github.com/lewang I was able to use it. helm-M-x shows
score now. However, it's only available for helm-M-x.

The sources are cached, which prevents them from reflecting the updated
sorter value. @thierry will have to have a look at what to do about this.

helm-M-x is the only source that builds dynamically, but it's too dynamic
... :) It breaks turning on the mode due to dynamically binding to
helm-fuzzy-sort-fn

Another thing is that, if I insert a space, an error occurs:

Error running timer: (wrong-type-argument number-or-marker-p nil) [13
times]

Fixed here: https://github.com/lewang/helm-flx/blob/master/helm-flx.el

Le

Le

[thierryvolpiatto]

Aggree closed.

[ghost]

Not clear what came of this thread, but I've been using the snippet from https://github.com/emacs-helm/helm/issues/145#issuecomment-69478087 for a while now and find it works better then helm's built in matching. One small fix is required (at least in recent helm) because nil is not a legal score (mentioned here, and #1197):

  (defun helm-score-candidate-for-pattern (candidate pattern)
    (or (car (flx-score candidate pattern helm-flx-cache)) 0))

[PythonNut]

@AlKapote I'm now using this snippet:

(defun my/helm-fuzzy-matching-sort-fn (candidates _source &optional use-real)
  (require 'flx)
  (if (string= helm-pattern "")
      candidates
    (mapcar #'car
            (sort (mapcar
                   (lambda (cand)
                     (cons cand (or
                                 (car (flx-score
                                       (if (consp cand)
                                           (if use-real
                                               (cdr cand)
                                             (car cand))
                                         cand)
                                       helm-pattern helm-flx-cache))
                                 0)))
                   candidates)
                  (lambda (s1 s2)
                    (> (cdr s1)
                       (cdr s2)))))))

(defun my/helm-fuzzy-highlight-match (candidate)
  (require 'flx)
  (let* ((pair (and (consp candidate) candidate))
         (display (if pair (car pair) candidate))
         (real (cdr pair)))
    (with-temp-buffer
      (insert display)
      (goto-char (point-min))
      (if (string-match-p " " helm-pattern)
          (dolist (p (split-string helm-pattern))
            (when (search-forward p nil t)
              (add-text-properties
               (match-beginning 0) (match-end 0) '(face helm-match))))
        (dolist (index (cdr (flx-score
                             (substring-no-properties display)
                             helm-pattern helm-flx-cache)))
          (with-demoted-errors
              (add-text-properties
               (1+ index) (+ 2 index) '(face helm-match)))))
      (setq display (buffer-string)))
    (if real (cons display real) display)))

(setq helm-fuzzy-sort-fn #'my/helm-fuzzy-matching-sort-fn
      helm-fuzzy-matching-highlight-fn #'my/helm-fuzzy-highlight-match)

(require 'flx)
(setq helm-flx-cache (flx-make-string-cache #'flx-get-heatmap-file))

Performance is somewhat improved, and some bugs like the one you've mentioned have been ironed out.

Oh, and it's like 90% less hacky.

Would anyone like to see this made into a package?

[tuhdo]

@PythonNut I think yes: https://www.reddit.com/r/emacs/comments/3o8aye/ido2helm_migration_guide_for_people_who_like_the/

[priyadarshan]

Would anyone like to see this made into a package?

:+1: I would like that very much.

[halbtuerke]

Would anyone like to see this made into a package?

:+1:

[clemera]

@PythonNut

Hi,

Yes, that would be great :)

Thanks for your work on this!

I have some problems with the new version (look at the highlighting of save-persistent-scratch):

Old:

New:

[PythonNut]

@hatschipuh Strange... that doesn't reproduce for me.

With that said, you may need to run my code very early for it to take effect. In some cases, the code doesn't work if it's run _after_ the sources are initialized.

[clemera]

@PythonNut Earlier loading didn't fix it, probably I messed around with my config to much. I will try to find the cause.

[ghost]

Thanks @PythonNut I stuck it in my .emacs and it works fine. This is a long thread and hard to follow. Here is a summary for late arrivals:

• In 2012 helm didn't have fuzzy matching.
• @thierryvolpiatto added native fuzzy matching to helm for most things, controllable
  by variables called helm-<foo>-fuzzy-match. These include:

   helm-recentf-fuzzy-match 
   helm-mode-fuzzy-match  <- for describe-* (and any completing-read helm overrides?)
   helm-apropos-fuzzy-match  
   helm-buffers-fuzzy-matching 
   helm-imenu-fuzzy-match 
   helm-locate-fuzzy-match 
   helm-semantic-fuzzy-match 
... (use `describe-variable`)

• lewang's flx package uses a different algorithm for doing fuzzy matching and supports ido but not helm. It is allegedly slower (hasn't been a problem from me), but its heuristics for ranking results are a better match to what people expect, so the top results are what you were looking for more often.
• @PythonNut provided a snippet which uses a hybrid approach: it uses helm fuzzy matching to reduce the list of candidates to a list of matches and then uses flx to rank those from best to worst. This works extremely well.
• Because flx only sees the matches helm passes it you may not see the best result unless ramp up helm-candidate-number-limit. If helm finds more matches (no matter how they they score wrt each other) it ignores any remaining candidates. So increase helm-candidate-number-limit as far as is acceptable to you in terms of lag and your matches will get better. Or simply use longer queries so there are fewer matches and flx can find the best ones among them.

I'm using the helm fuzzy+flx for a while now and even with helm-candidate-number-limit set at 800, everything is fast enough for me. The two things I care about most, M-x and describe-<foo> supported, but so is all the rest.

After setting the options right, fuzzy matching with helm is a solved problem. I think flx could be sped up further, but it really hasn't been a problem. So @PythonNut - make it a package!

[PythonNut]

Ah, it's too late. I already did. :P

You guys can look for helm-flx on MELPA.

[Silex]

@PythonNut: thanks for mentionning, I missed it. I almost never use helm but I might start doing it :)

[priyadarshan]

Here is a summary for late arrivals

That is a great summary, it would be nice if it would be available on the helm-flx documentation somewhere. Thank you @AlKapote, and thank you @PythonNut for helm-flx!

[PythonNut]

That is a great summary, it would be nice if it would be available on the helm-flx documentation somewhere.

Sure, I can add it to the wiki when I find some time.

[ghost]

Let's get helm-adaptive working together with helm-flx... #1228

[EphramPerdition]

Helm didn't merge #1238 so I improved it even further and created a new package: helm-fuzzier.
It works together with helm-flx to make Helm fuzzy matching finally behave as I want it to.

Please test and report bugs, so I can put it on MELPA soon. Now on MELPA.

[bling]

awesome work guys. here's a tiny contribution from me. if you're the type that wants fuzzy turned on for everything, but don't want to declare t for all the various variables for each source you can use the following snippet:

(defun my-helm-make-source (f &rest args)
  (nconc args '(:fuzzy-match t))
  (apply f args))
(advice-add 'helm-make-source :around 'my-helm-make-source)

[Profpatsch]

Thanks for the summary @ghost!

[zw963]

I have a test about helm-flx and default fuzzy in helm-M-x. (that the main part i still use ido/smex)

What i want is to run many frequently used command with M-x + two or three key press.

e.g. I want to run helm-locate.

if not use helm-flx, when i press M-x hl, will get:

highlight-symbol
highlight-symbol-mode
helm-ag
helm-mu
helm-M-x
helm-info
helm-imenu
helm-locate # here is what i want ...

if used with helm-flx, helm-locate is always the first.

I think helm-flx result is better for me.

[zw963]

I still not give up ido completely. at least for me, I consider ido-find-file and smex
is not same equivalent in helm.

for ido-find-file, two functions seem like missing in helm. (Or anyone tell me how to enable
in helm?)

• When run ido-find-file, will guess current symbol if is a exist file in default directory.(and subdirectory)
  if found, ido will auto complete file name, else just keep the normal behavior.
  we can enable this with: (setq ido-use-filename-at-point 'guess)
  When press C-x C-f, ido will use (ffap-guesser) to find out the exist file.
• When run ido-find-file, If string not matched any file in current directory, will search
  subdirectory recursively with some tools like find/locate to try to find out correct
  file name. e.g. current-directory is /foo/, exit a file in /foo/bar/example.el
  when open ido-find-file in /foo/, input exam, waiting half a seconds, ido will
  auto complete the correct file name: /foo/bar/example.el which exist in subdirectory.

Anyone could tell me those two function is if implemented in current helm? (2016/04)
@thierryvolpiatto, Thanks.

for smex, it could auto recorder the last command I used. when I M-x again with same
hotkey, it score the most used command to the first position. this useful for me.

[priyadarshan]

@zw963 Did you try the tutorial at https://github.com/hatschipuh/helm-better-defaults? It solved all my issues with helm perfectly. I specially like the smex-helm integration.

[zw963]

@priyadarshan , I saw helm-better-defaults, and I star this package too. :)

helm-smex is cool, but i don't use it, because smex+flx perfect work well
for me current, I still not try this.

But, I don't found any solution about the first problem?

[priyadarshan]

No, I have no solution for a helm equivalent to ido-find-file usage as you mentioned. I miss that functionality from ido, I wish it would be available in helm-find-file too.

But, since this ticket is closed, perhaps a new one could be opened, referring to this one?

[zw963]

But, since this ticket is closed, perhaps a new one could be opened, referring to this one?

I add a new issue for this. #1488