Fzf: Searching for value containing space by escaping space

[netei]

When using fzf-history command with <Ctrl>-R, I sometimes search for small commands like og ls (a custom command I have created).

However, since the "words" are very short, Typing og and ls won't be filtering the history sufficiently, and some non-relevant commands are showing up. I then have to manually move up and down in fzf to select the command.

It would be great to be able to include a space in a search query, by for example escaping the space, eg writing :

og\ ls

which would take "og ls" as one single search query, instead of searching for og and ls separately`

I have --exact in my FZF_DEFAULT_OPTS, but I think that feature would be useful whatever your config is

[junegunn]

Then the question is how we are going to match literal \. And I really don't want to introduce something like \\ for that. I'm okay with fzf not covering 100% of cases if it makes it simpler.

So, the options are:

• CTRL-R to toggle sort. Does it help?
• --no-extended (or +x) in $FZF_DEFAULT_OPTS. I know it's not ideal, even we make it possible to apply it only on CTRL-R
• --no-exact and ogls instead of og ls (and also with CTRL-R)

[netei]

• CTRL-R doesnt help that much because the words I search for are so short that the matches are not
• I use extended mode a lot, so I wouldn't change it in the config, maybe it would be great to be able to change it at runtime, but I already know that you don't want to introduce that for now
• I've switched to exact mode because it is what works best for me when I have an input with long characters (my history) and because they were to many bad matches without it

The problem could also occur with all other characters that have specific meaning (' , ^, $ , |)

[junegunn]

The problem could also occur with all other characters that have specific meaning (' , ^, $ , |)

True, but the issues with those chars are limited as they are interpreted as such only when they appear at certain positions, i.e. ' and ^ at the beginning of the search term, $ at the end, and | should appear alone with surrounding spaces.

Dynamic switching of search mode is somewhat tricky as it involves

• Cache invalidation, one of the hardest problems in computer science :)
• Introducing visual cues of the current mode
  
  
  
  • like toggle-sort appending /S to the info line
  
  

[davidsu]

I also miss a lot being able to have a space as part of the query to search for. I do some juggling around it but it's not ideal. I would be thrilled to see this implemented. The place that I miss it most is searching my .vim files. I have my vimrc split through multiple files and some code that looks like the following

function! Vimrc(...)
  let query = get(a:000, 0, '^')
  if !len(query)
    let query = '^'
  endif
  call fzf#vim#ag_raw(
        \'--ignore ''autoload''  ' .
        \'--ignore ''plugged'' '.
        \query, 
        \fzf#vim#with_preview({'dir': '$DOTFILES/config/nvim/', 'up': '100%'}, 'up:40%', 'ctrl-g'))
endfunction
command! -nargs=? Vimrc call Vimrc(<q-args>)
map \v  :Vimrc<cr>

many times I wish to look for some map l approximately. Not to execute it, to open/edit or just see the command that it's linked to. this map l can take many forms though map <expr>l, map <buffer>l, map <silent><buffer>l... Unfortunately map is a rather common string/substring(besides being the name of a commonly used viml function). If I could only get a better ranking for those cases that actually have a space between the 'map' word and the 'l' character this would make things a ton better.
I'm just writing to let you know really, kind of 'upvoting' this issue. I am absolutely in love for you work and I can never appreciate you enough for having made my terminal/vim so much more pleasurable .The inability to match the space character is frustrating at times. I wish I knew go and could suggest something with actual code.

[nkgm]

Fully on board with @sudavid4. Short search terms will generate too much noise, rendering fzf's superfast fuzzy search useless. What's the point of maintaining simplicity when you're forcing the user to sift through 1000 results for map l? Maybe some simplicity has to be sacrificed after all! 😸

[junegunn]

My suggestion is that you type in mapl instead of map l.

1. You type less
2. fzf will perform faster as it's a single query instead of two (map and l)
3. It will match less number of items. It won't match l something map.
4. fzf will produce better result and most likely give you exactly what you want as the top match thanks to its ranking algorithm.

Or you can always disable extend-search mode (+x or --no-extended).

[nkgm]

@junegunn let me give a solid example. Assume a search for map p. Your suggestion is to use the search term mapp. In my dotfiles, this produces 18968 results the top of which contain mapping, mapper, mapped etc.

1. 2 extra keystrokes as in map\ p or likewise is no trouble at all if it means I don't get 18968 results back.
2. I really don't mind if it takes a bit longer. It would take me much longer sifting through the 18968 results.
3. 18968
4. Indeed the ranking algorithm will often bring the desired match near to or at the top, but in this specific case map p wasn't even listed within the first 100 results.

[junegunn]

So in this case you have absolutely no idea what would come after map p that you can't type in a few more characters to further narrow down the list? Does this happen a lot to you? I mean it kind of seems like a contrived example. Have you considered disabling --extended?

[junegunn]

Having said that, there are cases when I also wish that it's possible to match spaces, that is when I search my command history (CTRL-R) because shell commands have lots of relevant spaces. For example, git co in --extended search mode may not be precise enough for finding "git clone" commands. On the other hand, gitco may return irrelevant matches with higher scores.

However, I imagine escaping all the spaces in a command can be tedious, i.e. git\ clone\ https://..., and you can't copy & paste a command on fzf because it's not escaped. One option would be to allow customizing term separator. --query-separtor "," for splitting the query with commas, and --query-separator=" +" will split the query into terms only when there are two or more consecutive spaces.

[nkgm]

However, I imagine escaping all the spaces in a command can be tedious, i.e. git\ clone\ https://..., and you can't copy & paste a command on fzf because it's not escaped.

I'm pretty sure in most cases you will only need 1 or 2 spaces. In your example, git\ clone is the crucial term to get you in the ballpark of relevant results, and then you can just type the partial repo name.

--query-separtor "," for splitting the query with commas, and --query-separator=" +" will split the query into terms only when there are two or more consecutive spaces.

But what happens when you realize you need to squeeze in a comma in your search terms? I know I would hate having to start over picking a different --query-separator. People have years of muscle memory typing \to escape metacharacters in shells. Why not stick to the well-established convention?

Speaking of metacharacters, fzf uses space as the query separator and then there's '^!$. These characters can't be used literally in the beginning ('^!) or end ($) of a search term. Wouldn't it be super consistent to be able to escape these as well?

[davidsu]

Hi,
I am thrilled to see that you are considering a way to allow this.
Having said that I would like to way that I strongly agree with @nkgm

People have years of muscle memory typing \to escape metacharacters in shells. Why not stick to the well-established convention? ... and then there's '^!$

It would (at times) be convenient to be able to be able to escape those special characters too. besides, I wouldn't want to permanently change the separator character, neither would I want to have to know ahead of time that I will need to search for a real space.

I imagine escaping all the spaces in a command can be tedious, i.e. git\ clone\

I wouldn't mind, it's not that often that I need to match a real space anyway

and you can't copy & paste a command on fzf because it's not escaped

you have a point but with backward-word, beginning-of-line and the like keybinding it's not too bad. But I don't see myself copy/pasting into fzf at all so I guess I have no saying in that

[maoryosef2]

Also agree with @sudavid4 and @nkgm
It makes more sense to use escape characters for this purpose than to change the query seperator

[nkgm]

@junegunn thank you so much - this is working great so far. I was kind of hoping you'd also cover the \\ case for ultra consistency but these cases should be pretty rare - for example you can't match a literal \^ as the first 2 characters of a search term, but \^\ will match a literal ^\ which is still pretty good. And I don't mean to sound ungrateful. Great stuff overall! :1st_place_medal:

One thing I've noticed messing around with it is that even though a\^ will match these characters literally, '\^ will swallow the\ and match only ^ literally. Is this intentional?

[junegunn]

for example you can't match a literal ^ as the first 2 characters of a search term

I felt it was not worth it and it could cause extra confusion as we currently allow users to write \ in non-special contexts in unescaped form. e.g. foo\\$ will become ambiguous; it can both mean foo\\ + anchor $ and foo\ + literal $. \^, \\^, and \\\^ also make my head hurt. Just declaring some patterns are not allowed can remove such ambiguity.

Is this intentional?

No, thanks for pointing it out. I'll fix that.

[nkgm]

I felt it was not worth it and it could cause extra confusion as we currently allow users to write ...

There's no need to give \\ special treatment across the board.

There's already special processing for [^\]<space>\^, [^\]<space>\', [^\]<space>\!. We only need some more special sauce for [^\]<space>\\^, [^\]<space>\\', [^\]<space>\\! and that's it! Unless I'm mistaken, there are no other "blind spots" where \ can't regain its literal value. And I don't think this will encumber the user's cognitive load in any way.

Btw there was mention of | in a previous commit message. Could you share what it's about?

[junegunn]

If we always treat \\ prefix as literal \ (instead of special casing only \\[:meta:] prefix), there's no ambiguity, though users should know that \\a (non-meta) will match \a if it appears at the beginning of a term, but \\a otherwise (i.e. \\a => \a, but b\\a => b\\a).

What's your opinion on the case I mentioned above? Interpreting foo\\$.

\| allows you to search for literal |.

[junegunn]

Well, it's a choice. We can always introduce more rules to support exceptional cases, or we can keep things simple by not trying to do everything.

[davidsu]

I don't understand, I can't seem to see this working. I've pulled and re-run ~/.fzf/install but it still matches a literal '\' when I try some\<space>word. I've checked that commit e85a8a6 is indeed there and that I'm running the ~/.fzf/bin/fzf. What am I missing?

[nkgm]

What's your opinion on the case I mentioned above? Interpreting foo\$.

That's super cool too and was going to be my initial suggestion (I did in fact type quite a lengthy response about it)! Then I decided to rewrite my response to suggest the "precision" approach only cause I gradually realized it's not much different to what is already being done and it feels slightly more elegant (speaking strictly from the user perspective - not sure how this complicates things on the implementation side). The user already knows to watch out for a leading \^. How is keeping track of a leading \\^ (much) different? Whichever one you choose, it would be a more than welcome improvement to consistency seeing that \\$ already works to similar effect.

| allows you to search for literal |.

But I thought | didn't have any special meaning in a query - why does it have to be escaped?
Latest fix working great so far btw! 👍

@sudavid4 how about 'some\<space>word?

[junegunn]

@nkgm I think you misinterpreted my comment. Please re-read my previous comments. My point is if we allow trailing \\$, it introduces inherent ambiguity. Special treatment of leading \\^ also introduces similar problem. \\^foo will match \^foo, but then we can't match literal \\^foo and we are going to write \\\^foo. But then \\^ is no longer at the beginning of the term, so we just treat it literally, matching three backslashes. To resolve the issue, we have to treat leading \\ as a single literal backslash, which affects non-meta characters as well, such as \\a. It will match one backslash, but if it comes after other characters (e.g. b\\a) it will suddenly match two backslashes.

I thought | didn't have any special meaning in a query

It's described in the man page.

@sudavid4 install script will only download the prebuilt binary. You have to build fzf from source; make install.

[nkgm]

\^foo will match ^foo, but then we can't match literal \^foo and we are going to write \^foo

Of course, how could I forget that! I did read your comments very carefully and independently reached that same conclusion last night but then took a wrong turn trying to make it as concise as possible. Here's the relevant snippet from my initial (not posted) response:

If this is a new term boundary (ie <space> but not \<space>, \ will have special meaning only before \'^! in which case it matches the literal character (yes, \\ at the start of a term always translates to \ even when that term is \\abc).

Sorry for the mixup!

It's described in the man page.

Looking great as well.

[junegunn]

The more I think about it, the less I'm convinced that fzf should provide escaping of all meta characters for the sake of completeness. I can definitely see the appeal of \<space>, but the other ones, not so much. And they have led us to this lengthy, cryptic discussion of double escaping, ambiguous patterns, etc, when in practice few users will actually need or use them. How often do we need them really? I'm leaning towards only supporting escaping of spaces.

[nkgm]

I'm actually on the fence myself. The \\$ case seems hairy. How would you go about it?

Wondering what other people have to say on the matter.

[junegunn]

I don't have a good idea. Allowing \\$ introduces ambiguity that can only be resolved if we force users to always escape backslashes which is lame, but disallowing it (\$ to always match literal $) is also problematic, since I actually can see myself wanting to find trailing \ sometimes, since they are pretty common in script files.

So what I think I'm going to do is to only allow escaping of spaces, use it for a while, and see if it's not too painful not to be able to match meta characters at special positions. (Meta characters are activated only when they are at special positions so we can still match $env, bit^wise, or bang!.)

[nkgm]

It's probably the most sensible thing for now - get a chance to dogfood. Btw these latest refinements really inspire a lot of confidence when browsing through a codebase. Now I go straight to :Ag<CR>!

[davidsu]

@junegunn thanks for the explanation, it works for me now, after running make install and using ~/.fzf/target/fzf-darwin_amd64.
About all this escaping meta thing, I agree that escaping only \ is the most sensible for now. Besides, correct me if I'm wrong, for all other meta we can disable the extended by prefixing with ', the \ is special because there wasn't any way of matching a real space.

I suppose that finding some way to "pre-filter" by means of a real regular expression would give all the power we could possibly need while keeping the simplicity we learned to love. I believe I would consider a first term started withr a real regex pre-filter. Then if I wanted a first query starting with a real r I would prepend it with a \ which would cause the r to lose it's special meaning. To keep this simple for most users this feature could be enabled only when --with-regex-pre-filter option is passed or something of the like
It's just personal view, I thought I should share it.
Once more thanks a lot for all the great work and awesome support.

[qoxxxx]

I'm updating fzf using Plug 'junegunn/fzf', { 'dir': '~/.fzf', 'do': './install --all' }, so I'm at version 0.16.11 (ecb6b23).

Will that feature come later or is it already there ?

Which would mean I don't know how to use it.
echo -e 'what am I\ndoing wrong' | fzf --extended
-> am\ I or ng wro doesn't match.

[junegunn]

@qoxxxx No official binaries have been released with the change yet. I'm planning to release a new version in a couple weeks, but you can build fzf from source now with make install.

[junegunn]

fzf 0.17.0 is out with the change.