Oni: Add refine option to quick open

[CrossR]

Splitting this out from #1281,

Some fuzzy-finders have a 'refine' approach - where you could type main.tex, realize there's a bunch of them, but realize you're looking for the one in research folder, so you could then refine the result by typing space + research (so the filter text would be main.tex research). This would turn the filter into a two-pass filter - first, filter all the results with main.tex, then filter all of those results with research. In addition, if we sorted the search results by length, this might be a cheap way to make the fuzzy finder significantly more useful.

I'd certainly find something like this useful, so may have a look into adding it, if it isn't picked up before I get some free time.

[badosu]

@CrossR Where is the content research placed? If it's on the folder you can use the regex strategy and type researchmain.tex.

We could also adjust the regex strategy so that it works similar to other fuzzy finders where if you separate the words with space it tries to match both, e.g. main.text research as you intended. This is the ideal behaviour for the regex strategy for me.

[CrossR]

We could also adjust the regex strategy so that it works similar to other fuzzy finders where if you separate the words with space it tries to match both, e.g. main.text research as you intended. This is the ideal behaviour for the regex strategy for me.

Yeah I think having the ability to stick a space between is nice, just because I think a lot of people think along the lines of "I need file X" then realise they get back 20+ results so go "Ooops I mean I know its in this folder", but with the current system we can't really easily add that new folder on, we'd have to delete the query and start again. Being able to just react and type the folder on with a space between makes it a lot easier.

Does it make sense to do it as the original post describes?
I.e. can you see any issue with doing the second terms search on only the results of the first? Or should it be a single search with both terms? Not sure if it changes anything really.

[badosu]

Does it make sense to do it as the original post describes?

Yes, I used it this way with other fuzzy finders

I.e. can you see any issue with doing the second terms search on only the results of the first? Or should it be a single search with both terms? Not sure if it changes anything really.

Refining one on the other and using both terms is the same thing if both are exclusive. If one is exclusive and the other is just to prioritize the results then it's different. I prefer the first approach.

[CrossR]

Makes sense and looks simpler to do it that way!

Slightly off topic for this issue but I'm looking at the regex stuff now, and currently we convert every term using the following:

const filterRegExp = new RegExp(".*" + currentTerm.split("").join(".*") + ".*")

Which would turn my example of research into .*r.*e.*s.*e.*a.*r.*c.*h.*, where I would have expected .*research.* ie anything with the word research in, rather than those letters appearing anywhere in the term in order, but with anything between.

Does it make sense for that to be "fixed"? (I use quotes since that could be the desired behaviour, just seems odd to me). Kinda feels like it may be the root of some of the issues you and others have been having.

EDIT: I'm realising that is for the ability to fuzzy search and the highlighting of the terms in the QuickFind is confusing me. Should maybe make a specific ticket for sorting that too, since it feels like it highlights too much ie every instance of a letter when I would expect only one or between the biggest possible search term, not every instance.

Other than that...this was basically just splitting the term on a space and stick a for loop around the current logic. If its that easy for the non-regex (or it makes sense to add it there), I can do it there too. Need to check performance issues of that but I wouldn't expect any changes.

[badosu]

Tracking additional refinement proposals by @CrossR at https://github.com/onivim/oni/issues/1266#issuecomment-368906703 :

I've had a look into this in other editors, and I think there is a few things we can potentially do, put I'm not fully sure of their cost.

By the looks of VSCode they use a big scoring algorithm to apply scores to the results, then that is used for the sorting : https://github.com/Microsoft/vscode/blob/master/src/vs/base/parts/quickopen/common/quickOpenScorer.ts
Looks to basically just give higher scores depending on if certain criteria are met.

We could potentially do something like that, but it could also just slow down the search or cause more issues when we have to add edge cases after edge case. Though, there were still some issues complaining it wasn't as good as FZF still.

That said, it would improve the search further, since currently I don't think we do anything like this.

[bryphe]

Nice! I wonder if it would be doable to factor the VSCode implementation out to a common module? Would be cool to have a vscode-quickopen-scorer module.

This is what @DeltaEvo has done with the language servers, and I've also leveraged some of the code for snippet parsing:
https://github.com/onivim/vscode-snippet-parser

It looks well-designed, relatively self-contained, and modularized. If there was that module, we could do something like:
import { score } from "vscode-quickopen-scorer" and use it to sort our matches.

EDIT: I'm realising that is for the ability to fuzzy search and the highlighting of the terms in the QuickFind is confusing me. Should maybe make a specific ticket for sorting that too, since it feels like it highlights too much ie every instance of a letter when I would expect only one or between the biggest possible search term, not every instance.

Ya, the highlights are actually totally separate and may not be in sync. Would make sense to revisit this. It looks like the score implementation from VSCode gives match positions, which could be used to highlight.

Other than that...this was basically just splitting the term on a space and stick a for loop around the current logic. If its that easy for the non-regex (or it makes sense to add it there), I can do it there too. Need to check performance issues of that but I wouldn't expect any changes.

I think this would be great to have too! Lots of cases I run into where I start filtering, and then realize I need to refine at a higher level - having this capability would help, and I believe be orthogonal to scoring improvements.

[badosu]

@bryphe A simpler way to achieve this would be by just adding the word separated by space as a new term in the regex, no performance penalty and pretty predictable. I would love that as an alternative to the scorer.

[CrossR]

@badosu from my understanding, the scorer would work along side any filtering, with it just being used to sort the results from the filter. The refinement is useful, but I still sometimes get weird results.

I'm also not 100% on the regex strategy or if I should add multiple types. Ie the way I've done it currently I do each as a separate term, such that the second term is only searching the result of the first and so on. This feels pretty natural to me since I whittle down the set I can currently see.

Would you also want a mode that you just add on terms with a space and they are appended to the first search term? Or am I misunderstanding?

A simpler way to achieve this would be by just adding the word separated by space as a new term in the regex, no performance penalty and pretty predictable.

Regardless I'll stick up a WIP PR for what I have now, and have a look. It feels a lot better now, and I think with some more intelligent sorting of the results we should be great! Though that said adding this with multiple search terms could make scoring a whole bunch more awkward, at least if we attempt to take the VSCode bit as is.

[badosu]

I'm also not 100% on the regex strategy or if I should add multiple types. Ie the way I've done it currently I do each as a separate term, such that the second term is only searching the result of the first and so on.

Yeah, it's the same thing. As I said, if the terms are exclusive, searching terms separated by space (e.g. 'index controller' => /index|controller/) is the same thing as searching for one then the other. I am not sure, but I would imagine a single regex pass to be more performant.

For scoring I usually prefer to have it simple and predictable, e.g. frequency and order of the terms, if necessary at all (after all, the refine option idea is not to have to rely on the scorer too much).

Unless you're managing a really huge and complex file tree I don't see the overhead of having the scorer as an advantage.

[CrossR]

The filtering aspect of this has been addressed, but there is still the scoring aspect to be considered here.

[CrossR]

I've moved this out into its own ticket to make it a bit more focused.