Tiddlywiki5: [IDEA] :sort prefix and extensible filter run prefixes

[saqimtiaz]

Just as the :filter filter run prefix provides an inline alternative to the filter operator, it would be desirable to have a :sort filter run prefix where the list is sorted by comparing the results of applying the filter run to each list item. This would be an inline alternative to the sortsub operator.

[tag[HelloThere]] :sort[get[text]length[]]

instead of:

<$vars compare-by-title-length="[length[]]">
[tag[HelloThere]sortsub:number<compare-by-title-length>]
</$vars>

However, we currently support multiple types of sorting: string, integer, number, date and version

So either we would need to add 5 filter run prefixes, one for each of the sort types. :sortstring, :sortinteger etc...

Or we could consider supporting suffixes for filter runs. Example : [tag[HelloThere]] :sort:string[........] or [tag[HelloThere]] :sort:number[........]
While this second approach is a lot more flexible and powerful, the syntax might prove not so intuitive for new users.

Is there a third alternative that I am not thinking of?

I am interested in implementing this - even if only as a plugin - so I would appreciate any feedback or ideas.

[rmunn]

Not only that, there's also sortan for alphanumeric sorting, there's reverse sorting (high to low instead of low to high)... it could get very complicated very fast.

I strongly dislike the idea of having suffixes for filter run prefixes (:sort:number and so on). Definite comprehension issues for new users. If there's enough benefit to add a :sort prefix, then it's worth adding all of the various sorting variants as their own prefix (:sortn, :sortan, :revsort and so on). But I don't think there is enough benefit. I think <$vars sort-by="[some-filter[]]"> followed by sortsub<some-filter> is simple enough that the prefix version doesn't gain all that much, and when you take into account the dozen different sorting prefixes we would need to account for all possibilities, I think the complexity cost is more than what we'd gain.

[saqimtiaz]

@rmunn the feeling is mutual about suffixes for filter runs and comprehension. The sort options I listed are the ones supported by the sortsub operator rather than the sort operator.

As for the utility of the sort prefix, I find filters written inline far easier to read, comprehend and maintain.

when you take into account the dozen different sorting prefixes we would need to account for all possibilities, I think the complexity cost is more than what we'd gain.

Yes that is the issue and that is why this is an issue for discussion rather than a PR. I realize this might not be appropriate for the core and am happy writing this as a plugin for myself. The key advantage gained with the refactoring to support the filterrunprefix modules is that plugins can now add filter run prefixes.

However before I started writing code, I wanted to talk this out and see if there was a third alternative I was missing as neither proposed solution is entirely satisfactory.

[rmunn]

So are you going to end up creating a prefix version of every filter that takes a subfilter argument, like reduce, so that it can be written inline?

As for a third alternative, I'm not seeing one. I think you'll just have to implement a separate filter prefix for every possible variant, finding appropriate names. (Though implementation can just have one implementing function and all the others are simple wrappers around it.) A :revsort prefix could possibly be written inline as just :sort[some-filter] +[reverse[]], so you might be able to get away with not writing all the rev variations... but sorting and then reversing adds an extra O(N) step after the sort is finished, when that usually would have been quite easy to do inside the sort by just reversing the comparison function. So if you're going to write all those prefixes, adding a rev variant of each is probably worth it for the performance gains.

[saqimtiaz]

@rmunn unless I can think of a better implementation, this will likely be a plugin for personal use and as such I would only write the prefixes that I find myself needing. With the way that I use TW that is pretty easy to plan in advance.

As you say the sheer number of prefixes required would make this a non-starter as a public plugin.

Part of the reason I wanted to bring this up now before 5.1.23 went live was in case I had missed a trick with the filter run prefix implementation, and maybe someone else might spot a more flexible and elegant solution.

[Jermolene]

I like the idea of making things like "reduce" available in inline form. Having a profusion of filter prefixes doesn't feel bad because we can already be confident that the vast majority of users need never learn about their existence.

[saqimtiaz]

@Jermolene I'll do some prototyping post 5.1.23 and we can then discuss the merits while looking at some code.

[rmunn]

After thinking about this some more while writing #5246, I've come to the conclusion that filter run prefix suffixes, as ugly as that phrase is to type, might actually be necessary due to the fact that =1 =2 =3 :reduce[multiply<accumulator>] produces 0. There are two ways to make that produce the value you'd actually expect. One is to write it as =1 =2 =3 :reduce[<index>compare:number:gt[0]then<currentTiddler>multiply<accumulator>else<currentTiddler>]. The other is to allow a suffix (or second parameter) to filter run operators, so that you could write either =1 =2 =3 :reduce:1[multiply<accumulator>] or possibly as =1 =2 =3 :reduce[multiply<accumulator>],[1]. And I'm leaning more towards that being an option. Which would then allow for :sort:number, :sort:date, and so on.

[Jermolene]

Notwithstanding my earlier comment, I agree that adding filter run prefix suffixes makes a lot of sense.

With reference to the clumsy nomenclature, perhaps this is our last chance to rename "filterrunprefix" to avoid the overloading of the term "prefix". We might use:

• filterrunhead
• filterrunmethod
• filterruntype

Maybe the word "run" is redundant so we could have:

• filterhead
• filtermethod
• filtertype

[saqimtiaz]

@Jermolene I would suggest filterruntype.

I don't think we should drop the word "run" as I could potentially see filtertype clashing with some hypothetical type attribute of filter operators in the future. Also our documentation already describes the sequences of filter steps as a filter run, so filter run type is more intuitive.

[Jermolene]

I don't think we should drop the word "run" as I could potentially see filtertype clashing with some hypothetical type attribute of filter operators in the future.

That's rather my concern with "type": that it's too generic a term, and already littered through our code.

Also our documentation already describes the sequences of filter steps as a filter run, so filter run type is more intuitive.

Very good point.

Also to add:

• filterrunjoiner
• filterrunmodifier

[saqimtiaz]

filterrunmodifier and filterrunmodifier suffix?

Do note that we do already use the term filter run prefix in docs for some time, but a change might be warranted here to avoid more confusion down the road.

[Jermolene]

Do note that we do already use the term filter run prefix in docs for some time, but a change might be warranted here to avoid more confusion down the road.

Tsk I had forgotten, and thought we'd introduced the term with v5.1.23. I think the naming boat has sailed then, we'll have to live with it.