Tiddlywiki5: [IDEA] Intersection operator

[twMat]

From time to time there is need for finding the intersection between two lists. I propose an intersection operator for this.

For my latest use case, I managed to partly solve this like so:

<$set name="b" filter="""[enlist{!!list-b}split[ ]join[|]addprefix[(?=]addsuffix[)]]""" >
<$list filter="""[enlist{!!list-a}regexp<b>]""" >

</$list>
</$set>

i.e the regexp checks every item in "list-a" against "list-b" which is in this form: (?=Item1|Item2|Item3|...). Unfortunately this doesn't handle spaced titles sufficiently nor forbidden characers.

If the correct regexp is identified, I'd think it is fairly straightforward to convert this into a single operator that takes a list as input and another list as parameter and spits out the intersection.

[joshuafontany]

I may be overthinking things, and in fact I have done this exact thing in Javascript (combining a bunch of terms with pipes '|' to make a regex)....but in TW filter notation doesn't this do the same thing?

{{{ <<setA>> +<<setB>> }}}

https://tiddlywiki.com/#Filter%20Expression ?

[twMat]

@joshuafontany , maybe I'm doing something wrong but this does not work

I have these lists stored in fields

!!list-a : one [[two two]] three four five
!!list-b : uno one dos [[two two]] tres

Both of the following only gives output identical to the second list:

{{{ [enlist{!!list-a}] +[enlist{!!list-b}] }}}

<$vars a={{!!list-a}} b={{!!list-b}}>{{{ [enlist<a>] +[enlist<b>] }}}</$list>

[Jermolene]

Good idea @twMat. @inmysocks did some work on this in #3003 but now I wonder if it might not be more useful to implement intersection as another filter run prefix (ie alongside +,-,~ and =). For example we could use ^ so your example could be done as:

[enlist{!!list-a}] ^[enlist{!!list-b}]

[pmario]

I was thinking in a similar way but would have used # as a symbol, because it has "intersections"

[twMat]

@Jermolene

[...] as another filter run prefix

That probably makes sense.

Side note:
Referring to @joshuafontany 's post here above about using +, the docs do state that:

|+run | intersection of sets | ... AND run |

That description is probably not quite correct because A∩∅=∅ meaning that intersection with an empty set is empty (ref) but +run doesn't behave like this. I always thought of + as an kind of "forced match" rather than an intersection.

[Jermolene]

The reasoning for using ^ was that it's the closest plain ASCII symbol that's shaped like ⋂. But as @twMat notes it's probably too late to consistently use established notation.

[clutterstack]

Coming here via @inmysocks's PR. I have @tobibeer's contains filter operator plugin installed in one of my wikis and a personal plugin depends on it (matching a list of tags exactly, any, all, etc, using buttons). I know it's overloaded, but wow, is it easy to just give that a list and a parameter. I think this intersection operator would be a useful addition to the core.

[rmunn]

Good idea @twMat. @inmysocks did some work on this in #3003 but now I wonder if it might not be more useful to implement intersection as another filter run prefix (ie alongside +,-,~ and =). For example we could use ^ so your example could be done as:

[enlist{!!list-a}] ^[enlist{!!list-b}]

Now that https://github.com/Jermolene/TiddlyWiki5/pull/4915 is merged, it might be better to possible to add a new filter run prefix called :intersection, rather than adding another character like ^. It would also be good to have some documentation to explain how :intersection differs from :and (the + character), because as you'll see in #4915, :intersection was the originally-suggested name for the + prefix before we settled on the name :and instead.

But I'm not actually persuaded yet that this makes sense as a prefix. I'd much rather see @twMat's example implemented like this:

{{{ [enlist{!!list-a}] +[intersection{!!list-b}] }}}

than like this:

{{{ [enlist{!!list-a}] ^[enlist{!!list-b}] }}}

Doing this as a prefix adds complexity that I think is unnecessary. Specifically, now there's + and ^ which have very similar, but not quite the same, behavior. Which one should the user select for his particular use case? Who knows? Figuring that out is an added cognitive burden. Whereas an intersection operator doesn't add cognitive burden to figuring out the difference between those two prefixes.

[Jermolene]

Thanks @rmunn I think the most compelling part of @saqimtiaz's proposal for a :filter run prefix is that it obviates the need for placing the subfilter in a variable. That's not a problem here, but it might point to criteria that we could use to determine whether a function should a run prefix or a filter operator.

[saqimtiaz]

Two thoughts here:

a) A filter run prefix wouldn't be without value when the list is generated via a sequence of filter steps and thus cannot be easily passed to list/enlist without extra set widgets.

[[field Operator]tags[]] :intersection[[after Operator]tags[]]

vs

<$set name="a" filter="[[field Operator]tags[]]" >
{{{ [[after Operator]tags[]] +[intersection<a>] }}}
</$set>

b) we are missing a filter operator that allows a string or variable to be interpreted as a title list. Having such an operator would be useful in its own rights as title lists are common in TiddlyWiki. Such an operator would make the argument for an intersection filter run prefix stronger.

For example consider [{!!parent}get[children]]
where children is a field holding a title list. Currently this requires two steps to operate on the list of children, first getting children and then enlist<children>

<$vars children={{{ [{!!parent}get[children]] }}}/>
<$list filter="[enlist<children>]">
<!-- do something -->
</$list>
</$vars>

I've considered extending enlist[] to operate on its input when no operand is present, but that might be confusing for users.
The other option could be a string operator like aslist[]
The above code would then be:

[{!!parent}get[children]aslist[]]

Coming back to intersections, then it could be useful to be able to do:
[[A]get[children]aslist[]] :intersection[[B]get[children]aslist[]]

[rmunn]

For example consider [{!!parent}get[children]] where children is a field holding a title list.

Yes, that would have come in handy for me in various parts of my current TiddlyWiki project.

And I'm starting to see the need for an :intersection field run prefix, too, and how to explain the difference between :intersection and :and. The way :and works is that the output from run 1 is passed as input into run 2. But if run 2 wants to use an input-constructing operator anywhere, such as [[B]get[children]], then the output of run 1 gets discarded. Whereas :intersection will take the input from run 1 and store it away, then provide run 2 with a brand-new input consisting of all tiddler titles (i.e., same input as if run 2 was specified without a prefix). When run 2 finishes, the outputs of runs 1 and 2 are compared, and only titles appearing in both runs are kept in the output.

So I think I'm now in favor of adding a new :intersection prefix, and changing the docs for the + prefix so that it does not talk about set intersection, but rather about adding more conditions. I don't know whether it should have a single-character prefix or not, but if it does, I think I'd prefer & rather than ^, because if we later decide to add an :xor prefix then ^ is the natural choice for :xor (at least, ^ seems like the natural choice to a programmer like me, because that's the XOR operator in the programming languages I'm aware of).