Julia: zip-like version of skipmissing for multiple iterators?

[stevengj]

As suggested on discourse, a method like

skipmissing(itr, itrs...) = skipmissing(mapfoldl(ismissing, |, tuple) ? missing : tuple for tuple in zip(itr, itrs...))

might be useful.

(Not using any here to exploit the optimized mapfoldl for tuples from #30471.)

[stevengj]

One incongruity with the above API is that skipmissing(itr) produces a sequence of elements whereas skipmissing(itr1, itr2) produces a sequence of tuples. Makes me think that the tuple version should be called something different, maybe zipmissing.

[pdeffebach]

Thanks for making this issue!

I think the behavior I'm looking for is actually OR, so you get complete cases for everything.

skipmissing(itr, itrs...) = skipmissing(mapfoldl(ismissing, |, tuple) ? missing : tuple for tuple in zip(itr, itrs...))

definitely separate objects rather than an iterator of pairs is nice. The use-case I imagine is when you have arguments that need to be synced by index and inputted into functions.

[tkf]

FYI @galenlynch implemented skipoftype in #30549 (thx!). I haven't tried it, but I think it let us write skipoftype(Tuple{Vararg{Missing}}, zip(itrs...)) and should give us correct eltype for free.

[nalimilan]

zipmissing sounds like the appropriate name. But is this really what people need? For example, if you want to compute the correlation between two vectors with missing values, cor(skipmissing(zipmissing(x, y))...) won't work. What's needed is a function returning a tuple of iterators which return entries for complete cases in x and y.

[piever]

What's needed is a function returning a tuple of iterators which return entries for complete cases in x and y.

Maybe skipmissing(a, b) could return two iterators, one for a and one for b. Then one would simply use zip(skipmissing(a, b)...) to have the iterator of tuples, or we could define zipmissing(itr...) = zip(skipmissing(itr...)...).

OTOH the tuple iterator as implemented in the first post seems more natural as it has to traverse the data only once, so an alternative would be to go with zipmissing and for the cor use case do something like unzip(zipmissing(a, b)) (provided unzip gets implemented).

[nalimilan]

Maybe skipmissing(a, b) could return two iterators, one for a and one for b. Then one would simply use zip(skipmissing(a, b)...) to have the iterator of tuples, or we could define zipmissing(itr...) = zip(skipmissing(itr...)...).

The problem @stevengj highlighted is that skipmissing(a) currently returns an iterator, but skipmissing(a, b) would have to return a tuple of iterators, which would be inconsistent with the former method (which should return a single-iterator tuple for consistency).

OTOH the tuple iterator as implemented in the first post seems more natural as it has to traverse the data only once, so an alternative would be to go with zipmissing and for the cor use case do something like unzip(zipmissing(a, b)) (provided unzip gets implemented).

If unzip(zipmissing(a, b)) worked without collecting the iterator first (not sure how that could be done), I don't see why a function wouldn't be able to do that directly, with a single pass. Anyway in the end everything depends on the order in which you go over each iterator.

[pdeffebach]

I'm curious if this idea makes any sense.

Now that we can index a skipmissing object, we could define zip(x::Skipmissing, y::Vector) to only include the elements of y that where x is not missing.

Then we could mandate that functions that require arrays of equal length to zip them, guaranteeing two matching iterators.

[nalimilan]

That would break the current behavior of zip, as this pattern already works. Also it would be weird not to follow the general behavior implemented for other iterables.

[pdeffebach]

Based on the above conversation, a good way forward would be a multiskipmissing function that returns a tuple of iterators. It will not be called zipmissing because zip returns an iterator of tuples, not a tuple of iterators.

Should I submit a PR to Missings.jl for this?

[pdeffebach]

I've implemented a multiskipmissing iterator in Missings.jl here.