Mlj.jl: Sparse data and abstract matrix input

[ablaom]

Discussions at MLJ meetings have turned to the problem of sparse data. Data can be observation-sparse or feature-sparse (or both). My feeling that the feature-sparse case is the more important use-case, and the more tricky to deal with. I originally thought one might handle this within the current tabular data format but I think this requires extra infra-structure that does not exist yet in Julia. Given limited resources the pragmatic thing to do would be to allow models that handle feature-sparse data to ingest the data as an abstract matrix (in addition to a dense table). When this is sparse, the performance benefits kick in.

Having "given up" on the uniform requirement of tabular data, we might just as well allow arbitrary models that currently take tabular input to ingest data in the form of matrices as well. It would be quite natural to roll this out at the same time as implementing the new optional data front-end for models. If we do allow matrices, an important design decision regards the output of models (say of transformers, or the form of the target in multi-target supervised models). I guess if we train on matrices, then matrices should be the output and similarly for tables.

Thoughts anyone?

@OkonSamuel
@tlienart

[tlienart]

I seem to have outdated information for a big chunk of this discussion but fwiw I agree with you that "feature-sparse" seems the more relevant/important use case.

Note that in my experience, the story with the likes of pandas is not ideal either. Last I checked if you do get_dummies or some function of that name to do one-hot-encoding in pandas and pass that to scikitlearn it gets "densified" along the way (I may be incorrect).

One note: a big chunk of the cases for sparse data is encoding (like OHE), given models in MLJ can ingest data then do their own stuff on the data, you could imagine they do their own encoding as well and handle the sparsity as well which may be what you're already suggesting?

[OkonSamuel]

Allowing input to models to be an instance AbstractMatrix is a good solution to the sparsity issue.(just let the model implementers worry about it). My only concern is the inconsistency in API.
We could also add something like SparseTable scitype to ScentificTypes.jl and define some method stubs that any instance of this type should support. Then let developers worry about implementing these methods for their sparse table type.