Mlj.jl: Relationship with other efforts such as StatsModels.jl

[juliohm]

First off, thanks for the initiative! We really need a pure ML framework in Julia :100:

Could you please comment on how the project relates to efforts such as StatsModels.jl? I was planning to adhere to their interface in some of my packages, and became aware of MLJ.jl today.

Also, is it correct to say that MLJ.jl is an attempt to have a "sckitit-learn" for Julia? If yes, how MLJ.jl compares with ScitkitLearn.jl?

Can you please clarify the project goals?

[vollmersj]

Dear Julio,

thank you for reaching out. We should probably have a summary of the differences at the top of the read me - since this is a common question.

n principal, I think it might have been possible to use scikitlearnbase as our "basement level" API. However to accomodate the feature enhancements we want, the MLJ user interface is necessarily very different. So, to keep the scikitlearn basement level API would have been putting a square peg in a round hole - and so more work in the end, I believe. And more complicated.

Why is the user interface necessarily different? A key example is in the construction of MLJ "learning networks" (see our tour https://github.com/alan-turing-institute/MLJ.jl/blob/master/doc/tour.ipynb a taste). These are very flexible and have nice properties (eg, "smart" updating selected components after parameter changes - meta data about algorithm are stored) Unless it is a recent improvement, there are some simple things sckitlearn pipelines cannot do, or are complicated to do, such as incorporate target transformations/inversions and stacking. I don't believe there is a way to improve this without fundamentally altering the user interface.

There are also differences between the two approaches at the basement level. Here are two that come to mind:

1. Supervised "models" in Scikitlearn are divided into Classifiers and Regressors. If a model can do probabilistic prediction, there is extra method (predict_proba) that is added on. However, to meet the requirements of a MLR-like task interface, and to better address proababilistic prediction, the classification in MLJ into models is much finer: There is a probabilistic/deterministic classification, and on top of that a classification into models based on target type: :continuous, :binary, :multiclass, :ordered_factor_finite, :ordered_factor_infinite. So a lot more models.

2. in scikitlearn a "model" bundles hyperparameters and learned parameters together in one object. In MLJ's basement level API, a "model" is hyperparameters only. The basement level API (we call it the model interface) is purely functional (no structs except for the one storing hyperparameters). Structure is added at a higher level of abstraction.

Finally, I should mention that there is a project underway to wrap the ScikitLearn.jl models in MLJ.

[vollmersj]

We would be very grateful if you decide to write interface to MLJ for StatsModels.jl guide for new models or review our attempt once we get there.

[juliohm]

Thank you @vollmersj for the clarifications, I am excited to hear of such an effort in Julia :100:

I am not an author nor contributor to StatsModels.jl, and have only asked about it because it seemed quite related. My main interest lies in the automatic adaptation of the ML models already implemented in a given well-thought interface for use in a spatial context: https://github.com/juliohm/GeoStats.jl As soon as I have more time, I will start reading the MLJ.jl project more carefully. I should be able to leverage the interface you guys are working on to automatically convert non-spatial ML models into spatial ones.

One thing that I would like to ask before closing the issue has to do with multivariate models. Have you guys established an interface for regressing more than one variable simultaneously for example? Imagine that I have measurements of temperate and pressure in a given region of the globe. I would like to be able to specify a fit method that makes use of both variables at once because there are correlations to be exploited. Similarly, there should be some corresponding predict method to give the answer as a list of pairs (temp,press).

Looking forward to collaborating on a better ML system in Julia :slightly_smiling_face:

[fkiraly]

@juliohm , I note you have written a very nice instance of what I'd call a "task interface" with EstimationProblem (= what we'd call "task", of type geospatial interpolation) vs solver (which we'd call "estimator"). In our terminology, solve = fit, I believe? You've even written some benchmarking functionality ("compare").

Up to the different names, your high-level interface isn't so different of what we're aiming for.
Thus, I wonder whether it would make sense to think of comparing interfaces carefully?

PS: for evaluation, you may also like to have a look at spatial cross-validation, as in
https://cran.r-project.org/web/packages/sperrorest

[fkiraly]

PPS: I agree with you that Krige and Matheron should be cited more often.

[juliohm]

Thank you @fkiraly , I am happy to brainstorm a more general interface, it would be great to compare what is already available in both projects. The solver term I am using in GeoStats.jl would be equivalent to fit + predict: it takes a fully-specified problem and solves it giving the final solution on the locations of the "test" set. I don't have much control on the intermediate results with this interface.

Ideally, we would be able to design a general interface for ML problems in MLJ.jl, and then projects like GeoStats.jl would specialize this interface to handle specificities that only exist in the application domain (e.g. spatial data).

[ablaom]

Regarding multivariate targets: we plan to handle them but this is not fully implemented yet.

[juliohm]

Thank you all for the clarifications. I will mark the issue as solved.

Looking forward to brainstorm ideas, and better integration in the near future. :+1: