Mlj.jl: Implementing MLJ model interface for EvoTrees.jl

[jeremiedb]

As it appears there isn't an actively maintained pure Julia gradient boosted tree package, would there be interest in integrating the following: https://github.com/Evovest/EvoTrees.jl
Not as comprehensive as xgboost at the moment (only dense matrix) and no histogram method implemented for speed on larger datastes, though would hopefully catch up eventually.

If so, some indications on potential interface adaptation for nigration would be welcomed.

[ablaom]

We would love to see your model implemented!

I suggest you implement the MLJ model interface specified in the guide natively, ie, in your own package. There is also a simplified model interface you can start to play around with here. Once your implementation is working well we can register your package at MLJRegsitry to make it available to any MLJ user.

In MLJ you will be implementing multiple models, depending on what you currently provide: EvoTreesRegressor for Continuous targets, EvoTreesClassifier for Multiclass or FiniteOrderedFactor target, and EvoTreesCount for Count targets (infinite ordered factors). Looks like you have at least a regressor, which is the easiest to implement, so start with this. For each model you must decide if you are making regular "point" predictions (probably the case for your regressor) or probabilistic predictions (probably the case for your classifier). In the latter case MLJ requires you to predict actual Distribution objects from Distributions.jl (see the guide) not simply a numerical probabilities.

An implementation of the MLJ model interface for XGBoost.jl is here, which you can use as a template. Note that the convention that XGBoost.jl uses for outputting probabilistic predictions is probably going to be different from yours so things won't be identical there.

Use the trait functions input_scitype_union and target_scitype_union to constrain the types of the inputs and target your models can handle, as explained in the guide. For example, you might declare target_scitype_union(EvoTreesClassifier) = Muliticlass{2} if you only support binary classification. If you want you could put the implementation code within an EvoTrees submodule but this is not necessary (and I wouldn't bother). Just import MLJBase, define the model struct (container for hyperparameters, eg EvoTreesRegressor) and start overloading MLJBase.fit, MLJBase.predict, etc for your model struct.

By the way, if you want to add histogram support to EvoTrees, feel free to lift Julia code from my KoalaTrees.jl package (see the histogram method).

Technical note: In the current model API Supervised, Probabilistic and Deterministic are parameterised by "fit-result" type. This will disappear very soon #93. In the meantime, you can just make it this parameter Any.

Happy to provide further support. Any questions?

[jeremiedb]

Thanks a lot for these indications, I look forward making an integration in the upcoming weeks.
As for the histogram, I'll give a try at your implementation. Have you performed any benchmark against Xgboost histogram performance? So far, greatest speeds on evotrees is obtained through exact method on discretized UInt8 features. Not yet as fast as xgboost hist, but still decent.

About the Extreme method (evaluating a single random split point as I understand it) is it an approach leverage by other libraries like xgboost/lightgbm/catboost?

[ablaom]

Great. Keep an eye on MLJ News for minor API changes which occasionally do effect the model interface.

[jeremiedb]

@ablaom

I think I got close to having a MVP, linear, logistic, Poisson and Quantile are now functional with performance that appears competitive with xgboost in various settings.

However, I have the following error following an attemp to run a demo withing the MLJ framework at the fit! step:

using Tables
using MLJ
import EvoTrees: EvoTreeRegressor
using EvoTrees: logit, sigmoid

features = rand(10_000) .* 5 .- 2
X = reshape(features, (size(features)[1], 1))
Y = sin.(features) .* 0.5 .+ 0.5
Y = logit(Y) + randn(size(Y))
Y = sigmoid(Y)
y=Y
X = Tables.table(X)

@load EvoTreeRegressor
tree_model = EvoTreeRegressor(max_depth=5)
tree = machine(tree_model, X, y)
train, test = partition(eachindex(y), 0.7, shuffle=true); # 70:30 split
fit!(tree, rows=train)

I first though it would be a conflict from importing EvoTreeRegressor, but if it's omitted, then the @load EvoTreeRegressor fails.

MLJ implementation are essentially in src/MLJ.jl: https://github.com/Evovest/EvoTrees.jl/blob/MLJ/src/MLJ.jl

With the above minimal example in https://github.com/Evovest/EvoTrees.jl/blob/MLJ/test/MLJ.jl

Any help on where I screw up would be much appreciated!

[ablaom]

Ooh that's exciting news! I can hopefully get back to you within 24 hours. The short answer to your problem is, I think, that you have not screwed up! It isn't possible to @load your model until I register it at MLJRegistry.

Is your package registered with General yet?

[ablaom]

However, I will explain a work around for testing MLJ functionality tomorrow.

[jeremiedb]

I've just open the request for adding to the MLJ compatible EvoTree to General registry:
https://github.com/JuliaRegistries/General/pull/948

[ablaom]

Okay, in more detail:

Your bug can be fixed by changing the type annotation on verbosity on your MLJBase.fit method:

# this is wrong:
function MLJBase.fit(model::EvoTreeRegressor, verbosity::Integer, X, y)

```julia

this is right:

function MLJBase.fit(model::EvoTreeRegressor, verbosity::Int, X, y)

I've just changed MLJBase so that either works, so after the next update it won't matter.

However, you still get a bug on the predict call at the end, but I'm guessing you can sort that one out.

Incidentally, my version of Tables does not have a Tables.table method; I used this code instead, which works apart from the last line.

```julia
using Tables
using MLJ
import EvoTrees: EvoTreeRegressor
using EvoTrees: logit, sigmoid

features = rand(10_000) .* 5 .- 2
X = reshape(features, (size(features)[1], 1))
Y = sin.(features) .* 0.5 .+ 0.5
Y = logit(Y) + randn(size(Y))
Y = sigmoid(Y)
y=Y
X = MLJ.table(X)

#@load EvoTreeRegressor
tree_model = EvoTreeRegressor(max_depth=5)
tree = machine(tree_model, X, y)
train, test = partition(eachindex(y), 0.7, shuffle=true); # 70:30 split
fit!(tree, rows=train)
yhat = predict(tree, MLJ.selectrows(X,test))

If you using MLJ and have imported EvoTrees.EvoTreeRegressor (as in your tests) then the @load line of code is redundant.

Once your package is registered at MLJRegistry, "EvoTreeRegressor" will appear when any MLJ user calls models() or models(task) for a task that matches your model traits, even if they have not loaded your package. They can load your model (without explicitly loading your package first) with @load EvoTreeRegressor. For all this to work you will need to add the following in EvoTrees/src/MLJ.jl:

MLJBase.load_path(::Type{<:EvoTreeRegressor}) = "EvoTrees.MLJ.EvoTreeRegressor"

[jeremiedb]

Awesome, I've just pushed your suggested changes and everything seems working now. Thanks for taking time providing guidance, I'm catching up on the Julia ecosystem, registries and cross-package integration are all new to me.

The request on for General registry is still pending, I understand it may take a few days for a new package.

As for the quantile regression, I'll need to bring some adaptations to make it funcitonal, I realized I overlooked some considerations discussed here about LightGBM, which I'll look to implement: http://jmarkhou.com/lgbqr/

Finally, there's a design aspecct I'm unsure to understand about MLJ. For boosting algorithms it's common practice to keep track of of performance metric on an evaluation dataset to identify the optimal number of iterations, though MLJ seams not to support this sort of parallel metric tracking. Is it a task that is to be supported?

[ablaom]

Finally, there's a design aspecct I'm unsure to understand about MLJ. For boosting algorithms it's common practice to keep track of of performance metric on an evaluation dataset to identify the optimal number of iterations, though MLJ seams not to support this sort of parallel metric tracking. Is it a task that is to be supported?

Good question. MLJ will be providing general interface for externally controlling the training of iterative models https://github.com/alan-turing-institute/MLJ.jl/issues/139, so that you won't have to do this internally. However, this functionality will only be an efficient alternative if you overload MLJBase.update along the lines explained in the guide.

To restart a gradient boosting algorithm from where it left off, there may be some extra data (beyond the fitresult) that you need to hold back at the conclusion of the previous iterations (call to fit). This is the data your MLJBase.fit method should return as cache, which the machine fit! method passes on when calling MLJBase.update.

If, after reading the guide, you still have questions about this, let me know. Happy to vet your update code.

[jeremiedb]

Hello, I've just pushed a branch that includes the MLJbase.update function:
https://github.com/Evovest/EvoTrees.jl/tree/MLJupdate

It call the grow_gbtree!() function, a slightly modified version of the original grow_gbtree.
grow_gbtree! seems to work fine to continue training.
However, the test on MLJ with the fit resulted in the same predictions after the continued training. It actually seems to behave as if the fit! fallback to the MLJbase.fit rather than MLJbase.update, although the call indicates that it performs updating rather than training:

[ Info: Updating Machine{EvoTreeRegressor} @ 1…41.

Maybe you may point why there's such fallback that's being applied?

Finally, the quantile regression has been fixed based on ideas discussed for lightgbm last summer and is now functional.

[ablaom]

Yeah, the guide is a bit thin here. I have done this for you:

https://github.com/ablaom/EvoTrees.jl/tree/MLJupdate

I have added a demo in /test/MLJ.jl to show you the new behaviour you get with update overloaded, and show you how to generate learning curves using MLJ. You can delete this.

Some remarks:

• Note that you did not have the signature of update correct; verbosity was not in the correct place

• At present, changing the learning rate does not trigger retraining from scratch. This is to allow adaptive learning rate control using MLJ (when we implement this!). You might choose to add a flag to your model hyperparameters to toggle this behaviour (although, you can always force training from scratch using fit!(tree, force=true.))

• I recommend against you using different values in the keyword constructor than the values you actually instantiate (eg, user says "loss=:linear" but you assign "loss=Linear()"). Strictly speaking MLJ does not allow this (although it doesn't seem to matter in your case). Either get the user to do "EvoTreeRegressor(loss=Linear())" or find a workaround. Note that Symbols can be type parameters, so you could have EvoTreesRegressor{:linear, Float64}. You could also do trait-style dispatch (using the types Val{:linear}, etc).

A general remark about EvoTrees: I couldn't see a way to control RNG seeding. In particular, if I set rowsample=0.5 (in other words not equal to one) and plot multiple learning curves, then they all look the same. Are you always starting training with a fixed seeding of the default RNG? Maybe add rng::Union{Int,AbstractRNG} as a hyperparameter`, as we do in MLJ/src/ensembles.jl? I think something like this is pretty standard. Then user has control.

[ablaom]

Update: My comments about the RNG may be out of place. See #163

[jeremiedb]

A seed parameter has been added to master. It matches the type of parametrization found in xgboost and lightgbm.

I've put more efforts lately on improving the histogram approach using bitsets, leading to significant speed improvements.

Regarding the constructor of the loss parameter, my initial implementation actually used the Val{:linear} approach. I moved away however in order to support more diverse learning paradigms while avoiding as possible duplicates of some functions common with normal gradient approach vs. quantile for example.
If you see any practical issue with this new approach, I'll to revert to the Val{:linear} way.

I'm unclear on the behavior of MLJ.update: why does it needs to perform a deepcopy of the model? I would have expected a simple mutation of the current model, similar to what is performed by grow_gbtree! vs grow_gbtree. I'm having concerns about the resulting overhead of copying the model at each iteration for the purpose of early stop. I'd tend to see the early stopping mechanism inherent to the fitting process, not unlike the number of iterations of GLM solver. As such, although it may sounds extreme, I'd almost see the evaluation data as part of the hyper-parameters. Any thoughst?

[ablaom]

Thanks for that. Good questions.

Regarding the constructor of the loss parameter, my initial implementation actually used the Val{:linear} approach. I moved away however in order to support more diverse learning paradigms while avoiding as possible duplicates of some functions common with normal gradient approach vs. quantile for example.
If you see any practical issue with this new approach, I'll to revert to the Val{:linear} way.

One problem may be that you are trying to bundle more functionality into a single model than MLJ allows. Presently one cannot use a single model struct to represent algorithms which predict different kinds of targets (eg, Continuous and Count). Also, the same model cannot do both probabilistic prediction (eg, logistic regression) and deterministic prediction (predict a single class). You can see some discussion regarding this design choice in #81.

Of course, this doesn't necessarily imply a lot of duplication of code for fitting, just more model struct definitions.

I'm unclear on the behavior of MLJ.update: why does it needs to perform a deepcopy of the model? I would have expected a simple mutation of the current model, similar to what is performed by grow_gbtree! vs grow_gbtree. I'm having concerns about the resulting overhead of copying the model at each iteration for the purpose of early stop. I'd tend to see the early stopping mechanism inherent to the fitting process, not unlike the number of iterations of GLM solver. As such, although it may sounds extreme, I'd almost see the evaluation data as part of the hyper-parameters. Any thoughst?

Are you saying your "model" is more than just hyperparameters? Ie, includes fitted parameters as well? In that case you are not following the MLJ API model spec. The Model subtype you expose to MLJ should be hyperparameters only (which are trivial to copy).

[ablaom]

Regarding internal control of iterative behaviour. Why would you want to do this internally? There are dozens or algorithms that all require some kind of iteration control and iteration control strategies are becoming increasingly more sophisticated and varied (eg, cylic control of learning rate popular in deep learning). From the point-of-view of the broader eco-system, we feel we ought to provide an interface for this kind of thing, and implementing this is a high priority. Why wouldn't you want to buy into it? Why duplicate effort?

That said, there is nothing stopping you from exposing your internal control to the MLJ user. You do this by making the control specs part of the hyperparameters (eg, patience level, evaluation strategy - such as holdout or cv - and so forth). To expose the results of internal performance estimates, you can return these in report, eg `report = (cv_scores=..., nepochs_used=...)' or whatever.

[tlienart]

Closing this issue as it's quoted in https://github.com/alan-turing-institute/MLJModels.jl/issues/42

[ablaom]

@jeremiedb Haven't heard from you in a while. It seemed you were very close to having an MLJ interface.

Is EvoTrees now registered?

Have you separated learned parameters and hyperparameters as discussed above to avoid data copying?

[jeremiedb]

@ablaom Hello! The latest efforts went into performance (pretty close to xgboost now) and expanding model coverage (multiclass and gaussian likelihood).

Yes, library is now registered, I'm about to release 0.3.0 with the above additions.

Haven't put more time into the MLJ integration however. I can imagine solving the performance concern by having having the histograms and gradients returned in the cache which would then get copied and passed by reference. I'd need to perform regression test for such approach as I'm primarily seeking a performance first approach on the package and I'm relying on it for work.

I noticed the MLJ dependency forcing DataFrames to v0.18. Is it something expected to be resolved short term? I faced some dependency issues trying to plug-in the MLJ interface into the master.

[ablaom]

Thanks for the update!

I noticed the MLJ dependency forcing DataFrames to v0.18. Is it something expected to be resolved short term? I faced some dependency issues trying to plug-in the MLJ interface into the master.

This is an issue: https://github.com/alan-turing-institute/MLJBase.jl/issues/44

I'm currently on leave till Monday. Will be onto this. 1-3 weeks.

[jeremiedb]

Regarding the update / fit! functions, can it be assumed that they are only applicable to the very same input and target data? Ie: it's not to be used to fine tune training on a new dataset?
Otherwise, it may put some additional burden for the cache returned by fit to be reusable without incurring some preprocessing.

[tlienart]

@ablaom will get back to you within a couple of days :)

[ablaom]

Yes, update will only ever be called on the same data provided to a previous call to fit. If an MLJ machine receives new data (or new rows of data are selected for training), the model fit will be called anew. The details on the logic are here (see the last part). Note that the X and y appearing in the update signature are there for convenience. They will not be different from the X and y passed to fit previously (and you might be ignoring them if you passed data through cache to avoid repeating some data pre-processing).

In the future you will able to define an update_data method to enable "online learning" - that is, to update the learned parameters based on new data - but that facility is not there yet.

Does this make sense? Does this actually answer your query? If not, please let me know and I will get back to you early next week.

[jeremiedb]

Thanks, yes this all sounds clear.
I'll give a shot in the next days for having a workaple update method in order to have a MVP for EvoTrees supporting MLJ.

[ablaom]

Minor note. In the update method there is a comparison of the model (ie hyperparameters) with previous state, to see if only the number of iterations or learning rate has changed. (I'm referring to the PR I did a while back). This is now easier with a new method in MLJBase, is_same_except(model1, model2, exceptions...).

[jeremiedb]

I've pushed a MLJ branch which appears to provide proper behavior for the MLJ fit!, including the underlying fit and update as well as the predict method:
https://github.com/Evovest/EvoTrees.jl/tree/MLJ

Regarding the continuation of the model training, it however appears that the training speed about 50% when iterating over update instead of going through a single fit with a corresponding number of rounds. Still fairly fast given the performane is now on par with xgboost hist, but I'm assuming that the caching procedure is responsible for this.
See: https://github.com/Evovest/EvoTrees.jl/blob/d13e4ba2d8beeca24b8719f8c58d1c3cb6f82be2/test/MLJ.jl#L70

For tracking the fitting process of iterative algos such as gradient boosting, I'm still unclear on the recommended tooling in the MLJ ecosystem. In EvoTrees, the approach taken is like in xgboost and lightgbm: an eval data is provided in addition to the train data to track at which iteration to stock the process. My understanding is that MLJ approach would be a loop that would alternate calls to update followed by predict on the eval data in order to decide when fitting is complete. Is there such tool already includeed in MLJ?
A potential drawback with such approach (other than the above slowdown in update due to caching that can hopefully be resolved) is that the predict on eval data would involved refitting the full model (all its trees), while when the evaluation tracking is performed within the fitting algo, the predictions can be performed on the incremental trees.
A predict! method with an optional parameter specifying the last N trees to add to current predictions could circumvent this small inefficiency. Ultimately, I think it depends on the MLJ API approach for handling early stopping process on gradient boosting algos.

I've yet to include the is_same_except, it will effectively lighten up the code!

[ablaom]

You raise a number of good points here, thanks!

For tracking the fitting process of iterative algos such as gradient boosting, I'm still unclear on the recommended tooling in the MLJ ecosystem. In EvoTrees, the approach taken is like in xgboost and lightgbm: an eval data is provided in addition to the train data to track at which iteration to stock the process. My understanding is that MLJ approach would be a loop that would alternate calls to update followed by predict on the eval data in order to decide when fitting is complete. Is there such tool already includeed in MLJ?

Yes this is the general idea. The tool is not yet written but a design is suggested here: #139 I nevertheless think this is the way to go, because this will ultimately give the user more control and save you duplicating a lot of functionality that's common to all iterative algorithms.

If, nevertheless, you wish to provide an internal estimation of the the performance you could do it something like this (assuming you are only intending simple holdout resampling):

• add two hyperparameters, fraction_train, measures, where measures is a single measure or list of measures the user wants estimates for; ideally, any MLJ measure could be given here. See the measures API for detail; in particular, you will be calling something like aggregate(value(measure, yhat, X, y, nothing), measure) to evaluate each measure.

• add hyperparameters specifying the stopping criterion

• return the history of measure evaluations in your fit report (a named tuple)

involved refitting the full model (all its trees) ...

Sorry, I guess I'm missing something here. Perhaps I'm not sufficiently familiar with the algorithm. Do you not store all trees that have been fit? Why do you need to re-fit them? Can't you cache these trees?

[jeremiedb]

involved refitting the full model (all its trees) ...

My bad, the use of refitting was inappropriate. The gradient boosted model is effectively storing all the trees as a vector a trees. The sub-optimal efficiency I was referring to was about a situation like the following:
Let's assume the fitting process built 100 trees. Calling the predict on the holdout data in order to get the evaluation metric involves summing the predicion of those 100 trees. Let's say the fit continues for another 10 rounds. predict is once again called on the holdout data, resulting in summing the predicitons for the 110 trees. However, that calculation is redundent with the previous one. We could have just added the 10 latest trees to the previous predictions.
Likely not critical on a performance perspective though.

Thanks for the indications about potential ways to integrate the eval metric. I agree though that in term of design, decoupling of the fitting and evaluation results in leaner building blocks for the library.

Are you seeing remaining aspects for having a legitimate initial MLJ integration of the EvoTrees?

[ablaom]

RIght, I understand now. Yes, with our design there is a performance penalty for external control but I also agree this will be very small.

Are you seeing remaining aspects for having a legitimate initial MLJ integration of the EvoTrees?

I'm certainly very impressed so far. There is one issue to discuss, in terms of the big picture and ultimate goal for integration. Currently EvoTrees exposes just a single model EvoTreeRegressor combining all the different objective functions, and this is fine for the base API. In MLJ, however, one generally has a different model struct for each "scientific type" of target you are trying to predict, where this type is declared by the target_scitype trait. In the present case, this would mean having something like this:

model | target_scitype value
----------------------------------|--------------------------
EvoTreeRegressor | AbstractVector{<:Continuous}
EvoTreeClassifier | AbstractVector{<:Finite}
EvoTreeCount | AbstractVector{<:Count}
MultivariateEvoTreeRegressor ? | Table(Continuous)

In each case the y that your methods receives from the user, through the MLJ interface, is different: Continuous means AbstractFloat, Finite means CategoricalValue, and so forth (see also the end of Getting Started). In the case of the classifier, for example, only the logistic and softmax objective functions make sense, and the idea is your algorithm, by looking at the number of classes, would automatically pick the appropriate one for the user, and the other options would not even be available. For the case of EvoTreeCount only the Poisson objective function applies. For EvoTreeRegressor you may want to allow the user more choices of objective function, but issue a warning if the user makes a "strange" choice; see this is issue.

There is also the important distinction, enforced in MLJ, about whether you are making a point "deterministic" prediction, or a probabilistic one, which determines the super type of the model (Deterministic or Probabilistic). At present some of your objective functions implicitly indicate that deterministic predictions are being made, while others (eg, logit) that (usually) a probabilistic prediction is begin made.

Fortunately, there is already a template on which to base your separation of the models and their methods, namely the XGBoost/MLJ integration at https://github.com/alan-turing-institute/MLJModels.jl/blob/master/src/XGBoost.jl . (There is a lot of code duplication there. If you are comfortable with some metaprogramming, this could be greatly reduced. This was one the early interfaces, done by a PhD student.) The trickiest part to get correct is the (probabilistic) predictions for classification. XGBoost has (in my view) a weird and poorly documented way to output the various probabilities, which need to be collated for outputting the UnivariateFinite distributions required by MLJ. If you use the same convention as XGBoost, then you can just copy and paste; otherwise you need to think this out carefully (and to test in any event).

As far as input features are concerned, all your models will get the same trait value, declared as follows:

MLJBase.input_scitype(::Type{<:EvoTypes}) = MLJBase.Table(MLJBase.Continuous)

assuming you are allowing X to be any Tables.jl supported table having columns that are AbstractFloat. If (as I suspect) you can also handle count data (encoded as integers) you could expand this to MLJBase.Table(Continuous, Count). You could, I expect, even include OrderedFactor columns, but as these will come as categorical vectors, you will need to internally convert them to integers or floats, which you can do with using MLBase; Xfixed = coerce(X, OrderedFactor=>Continuous).

Next steps

We could look at adding EvoTreesRegressor to MLJ more-or-less immediately, but with limited options for the objective function (only those appropriate for making deterministic predictions of a univariate continuous target). However, I would strongly encourage you to think about finishing the rest of the interface first, as I think this will make for a winning integration we can promote to our users (in MLJ tutorials, and so forth). Users looking for a pure julia alternative to XGBoost are going to disappointed if they can't do classification.

If you just want to proceed in stages let me know when you have a release that supports the interface (or point me to a branch if you prefer) and I will test it and add it to the registry (when live). Keep me posted with technical queries as they arise.

[jeremiedb]

Thanks for the detailed feedback.
I'll move forward with cleaning the Regressor and Classifier parts in order to have a reasonably good coverage to start with. Usage should then show what next stages to be prioritized.
I'll keep you posted once the Regressor/Classifier interface is ready.

[jeremiedb]

@ablaom
Support for various regression typres (linear, quantile, logistic) as well as classification has progressed, there's the support for classification predictions to be of Distribution type that remains to be integrated.

Regarding performance, the underlying fit and update are both fast as the original implementation. However, it seems like MLJ inherent functions add significant overhead.

Two points:

1. machine gets significantly time consuming from medium sized data. For example, with a 100K row,
   X 100 features:

@time tree = machine(tree_model, X, Y)
julia> 
 50.686730 seconds (1.09 k allocations: 228.934 MiB, 0.04% gc time)

1. fit! methods adds significant time and allocations compared to the underlying MLJ.update:

using MLJBase
tree.model.nrounds += 1
@time EvoTrees.grow_gbtree_MLJ!(tree.fitresult, tree.cache, verbosity=1)
julia> 
  0.018894 seconds (4.05 k allocations: 4.936 MiB)

tree.model.nrounds += 1
@time MLJBase.update(tree.model, 0, tree.fitresult, tree.cache, X, Y)
julia> 
 0.017662 seconds (4.05 k allocations: 4.935 MiB)

tree.model.nrounds += 1
@time MLJ.fit!(tree, rows=train, verbosity=0)
julia> 
 0.086035 seconds (4.60 k allocations: 142.903 MiB, 5.79% gc time)

MLJBase.update has runtime in line with EvoTrees.grow_gbtree_MLJ!, which is execpted since it's the only task update performs outside basic validation prior to resume training. MLJ.fit! is however much slower along higher allocations footprint. Are there overheads brought by fit! that can be avoided? At least from a EvoTrees perspective, I'm not seeing why fit! would add overhead compred to update. Considering that a boosting tree fitting process involves numerous calls to update if we want to be able to track evaluation metrics, the importance of the slowdown with fit! appears problematic. At least, I find difficult to justify 2x-3x slowdown.

Above tests are in the MLJ branch:
https://github.com/Evovest/EvoTrees.jl/blob/f1039c59c4d71dfc1c858c85b23e1c7635b20906/test/MLJ.jl#L67

[jeremiedb]

Regarding point 2, the following section in fit! seems to be in cause for the overhead:
https://github.com/alan-turing-institute/MLJBase.jl/blob/master/src/machines.jl#L164

Could the reassignation of args be skipped and directly reused existing X,y in machine?

[ablaom]

These are really useful observations! Many thanks for reporting and for the further investigation.

Not sure about 1 yet.

Issue 2. This is more serious, for the reasons you point out. One solution is for machines to cache (a reference) to previous data selections (and not just previously used rows). This is not currently done (which is why your suggested solution will not work). The better solution, I think, is to speed up the calls to selectrows in the case of in-memory tabular data. Unfortunately, Tables.jl does not support fast access for tables that happen to be in-memory.

The optimisation you seek may have to wait a bit, but I have opened an issue: https://github.com/alan-turing-institute/MLJBase.jl/issues/151

[ablaom]

@jeremiedb Further to issue 2: selectrows is now optimised in MLJBase 0.9.2 in the special case of a row table (iterators of named-tuples). You can convert any table X into a row table with Tables.rowtable(X). Can you please see what improvement you obtain in your benchmarks if you use a row table instead of the Tables.MatrixTable you are currently using (currently not optimised)?

It would also be of interest to see how DataFrames perform, another special case optimised in MLJBase 0.9.2.

[jeremiedb]

@ablaom
Fairly good news, with Tables.rowtable(X), there's no significant overhead by having iterative calls to fit! method, only a 20% increase in allocations.

However, with DataFrames, iterative training is still significantly impaired:

0.662708 seconds (44.94 k allocations: 666.022 MiB, 3.74% gc time)

Compared to the following with plain update():

0.219162 seconds (40.47 k allocations: 49.358 MiB)

Maybe I'm overlooking potential issues, but I was wondering if bringing a slight modification to MLJBase.fit!() could resolves issues for all X formats. By changing this line for :

args = rows_have_changed ? [selectrows(arg, rows) for arg in mach.args] : mach.args

It skips the allocations by directly referring to current machine args is rows haven't changed. Using this modified version of MLJBase.fit!, execution time and allocations is now on par with plain update:

@time for i in 1:10
    tree.model.nrounds += 1
    test_fit!(tree, rows=train, verbosity=0)
end
0.235022 seconds (40.88 k allocations: 49.379 MiB)

[ablaom]

Thanks indeed for that! That confirms your suggestion about where the problem lies.

No, sadly your suggestion misses some logic. args is a constant thoughout the life of the machine. The rows may not have changed, but their previous value need not have been the complete set of indices either. At the moment we are only caching rows between fit/update calls and not the data itself. That's why there is not a quick fix.

Would you mind doing a benchmark for Tables.columntable(X) as well, which has also been optimised? It would be good to see if the DataFrames slowdown is due to that being a column-based format, or because of some other issue.

[jeremiedb]

Oh right, I misseed that args refer to the original X,Y fed to the machine than the subset defined by the rows, thanks for clarifying.

As for the Tables.columntable(X), slowdown is essentially the same as for DataFrames. So above figures of roughly 0.66 sec vs 0.22 sec. also applies.

[jeremiedb]

@ablaom I've just completed some refactor to have a common training logic regardless of training through MLJ interface or natively through the package.
EvoTreeRegressor is ready for supporting linear as well as quantile and logistic regression.
EvoTreeClassifier is also ready and has integrated the probabilistic predictions UnivariateFinite and as well as the predict_mode.

There are minimal example of fitting, updating and prediction on regression and classification tasks on the MLJ branch at https://github.com/Evovest/EvoTrees.jl/blob/MLJ/test/MLJ.jl.

Let me know if it seems aligned for a release. I'll only need to make some minor adjustments to the evaluation metric tracking for the native fitting process and should then be ready to merge on master.

[ablaom]

@jeremiedb Many thanks for the update. Been on vacation but will try to get to this soonish.

[ablaom]

Okay, I have reviewed the MLJ.jl tests you link above and some other
related files.

The MLJ interface

I am assuming that test/MLJ.jl only tests the API not the core
model. This is looking very good. Only minor comments:

• No clean! methods implemented to ensure hyperparameters in correct range

• Still not too happy about the that fact user enters :linear but
  keyword constructor changes this to Linear(), and so forth. MLJ
  now has 125 models and none of them touch user input unless a
  hyper-parameter is out of range (and then only through the clean!
  method). This could cause problems in the future (if not
  already). For example, info(EvoTreeRegressor).hyperparameter_types
  shows objective as Symbol but that will never be the actual
  type. I suggest the user specifies objective=Linear() or you
  simply add another layer of dispatch using Val{:linear} in your
  core algorithm. BTW, thanks to @tlienart you can now create the model type,
  constructor and clean! method in one hit, with the @mlj_model; see
  the
  manual
  for details.

• On line 185 of test/MLJ.jl you have a predict_train which has not been
  defined (except in earlier section in reference to different model).

• Optional: You can now specify a hyperapameter_ranges trait, which
  will allow (in the future) hyperparameter optimization strategies to
  deduce a default range for their hyperparameters. See the manual.

• you have MLJModels as a dependency, which I am guessing is
  unnecessary. You don't use any other MLJ models, do you?

Other comments

Here are some other observations. Perhaps you are already aware of
them. The last point might require prompt attention:

• using EvoTrees seems to take a very long time: 86 sec on my
  machine. This is almost twice as long as The Beast, Plots.jl. I
  haven't investigated, but you might want to make some dependencies
  "optional" using Requires.jl. Perhaps you could use Tables.jl to
  dump DataFrames and CSV as dependencies; they are quite large. Note
  that MLJBase has become somewhat bloated but there is
  restructuring
  underway to dramatically reduce this.

• There is something amiss with EvoTreeClassifier. It is two orders of magnitude slower
  thanXGBoost. And (trying my
  best to match the hyperparameters) I get wildly different
  behaviour. In particular, the HingeLoss() get's worse as nrounds
  increases, instead of better:

using MLJ, StatsBase, Random, Plots, CategoricalArrays, LossFunctions,
    EvoTrees

pyplot()

X, y = @load_crabs


## XGBOOST

xgb_model = @load XGBoostClassifier
xgb = machine(xgb_model, X, y)

r = range(xgb_model, :num_round, lower=50, upper=500)
@time curve = learning_curve!(xgb, range=r, resolution=10,
                              measure=HingeLoss())
# 0.193890 seconds (63.82 k allocations: 4.269 MiB)

plot(curve.parameter_values, curve.measurements)
savefig("xgb.png")


## EVOTREES

evo_model = EvoTreeClassifier(max_depth=6, η=0.3,
                               λ=1.0, γ=0.0, nrounds=10, nbins=256, α=0.0)
evo = machine(evo_model, X, y)

r = range(evo_model, :nrounds, lower=50, upper=500)
@time curve = learning_curve!(evo, range=r, resolution=10,
                        measure=HingeLoss())
# 149.250696 seconds (42.90 M allocations: 2.974 GiB, 0.88% gc time)

plot(curve.parameter_values, curve.measurements)
savefig("evo.png")

XGBoost learning curve

EvoTrees learning curve

[jeremiedb]

Thanks for the feedback.

About the Val{:linear}, it actually was the original route I used then moved toward a the current one to better handle the hierarchy in how predictions get calculated. I think it adds not so much to the code efficiency so will consider reverting to the suggested Val{:linear}. I wasn't totally sure if that approach was resulting in an overhead in the dispatch mechanism, let me know if you're aware, otherwise I'll perform regression tests to confirm it's safe.

Effectively, I haven't spent time around the cleaning methods and range restrictions. Admittedly, I'm focusing more on the performance side right now, notably GPU implementation. But will condsider adding gatekeepers in the short term.

Good point about the dependencies, I've clean out DataFrames and CSV. I think Flux wasn't helping and was only used for a minor utility function so I've dropped it as well. Loading time seem to have imprived singificantly from those fixes.

Finally, I have difficulty figuring out was might have caused the poor performance in your crab test.
I've just pushed an update to MLJ branch in which the above test was added: https://github.com/Evovest/EvoTrees.jl/blob/8c05e6eac7984f6863e51375d38e01b3cd407d97/test/MLJ.jl#L76
It runs in 1.27 sec on my laptop. I know the softmax is little less performant than the other regressors, though on the tests I've performed on reasonably sized datasets, I always got performance aligned with xgboost fast histogram methods.
For the graph, I also got a similar graph to yours on the HingeLoss. However, looking at the cross-entropy through training, it indicates that model is gaining in performance. As well, looking through small steps iterations on line 73 to 79, it can be seen that the accuracy progressively moves toward 100%. Not familiar with LossFunctions, but is it possible that HingeLoss expects a single metric? EvoTrees classifier exclusively uses softmax for classification, so it always produces an matrix of predictions where ncols = number of categories. Maybe issue is that hinge expect a single vector of probabilities associateed with second level, which might explain the apparent bad fit?

[jeremiedb]

Quick update regarding the poor softmax performance. From what environment were you running it? I realized that while running from Juno on Windows was going fine, softmax and quantile performance on the the R package that wrap EvoTrees was quite bad. Turns out that after activating the deprecations warning in Juno, things turned bad as well. Long story short, there were some deprecations arounds broadcasting operations on StaticArrays that caused some fuss. These have been fixed, and following the push, softmax is now faster than xgboost on 100K obs / 100 features toy data.

[ablaom]

However, looking at the cross-entropy through training, it indicates that model is gaining in performance.

Consider the cross-entropy result with caution. See this discussion on the crab data set.

I repeated my experiments for BrierScore and again got improvement from XGBoost, but deterioration for EvoTrees. Are you saying your algorithm is different from XGBoost in the case of probabilistic classification, so the comparison is unfair?

I have not checked your new claims on performance improvements.

From what environment were you running it?

I'm running from the REPL through emacs. Sorry I don't recall the julia version. I will get back to you on this when I redo the performance checks.

(wo3) pkg> st -m
    Status `~/.julia/environments/wo3/Manifest.toml`
  [621f4979] AbstractFFTs v0.5.0
  [1520ce14] AbstractTrees v0.3.1
  [79e6a3ab] Adapt v1.0.0
  [ec485272] ArnoldiMethod v0.0.4
  [7d9fca2a] Arpack v0.3.2
  [6e4b80f9] BenchmarkTools v0.4.3
  [9e28174c] BinDeps v1.0.0
  [b99e7846] BinaryProvider v0.5.8
  [fa961155] CEnum v0.2.0
  [336ed68f] CSV v0.5.22
  [3895d2a7] CUDAapi v2.1.0
  [c5f51814] CUDAdrv v5.0.1
  [be33ccc6] CUDAnative v2.7.0
  [7057c7e9] Cassette v0.3.1
  [324d7699] CategoricalArrays v0.7.7
  [da1fd8a2] CodeTracking v0.5.8
  [944b1d66] CodecZlib v0.6.0
  [3da002f7] ColorTypes v0.8.1
  [5ae59095] Colors v0.9.6
  [bbf7d656] CommonSubexpressions v0.2.0
  [34da2185] Compat v2.2.0
  [ed09eef8] ComputationalResources v0.3.0
  [8f4d0f93] Conda v1.3.0
  [d38c429a] Contour v0.5.1
  [a8cc5b0e] Crayons v4.0.1
  [3a865a2d] CuArrays v1.6.0
  [9a962f9c] DataAPI v1.1.0
  [a93c6f00] DataFrames v0.20.0
  [864edb3b] DataStructures v0.17.9
  [e2d170a0] DataValueInterfaces v1.0.0
  [7806a523] DecisionTree v0.10.1
  [163ba53b] DiffResults v1.0.2
  [b552c78f] DiffRules v1.0.0
  [b4f34e82] Distances v0.8.2
  [31c24e10] Distributions v0.21.12
  [ffbed154] DocStringExtensions v0.8.1
  [f6006082] EvoTrees v0.4.0 [`../../../Dropbox/Julia7/EvoTrees`]
  [c87230d0] FFMPEG v0.2.4
  [7a1cc6ca] FFTW v1.1.0
  [48062228] FilePathsBase v0.7.0
  [1a297f60] FillArrays v0.8.4
  [53c48c17] FixedPointNumbers v0.6.1
  [587475ba] Flux v0.10.1
  [59287772] Formatting v0.4.1
  [f6369f11] ForwardDiff v0.10.9
  [38e38edf] GLM v1.3.6
  [0c68f7d7] GPUArrays v2.0.1
  [28b8d3ca] GR v0.44.0
  [4d00f742] GeometryTypes v0.7.7
  [bd48cda9] GraphRecipes v0.4.0
  [7869d1d1] IRTools v0.3.0
  [d25df0c9] Inflate v0.1.1
  [41ab1584] InvertedIndices v1.0.0
  [82899510] IteratorInterfaceExtensions v1.0.0
  [682c06a0] JSON v0.21.0
  [aa1ae85d] JuliaInterpreter v0.7.8
  [e5e0dc1b] Juno v0.7.2
  [929cbde3] LLVM v1.3.3
  [b964fa9f] LaTeXStrings v1.0.3
  [7f8f8fb0] LearnBase v0.2.2
  [093fc24a] LightGraphs v1.3.0
  [30fc2ffe] LossFunctions v0.5.1
  [6f1432cf] LoweredCodeUtils v0.4.2
  [add582a8] MLJ v0.7.0
  [a7f614a8] MLJBase v0.10.1
  [d491faf4] MLJModels v0.7.1
  [03970b2e] MLJTuning v0.1.0 [`../../../Dropbox/Julia7/MLJ/MLJTuning`]
  [1914dd2f] MacroTools v0.5.3
  [442fdcdd] Measures v0.3.1
  [e89f7d12] Media v0.5.0
  [e1d29d7a] Missings v0.4.3
  [6f286f6a] MultivariateStats v0.7.0
  [872c559c] NNlib v0.6.4
  [77ba4419] NaNMath v0.3.3
  [b8a86587] NearestNeighbors v0.4.4
  [46757867] NetworkLayout v0.2.0
  [bac558e1] OrderedCollections v1.1.0
  [90014a1f] PDMats v0.9.11
  [d96e819e] Parameters v0.12.0
  [69de0a69] Parsers v0.3.10
  [ccf2f8ad] PlotThemes v1.0.1
  [995b91a9] PlotUtils v0.6.2
  [91a5bcdd] Plots v0.28.4
  [2dfb63ee] PooledArrays v0.5.3
  [08abe8d2] PrettyTables v0.6.0
  [92933f4c] ProgressMeter v1.2.0
  [438e738f] PyCall v1.91.2
  [d330b81b] PyPlot v2.8.2
  [1fd47b50] QuadGK v2.3.1
  [3cdcf5f2] RecipesBase v0.7.0
  [189a3867] Reexport v0.2.0
  [ae029012] Requires v0.5.2
  [295af30f] Revise v2.5.0
  [79098fc4] Rmath v0.6.0
  [321657f4] ScientificTypes v0.5.1
  [6e75b9c4] ScikitLearnBase v0.5.0
  [1277b4bf] ShiftedArrays v1.0.0
  [992d4aef] Showoff v0.3.1
  [699a6c99] SimpleTraits v0.9.1
  [a2af1166] SortingAlgorithms v0.3.1
  [276daf66] SpecialFunctions v0.8.0
  [90137ffa] StaticArrays v0.12.1
  [2913bbd2] StatsBase v0.32.0
  [4c63d2b9] StatsFuns v0.9.3
  [3eaba693] StatsModels v0.6.7
  [3783bdb8] TableTraits v1.0.0
  [bd369af6] Tables v0.2.11
  [a759f4b9] TimerOutputs v0.5.3
  [37b6cedf] Traceur v0.3.0
  [3bb67fe8] TranscodingStreams v0.9.5
  [30578b45] URIParser v0.4.0
  [81def892] VersionParsing v1.2.0
  [ea10d353] WeakRefStrings v0.6.2
  [009559a3] XGBoost v0.4.2
  [a5390f91] ZipFile v0.8.4
  [e88e6eb3] Zygote v0.4.6
  [700de1a5] ZygoteRules v0.2.0
  [2a0f44e3] Base64 
  [ade2ca70] Dates 
  [8bb1440f] DelimitedFiles 
  [8ba89e20] Distributed 
  [7b1f6079] FileWatching 
  [9fa8497b] Future 
  [b77e0a4c] InteractiveUtils 
  [76f85450] LibGit2 
  [8f399da3] Libdl 
  [37e2e46d] LinearAlgebra 
  [56ddb016] Logging 
  [d6f4376e] Markdown 
  [a63ad114] Mmap 
  [44cfe95a] Pkg 
  [de0858da] Printf 
  [9abbd945] Profile 
  [3fa0cd96] REPL 
  [9a3f8284] Random 
  [ea8e919c] SHA 
  [9e88b42a] Serialization 
  [1a1011a3] SharedArrays 
  [6462fe0b] Sockets 
  [2f01184e] SparseArrays 
  [10745b16] Statistics 
  [4607b0f0] SuiteSparse 
  [8dfed614] Test 
  [cf7118a7] UUIDs 
  [4ec0a83e] Unicode 

[jeremiedb]

The the softmax actually optmizes the the logloss measure and thus the loss measurement on the training dataset should be continuously decreasing. This is what can be observed on both the logloss and the accuracy if looping the training process as on lines 73-79 in test/MLJ.jl.

Given the nature of a boosting model, it's not recommended to track model performance on the training dataset as the evaluation measure will improves until an almost perfect overfitting of the data. So for a boosting tree, making a search on the number of iterations by looking at some performance metric on the train will result in pushing for always more iterations is the evaluation metric is aligned with the loss function. If adding lines 81-82 to the train loop, it will add the evaluation of the model on the eval dataset. It can be observed that the logloss does improves prior to stagnating, which again is aligned with the expected behavior and allow to select an optimal nrounds.

All the above observations points towards the appropriateness of the model fit. I don't know how to illustrate it more clearly. If looking at the predictions after some large number of iterations, it can also be observed that the predictions are strongly aligned with the observed target. With accuracy at 100% and such observation, I think it makes it quite obvious that model fit is strong (on training data).

What I think resulted in the weird Hinge/Brier is that the prediciton of the Evo classifier is a matrix of size [sample size, number of classes]. I guess that the Hinge/Brier loss measure was expecting a vector of [sample size]. However, for the softmax, I want to stay consistent on the prediction format and stock to the matrix with ncols = number of classes.

Given the very small crab dataset, I'd recomment choosing a smaller model, like depth 3 or 4 and a learnign rate around 0.01 if performing 50 to 500 iterations.

[ablaom]

Given the nature of a boosting model, it's not recommended to track model performance on the training dataset as the evaluation measure will improves until an almost perfect overfitting of the data. So for a boosting tree, making a search on the number of iterations by looking at some performance metric on the train will result in pushing for always more iterations is the evaluation metric is aligned with the loss function.

FYI The learning_curve! function does not track the training error, but uses resampling, in this case Holdout.

I agree that if your algorithm is working to minimise training log loss, then the training cross-entropy should go down. I guess your algorithm is just not the same as the XGBoost one, at least with the hyperaparameter settings I chose.

What I think resulted in the weird Hinge/Brier is that the prediciton of the Evo classifier is a matrix of size [sample size, number of classes]. I guess that the Hinge/Brier loss measure was expecting a vector of [sample size]. However, for the softmax, I want to stay consistent on the prediction format and stock to the matrix with ncols = number of classes.

I'm sorry I don't follow this. This seems to be an implementation detail unrelated to the tests. Predictions of MLJ probabilistic classifiers (for univariate targets) are vectors of UnivariteFinite distributions (as I indeed observed in testing) and never matrices. All the MLJ metrics (including the LossFunctions ones) expect predictions in this form (there is an internal adaptor to get the LossFunctions to work this way). Since I was testing your algorithm through the MLJ interface, how could your (internal) matrix predictions be relevant?

In any case, I am happy for you to take responsibility for the algorithm correctness. I just wanted to share the fact that I was not able to easily reproduce the XGBoost results.

I will check the improvements to fit time for classifiers shortly.

[ablaom]

Okay all the following are now resolved at my end:

• slow using EvoTrees
• slow fit of classifier

Note also that the crab tests are less surprising after shuffling the data (which was ordered by target value):

using Random

data = load_crabs()
data = selectrows(data, shuffle(1:nrows(X)))
data = coerce(data, Count => MLJ.Continuous)

y, X = unpack(data, ==(:sp), x->!=(x,:sex))

using LossFunctions, Plots
pyplot()

evo_model = EvoTreeClassifier(max_depth=3, η=0.1,
                              λ=1.0, γ=0.0, nbins=256, α=0.0)
evo = machine(evo_model, X, y)

r = range(evo_model, :nrounds, lower=1, upper=100)
plt = plot()
for m in [HingeLoss(), cross_entropy, BrierScore()]
    @time curve = learning_curve!(evo, range=r, resolution=100,
                                  measure=m)
    plot!(curve.parameter_values, curve.measurements, label="$m")
end
plt
savefig("~/evo.png")

Some more minor observations:

• your tests will fail under julia 1.0 because you use the eachrow method which is unsupported before 1.1. Either replace (recommended) or update [compat] julia = "^1.1"

• I'm guessing you'll want to drop Plots as test dependency (you`re not actually using it here anyhow)

• Ditto Revise

• I notice you overload predict_mean, predict_mode, and so forth but it looks like you just use the fallback definitions. You only need to overload if you think it's going to improve performance significantly to do so

• You may want to extend StatsBase.predict rather defining your own predict method, unless there are going to be signature conflicts you can't resolve. Note MLJBase.predict is just an extension of StatsBase.predict. The consensus in the ML/Stats community seems to be to extend StatsBase.fit, StatsBase.fit! and StatsBase.predict rather than defining your own.

Let me know when you tag a release of EvoTrees with the MLJ interface and we'll add your package/models to the MLJ registry to make them MLJ-findable.

[jeremiedb]

Thanks for these precisions.

• I've removed the unecessary predict_X funcitons and extent the StatsBase.predict.

• No Plots, Revise and BenchmarkTools dependency in src or tests.

• Eachrow removed.

• Moved the Count from Deterministic to Probabilistic (Poisson distribution).

Also made a fix the resulted in poor binning with categorical inputs. There are definitly improvements to be made for input format management, as it currently expects Float64 and not necessarily handle well disparate types.

Finally, the CI tests seem to have issues with the SpecialFunctions dependencies. Not sure about the cause, but noticed couple of version restrictions on MLJBase, notably on Distributions and prettytables. Are these necessary, or is it expected that more recent versions be accepted by MLJBase?
I notably got issues with importing MLJ: UndefVarError: FiniteOrderedFactor not defined, so I have a doubt if I got the package compatibility right.

• Release 0.4.0 is now available.

[ablaom]

I did a quick test of the Poisson and it looks good.

Distributions and PrettyTables will get [compat] updates in the next release of MLJBase.

Any day now we will be announcing a new package MLJModelInterface, which you will be able to use instead of MLJBase as your dependency (testing will still require MLJBase, or MLJ). This package is super light, with only ONE dependency, ScientificTypes, which will itself be tiny and have ZERO dependencies. This ought to rule MLJBase out as a source of any dependency issues you are having. Would you mind waiting for this release before we register your package?

re: UndefVarError: FiniteOrderedFactor not defined This sounds like a very old version. I've just installed EvoTrees and MLJ in a fresh environment and did not encounter any problems. Here's what got installed:

(junk) pkg> st -m
    Status `~/Dropbox/Julia7/MLJ/MLJ/sandbox/junk/Manifest.toml`
  [7d9fca2a] Arpack v0.4.0
  [68821587] Arpack_jll v3.5.0+2
  [b99e7846] BinaryProvider v0.5.8
  [324d7699] CategoricalArrays v0.7.7
  [3da002f7] ColorTypes v0.8.1
  [34da2185] Compat v2.2.0
  [ed09eef8] ComputationalResources v0.3.0
  [a8cc5b0e] Crayons v4.0.1
  [9a962f9c] DataAPI v1.1.0
  [864edb3b] DataStructures v0.17.9
  [e2d170a0] DataValueInterfaces v1.0.0
  [b4f34e82] Distances v0.8.2
  [31c24e10] Distributions v0.21.12
  [ffbed154] DocStringExtensions v0.8.1
  [f6006082] EvoTrees v0.4.1
  [1a297f60] FillArrays v0.8.4
  [53c48c17] FixedPointNumbers v0.7.1
  [59287772] Formatting v0.4.1
  [41ab1584] InvertedIndices v1.0.0
  [82899510] IteratorInterfaceExtensions v1.0.0
  [682c06a0] JSON v0.21.0
  [7f8f8fb0] LearnBase v0.2.2
  [30fc2ffe] LossFunctions v0.5.1
  [add582a8] MLJ v0.8.0
  [a7f614a8] MLJBase v0.10.1
  [d491faf4] MLJModels v0.7.2
  [03970b2e] MLJTuning v0.1.2
  [e1d29d7a] Missings v0.4.3
  [6f286f6a] MultivariateStats v0.7.0
  [4536629a] OpenBLAS_jll v0.3.7+5
  [efe28fd5] OpenSpecFun_jll v0.5.3+1
  [bac558e1] OrderedCollections v1.1.0
  [90014a1f] PDMats v0.9.11
  [d96e819e] Parameters v0.12.0
  [69de0a69] Parsers v0.3.11
  [08abe8d2] PrettyTables v0.6.0
  [92933f4c] ProgressMeter v1.2.0
  [1fd47b50] QuadGK v2.3.1
  [3cdcf5f2] RecipesBase v0.7.0
  [189a3867] Reexport v0.2.0
  [ae029012] Requires v1.0.1
  [79098fc4] Rmath v0.6.0
  [321657f4] ScientificTypes v0.5.1
  [a2af1166] SortingAlgorithms v0.3.1
  [276daf66] SpecialFunctions v0.9.0
  [90137ffa] StaticArrays v0.12.1
  [2913bbd2] StatsBase v0.32.0
  [4c63d2b9] StatsFuns v0.9.3
  [3783bdb8] TableTraits v1.0.0
  [bd369af6] Tables v0.2.11
  [2a0f44e3] Base64 
  [ade2ca70] Dates 
  [8bb1440f] DelimitedFiles 
  [8ba89e20] Distributed 
  [9fa8497b] Future 
  [b77e0a4c] InteractiveUtils 
  [76f85450] LibGit2 
  [8f399da3] Libdl 
  [37e2e46d] LinearAlgebra 
  [56ddb016] Logging 
  [d6f4376e] Markdown 
  [a63ad114] Mmap 
  [44cfe95a] Pkg 
  [de0858da] Printf 
  [3fa0cd96] REPL 
  [9a3f8284] Random 
  [ea8e919c] SHA 
  [9e88b42a] Serialization 
  [1a1011a3] SharedArrays 
  [6462fe0b] Sockets 
  [2f01184e] SparseArrays 
  [10745b16] Statistics 
  [4607b0f0] SuiteSparse 
  [8dfed614] Test 
  [cf7118a7] UUIDs 
  [4ec0a83e] Unicode 

[jeremiedb]

Sounds like a good idea to opt for a lightweight interface, reducing the potential frinctions point for integration will certainly be beneficial to the onboarding of more models.
Sure, we can wait the new interface prior registering the package.

In the meantime, Gaussian probabilistic has also been integrated along histogram subtraction.

Please ping me when MLJModelInterface is ready.

[ablaom]

Sorry for the delay. Good to hear about the progress. We need to add some metrics to MLJ for your probabilistic predictors 😄

MLJModelInterface is now ready. Basically the deal is that your package module does import MLJModelInterface (or using MLJModelInterface) where before you would do import MLJBase. In testing you need using MLJBase just as before.

edit In your Project.toml file MLJBase moves from [deps] to [extras] and [targets] and you add MLJModelInterface to [deps].

If you refer explicitly to ScientificTypes in your package module, that can stay, but I expect all that you need is re-exported by MLJModelInterface. ScientificTypes (the lone dependency of MLJModelInterface) is super light-weight - basically just types. The MLJ scitype convention now lives in the new MLJScientificTypes package (which MLJBase imports) but this probably should not concern you.

Links to detailed instructions are here but this is mostly stuff you know. As you are one of the first third party packages to try this out, there may be some unexpected issues, but fingers-crossed it all works. This has been a substantial project for us, mostly the work of @tlienart .

[jeremiedb]

Integration with MLJModelInterface went fairly well. It's now available through latest EvoTrees version: 0.4.3.

Only glitch is that predict_median seems not to to be exported. My guess is that it stems from being missing from src/operations.jl in MLJBase.jl:
https://github.com/alan-turing-institute/MLJBase.jl/blob/4c9a32d617f4377a40d58b6f7a4dd81b9bf221fb/src/operations.jl#L24

Otherwise, only littly bug is about compatibility with previous Julia versions (1.0 - 1.2). Travis complains about SpecialFunctions being subject to an explicit requirement for 0.10 although I validated that Distributions and StatsFun, the 2 packages having SpecialFunctions as dependencies, had requirements for 0.8, 0.9 and 0.10. Any idea how that can be fixed?

[tlienart]

Re predict_median, thanks for spotting this, I'll fix this asap. (Done, pending release MLJBase 0.11.7)

With respect to your dependency problem, could you try removing your Manifest.toml from the repo (put it in your gitignore, you shouldn't leave it there) and see if it helps CI pass?

Edit: I can confirm that EvoTrees#master tests pass locally on Julia 1.0 on my machine so the Manifest.toml is likely the culprit

Edit2: PR https://github.com/Evovest/EvoTrees.jl/pull/41 fixes this, passes tests, and also replaces an isnothing (not Julia 1.0 compliant).

[ablaom]

I was not able to get a working registration of your models because the load_path trait (which you used to implement) is not explicitly set, and so falls back to a wrong value. Please add these for each model as in the following example:

MLJModelInterface.metadata_model(EvoTreeRegressor,
    input=MLJModelInterface.Table(MLJModelInterface.Continuous),
    target=AbstractVector{<:MLJModelInterface.Continuous},
    weights=false,
    descr=EvoTreeRegressor_desc,
    path = "EvoTrees.EvoTreeRegressor")                         # <------------- NEW 

[jeremiedb]

Just pushed the corrections for the path in metadata_model and submitted a 0.4.5 version to general registry. Thanks for the guidance!

[ablaom]

Working locally for me now 🚀 :

julia> using MLJ
julia> models(m -> m.package_name == "EvoTrees")
4-element Array{NamedTuple,1}:
 (name = EvoTreeClassifier, package_name = EvoTrees, ... )
 (name = EvoTreeCount, package_name = EvoTrees, ... )     
 (name = EvoTreeGaussian, package_name = EvoTrees, ... )  
 (name = EvoTreeRegressor, package_name = EvoTrees, ... ) 

julia> model = @load EvoTreeRegressor verbosity=1
import EvoTrees ✔
import EvoTrees.EvoTreeRegressor ✔
EvoTreeRegressor(
    loss = EvoTrees.Linear(),
    nrounds = 10,
    λ = 0.0,
    γ = 0.0,
    η = 0.1,
    max_depth = 5,
    min_weight = 1.0,
    rowsample = 1.0,
    colsample = 1.0,
    nbins = 64,
    α = 0.5,
    metric = :mse,
    seed = 444) @ 8…30

julia> X, y = @load_boston;
julia> evaluate(model, X, y)
Evaluating over 6 folds: 100%[=========================] Time: 0:00:00
┌───────────┬───────────────┬─────────────────────────────────────┐
│ _.measure │ _.measurement │ _.per_fold                          │
├───────────┼───────────────┼─────────────────────────────────────┤
│ rms       │ 6.06          │ [3.81, 4.9, 6.65, 6.06, 8.75, 4.91] │
└───────────┴───────────────┴─────────────────────────────────────┘
_.per_observation = [missing]

@jeremiedb After https://github.com/JuliaRegistries/General/pull/9835 is merged you can do your own testing and please close this issue if satisfied.

Further issues concerning MLJ integration could be opened on EvoTrees.jl, thanks.

[jeremiedb]

Looks good on my end as well. Thanks for your support!