Mlj.jl: Model wrapper for controlling iterative models.

[ablaom]

I would like a common interface for implementing "early stopping" of iterative methods, with possibly an option to adapt a "learning rate" or other hyperparameter as iteration progresses.

To buy into this interface (ie to get performance benefits) a model need only to implement the MLJBase.update method, so that increases to the iteration parameter do not trigger retraining from scratch (which they should do anyway). Also, any parameter, such as a "learning rate" that is to be adapted, should similarly not trigger retraining from scratch. An existing model that does this already (without the extra adaptive parameter) is EnsembleModel. @ayush-1506 and I are currently arranging this for the MLJ Flux models we are working on. Edit Done.

We could implement this as a model wrapper, just like TunedModel. Edit: Like the TunedModel, we would specify a resampling strategy from the ones that already exist for this purpose (eg, CV()), for specifying how a performance is estimated for purposes of deciding early stopping.

What do people think?

[darenasc]

Like in a general base wrapper for iterative methods? sounds good.

An alternative would be to pass an optional early_stopping parameter to the fit function and use it with the update method?

[ablaom]

Thanks for that!

I'd rather avoid passing parameters to calls to fit!. The philosophy in such cases has been to have an "instruction" hyperparameter for such models, but for then we need to ensure all model implementers of iterative methods use the same convention for naming this, and so forth. The advantage of my suggestion is that no such convention is necessary. All choices are "externalised" to the wrapper design.

We already have something like this in learning_curve!. You can generate a learning curve(s) with any choice of hyperparameter - it doesn't have to be the iteration parameter. If it is an "iteration parameter" (we never actually define this term and don't recally need do) then the function runs efficiently, assuming increasing that parameter (and leaving others alone) does not retrigger retraining from scratch (because there is a sensible overloading of the model update method).

I should add that if a sensible update method is not implemented, then we can get the same behaviour without this knowledge, it just takes longer. (Maybe should think of a way to expose this fact to the user?)

[vollmersj]

This sort of exists in LearningStrategies.jl.

[fkiraly]

I completely agree with @ablaom that "update" should be a full-blown interface point, rather than just having it within the interfaced models.
The argument that meta-astrategies that make use of learning curves etc are externalized I think makes this a canonical choice. Most importantly, perhaps, it would allow the common "inspect learning curve of neural network and eyeball and then do a black magic dance while inspecting the flight of the birds and the learning curve's entrails until you have found the right learning parameters" part of the deep learning workflow more reproducible and accessible to systematic automation.

I also remember a discussion about "update" somewhere else (other issue), but can't point my finger to it, so plead add issue no. if anyone remembers.

Anyway, I thin we ought to think carefully about:
(i) use cases. I think there are multiple significant use cases for an "update" interface point.
(ii) the method signatures, e.g., what is a "parameter" of update and what is a parameter of the method.

(i) use cases

There's multiple I can think of:

• on-line learning. Supervised learning in essence, the step of "add more training data and update the model fit". Only needs the "update" interface point.
• active learning. Give the method the feature/label pairs it asked for. Needs to be designed with a "query" in mind that might be a callback (?), or maybe a completely different interface point?
• Bayesian update. The model is fully Bayesian, and we use the method's on-board inference engine to update parameter (posteriors). This needs "update", and model parameters which are distributions(.jl)
• "one more iteration". Many - but by far not all - models rely on some internal optimizagion (SGD etc). One could use "update" to tell the model to continue "running", without ingesting more data.

The key issue that I see here is that the use cases rely to different extent on:

• continuing with model computation, or
• ingesting data

I think it's a valid question whether one should make these two interface points rather than one. Or, whether one should allow for multiple "updaters" which would then become first-order types along the lines of "myupdater : modelupdaterfor{mysupervisedlearner}".
Or, one could require the "update" function to cover the case where no additional data is passed?

(ii) hyper-parameters and input of the update
I agree with @ablaom that update-related meta-strategies should externalize parameter choices in a wrapper that calls update of the wrapped object.

However, "update" still needs inputs: if you think about the key case of neural networks, you could choose epoch and batch size in reference to a larger data set.
I'd leave thse with the wrapper, which has these as parameters? The wrapped update then sees the "right sized" dataset, sub-sampled from the larger training set, presented by the wrapper?

Is that what you had in mind, @ablaom ?

[ablaom]

@fkiraly In principle I think all of these use cases could be accommodated by the existing design (with a small question mark next to online-learning). However, I would suggest that only the case of fixed data (the last use case) is within the scope of core functionality. The active learning and online learning cases are significantly more complex because they require some sort of communication from the wrapper to the core algorithm about what part of the data that has been bound to the machine should actually be accessed at any give stage in the "progressive" computation. So some sort of convention about this has to be established, and the model implementer (provider of the core algorithm) has to understand this convention, and so forth.

@sjvollmer I think it is possible that LearningStrategies.jl could be used to save a little code, although the package itself is only about 400 lines. By using it we pay the price of slightly more abstraction and I'm not sure it's worth it. Some experimenting is in order: Can you use LearningStrategies.jl with MLJ to train a random forest along the lines in examples/ using a "patience" strategy. This strategy (which is not provided out-of-the-box by LearningStrategies) is the one that says you stop adding trees when the loss on a holdout set increases n consecutive times. Compare the code you write to the code you would write to do the same "by hand".

Edit: @ayush-1506 Could you play around with LearningStrategies.jl as suggested above?

[fkiraly]

@ablaom regarding online and active learning, I agree that this is an advanced interface question.

The main point that I was trying to make is, let's ensure interface back-compatibility, i.e., let's think about what "update" looks like so it's easy to extend to the more complex cases.
Both advanced cases will probably need an "update" interface point, plus sth else, and the former should probably be the same "update" interface point as in the simple.

The unresolved question I'd like to spell out is: should "one more iteration" be the same interface point as "one more data point" (or N more etc).

Regarding the advanced tasks, this is currently perhaps less importants, but have some random thoughts:
One way could be to wrap a data source in a model, e.g., a "data stream" data provider, or an "active learning" data provider, possibly with callbacks.

For online learning, we've had a discussion a while ago in #60, in relation to onlinestats, @joshday or @knutjaegersberg may want to comment.
There's also two sub-cases, reactive to "appearance of data trigger" and reactive to "user trigger". For the latter, a specific data provider or data source type may not be necessary.

[ablaom]

@fkiraly

update exists as a model method and is described in the documentation, along with how it fits in with the fitting of machines . See also the doc string for fit! below. If the existing model API will not serve as an interface point for "one more data point" (hopefully it will) then I don't think there's much we can do about it now. While we might add to the model API (in MLJBase) I am very reluctant to break anything there.

You raise some good questions. I will address them best I can at #60 when I have time, thanks.

[ablaom]

fit!(mach::Machine; rows=nothing, verbosity=1, force=false)

When called for the first time, call MLJBase.fit on mach.model and
store the returned fit-result and report. Subsequent calls do nothing
unless: (i) force=true, or (ii) the specified rows are different
from those used the last time a fit-result was computed, or (iii)
mach.model has changed since the last time a fit-result was computed
(the machine is stale). In cases (i) or (ii) MLJBase.fit is
called on mach.model. Otherwise, MLJBase.update is called.

[ablaom]

Basic proposal for IterativeModel wrapper:

To be realistic, I think we restrict performance estimates to
ones based on a Holdout strategy. We don't currently have a way to do
"incrementally" perform cross-validation or similar.

The wrapper specifies a range of values for a specified iteration
parameter, just like for learning_curve!.

Fitting the wrapped model will progressively train the atomic model on
train set and at each stage estimate the performanc on a test
set. This will build a history of performance, to which various
criteria can be applied. Formally, a criterion is just a
boolean-valued function of the history.

At each stage in the training process we apply one or more controls
specified by the user. A control consists of one or more criteria (to be combined disjunctively) and
an action. An action is either "stop" or "change some hyperparameter of the atomic model"
(e.g., a learning rate, or choice of optimizer).

I'm not sure how much of this can be implemented by building upon
LearningStrategies.jl. Perhaps @ayushh-1506 can comment after some
experimentation?

A simpler design option would be just to provide a few simple options
for stopping.

Side note: To buy into this interface (that is, for this inteface to
actually work efficiently) the model being wrapped must overload the
MLJBase update method appropriately. See previous comment for logic
governing update call by machine fit!. Specifically, the update
method should only force retraining from scratch (the fallback option)
if some hyperparameter other than the iteration parameter or some
"control" parameter, such as learning rate, has been changed since the
last call to fit or update. See EnsembleModel code at /src/ensembles.jl
for an example of this.

Formally, we have made no distinction between "control" parameters and
ordinary parameters. Since we have a safe fallback for update, which simply
treats all parameters as "non-control", I haven't been too concerned
with this.

How does this sound?

[fkiraly]

Sounds great, @ablaom .

In particular, I agree with:

• exposing "control" parameters via the same hyper-parameter interface points
• use of fit!as currently specified

I'd recommend however (thoughts?) that:

• abstract model performance evaluation methodology (simple hold-out, standard losses, machines/learners against data, nothing fancy) to be implemented first.
  Not sure whether that already exists? But it sounds like you would want to call that from within the model fitting/updating, rather than implementing from scratch. Just to have it as an interface point call that can be expanded, as opposed to custom code.
• a "control" object (struct type) is designed, with potential subcases being iteration-control, resample-control, evaluation-control, experiment-control, tuning-control, etc, not necessarily all to be implemented now.
  Should use the command pattern, if it's admissible to use the term in Julia...

[ablaom]

@fkiraly Thanks for the feedback.

Not sure whether that already exists?

Errrr, I believe evaluate! already does what we want, no?

julia> forest.model.n = 20
20

julia> evaluate!(forest, verbosity=3, resampling=Holdout())
┌ Info: Evaluating using a holdout set. 
│ fraction_train=0.7 
│ shuffle=false 
│ measure=MLJ.rms 
│ operation=StatsBase.predict 
└ Resampling from all rows. 
[ Info: Training Machine{DeterministicEnsembleModel{DecisionTreeRegressor}} @ 1…09.
[ Info: One hash per new atom trained: 
####################
7.564343735989894

julia> forest.model.n = 23
23

julia> evaluate!(forest, verbosity=3, resampling=Holdout())
┌ Info: Evaluating using a holdout set. 
│ fraction_train=0.7 
│ shuffle=false 
│ measure=MLJ.rms 
│ operation=StatsBase.predict 
└ Resampling from all rows. 
[ Info: Updating Machine{DeterministicEnsembleModel{DecisionTreeRegressor}} @ 1…09.
[ Info: Building on existing ensemble of length 20
[ Info: One hash per new atom trained: 
###
7.757009385251871

julia>?evaluate!

evaluate!(mach, resampling=CV(), measure=nothing, operation=predict, force=false, verbosity=1)

Estimate the performance of a machine mach using the specified
resampling strategy (defaulting to 6-fold cross-validation) and measure,
which can be a single measure or vector.

Although evaluate! is mutating, mach.model and mach.args are
preserved.

Resampling and testing is based exclusively on data in rows, when
specified.

If no measure is specified, then default_measure(mach.model) is used, unless
this default is nothing and an error is thrown.

[ablaom]

a "control" object (struct type) is designed, with potential subcases being iteration-control, resample-control, evaluation-control, experiment-control, tuning-control, etc,

Don't forget, it should be able to cook toast too.

Should use the command pattern, if it's admissible to use the term in Julia...

Thanks. If this translates into a Julia pattern and is appropriate, will use.

[fkiraly]

Don't forget, it should be able to cook toast too.

Toast-control, indeed the best thing since sliced bread.

Errrr, I believe evaluate! already does what we want, no?

Indeed it does!
What I was thinking about was a full benchmark orchestration interface, but I did miss the valid point that a model-local dispatch does the trick as well, and better.

[fkiraly]

But seriously, the "control" object discussion is perhaps worthwhile having.

It's a kind of input that appears in multiple places - and what I meant is not to have one object that does everything, but a common interface design.

[ablaom]

https://www.juliabloggers.com/julia-nomenclature/

[jpsamaroo]

I'm interested in adding active learning capabilities to MLJ so that I can train models from real-world streaming data like camera or microphone input. I assume this issue best fits that need? If so, can someone give me an example of what the update call to do active learning should look like from the user's perspective?

[ablaom]

Online/active learning is not currently supported by MLJ. The update method is generally for "more iterations" as opposed to "more data". Although this thread is obviously related, the more appropriate place to discuss this would be #60.

I think this one is a little way off (months), as I should probably be the one to implement the core design and my priorities are currently elsewhere. There'll probably be an new add_data method for models and this will have to be implemented for each model you want to fit online. Obviously, as not every model can do this, there won't be a fallback. There will be control wrapper very much like the one I have proposed in the current thread, and if you wanted to do both kinds of control, you would just chain the wraps.

[ablaom]

Some other things to keep in mind in this design: snapshots and callbacks.