Mlj.jl: Improve the tuning strategy interface

[ablaom]

Currently the only tuning strategy supported by MLJ is Grid for grid search. Improvements to the tuning strategy API would make implementing new strategies easier and allow for more general spaces of models to search.

I am not an expert on every tuning strategy out there, but I have reviewed the use-cases below and believe these are accommodated by the new design I am now putting forward for feedback. To save time later, the proposal is written as a manual for implementing new tuning strategies, as if the API were already in place, but the details are malleable.

I suggest that if you don't want to wade through the proposal in detail, you at least check your very-special-yummy-bestest of tuning strategies appears on the list below and I will endeavour to check it out. Please provide a link to a lucid exposition of your addition.

I hope to start implementation in a week, Thanks

Some notes

• The impact on the user syntax for setting up and executing a tuning task is not being changed (with the exception that a TunedModel is now iterative in all cases). Only more tuning strategies and "range" spaces are to be ultimately made available through the proposal. So no new workflows here.

• The only "range" spaces available out-of-the-box are: (i) explicit lists of models; (ii) Cartesian ranges (lists of the one-dimensional ParamRange objects); and (iii) Cartesian ranges "annotated" with resolutions, pdfs, or other per-dimension parameters. The design of more complicated spaces (tree-structured, etc) can be the subject of future discussions. The implementer just needs to define a setup(tuning::MyTuningStrategy, model, range::RangeType) method for each RangeType she wants to support.

• A subtle design decision concerns user-interface points for hyperparameters specific to a tuning strategy, such as the resolution of a grid along a specific dimension (ie model hyperparameter). There are two places such parameters can go: (i) fields of the strategy type, or (ii) part the range object. In the present proposal, the fields of the strategy type are reserved for parameters that "do not refer to specific models or specific model hyperparameters". So, under this principle, a default resolution parameter is a field of the type, but a dimension-specific resolution (or pdf) is part of the range.

• Every one-dimensional OrdinalRange is to be given two new fields, unit and origin, which for bounded ranges default to half the range width and the range midpoint. For unbounded ranges, these should be specified by the user on construction. In that way, it makes sense, for example, to construct a default clipped Guassian on bounded and unbounded ordinal ranges, for use in random searches, etc.

The interface does not directly address early stopping, but the general wrapper for iterative methods proposed in #139 should take care of this.

Use cases

Feel free to add to this list:

• [x] search a list of explicitly specified models list = [model1, model2, ...]
• [x] grid search
• [x] random search
• [x] simulated annealing
• [x] Bayesian optimization à la Gaussian processes
• [x] structured tree Parzen estimators
• [x] multi-objective (Pareto) optimization
• [x] genetic algorithms
• [x] AD-powered gradient descent methods

[tlienart]

Looks good to me, minor comments:

• a few typos / things that are unclear which I suggested fixes for
• this new package https://github.com/JuliaDiffEq/QuasiMonteCarlo.jl is interesting in that it could easily extend the randomized tuning (and it's lightweight). Things like Sobol/Latincube are sometimes seen in HP tuning as it fills the space better than uniform random sampling

The possibility to hook a user-defined mechanism is great and/or to interface with packages providing hyperparam optim techniques.

• _User should be able to specify a stream to which the history is written in each iteration._ this may lead to issues with threaded/parallelised implementations (?)

One small (possibly naive) question: how do you envisage restarts? you seem to specify that the history should be saved properly along the way so that we don't risk losing everything if a processor crash for instance but can you then re-start the tuning at the last point where we had something? i.e. is the history meant to contain sufficient information for this?

[baggepinnen]

I second the relevance of latin hypercube sampling, it's often my preferred hyper optimizer if it's applicable. Grid search is often by far the worst strategy, random search is a much stronger baseline. Grid search also becomes unusable when the number of parameters to tune becomes larger due to the exponential growth of the number of points to evaluate.

I'm not sure if the new interface for tuning strategies is live or not? If it is, I would be keen on getting a random sampler up and going. Currently I'm using my own package Hyperopt.jl to do the tuning, but it would be nice to use MLJ for MLJ workflows.

It would also be nice to have a "default TunedModel" that contains reasonable ranges for all hyperparameters that one might want to tune.

[ablaom]

@baggepinnen The Hyperopt.jl package looks nice. I will be playing around with it more shortly.

How difficult do you think it would be for someone to implement the tuning strategy API outlined above (not yet live) for the tuning strategies already in Hyperopt.jl?

[baggepinnen]

• Random sampling would be very easy to implement.
• Latin hypercube would be very easy to implement if the parameter ranges have a natural ordering, e.g., are real numbers, integers etc. For categorical parameters, one can consider Categorical latin hypercubes, supported by Hyperopt and the underlying LatinHypercubeSampling.jl
• Bayesian optimization and other similar methods are in general trickier to implement by relying on existing implementations since they often assume continuous search spaces. Indeed, it is hard to specify a meaningful kernel (prior etc.) in categorical dimensions.
• Hyperband requires its own set of features such as being able to resume training and that there exist some parameter that corresponds to computational effort, such as number of iterations, allowed wall-clock time etc.

[ablaom]

Sure, I understand that different tuning strategies will handle different kinds of search space (called a range object in the API). My question is more: How difficult would it be to refactor the Hyperopt.jl code so that it implements the API outlined at https://github.com/ablaom/MLJTuning.jl (implements the methods, setup, models!, and so forth)? I am already reasonably confident that this is possible for a large class of strategies (with possibly the exception of hyper band). I'm wondering about the difficulty in doing so.

[ablaom]

discussion on :default grids for model hyperparams and specifying total grid length instead of per-parameter lengths: https://github.com/alan-turing-institute/MLJ.jl/issues/416#issuecomment-571329340

[Sinansi]

Hello all, sorry to interrupt. I am using MLJ.jl grid search and it takes hours to complete even with few parameters. So I was wondering if there are any plans to merge or interface Hyperopt.jl random search with MLJ.jl.
Thank you!

[azev77]

@Sinansi I agree. I don’t even know how to automate HP tuning for a large set of models w/o specifying the hyper parameter ranges for each model.
Do you know how to do this?

[baggepinnen]

You can use Hyperopt directly in the meantime, example

# ho = @phyperopt for i=36, sampler=RandomSampler(), n=15:20, bf=LinRange(0.2,0.6,16), nsf=2:20, msl=1:15, pp=(false,true)
#     tree_model.n_subfeatures = nsf
#     tree_model.min_samples_leaf = msl
#     tree_model.post_prune = pp
#     forest_model = EnsembleModel(atom=tree_model, n=n, bagging_fraction=bf)
#     mean(1:3) do i
#         e1 = MLJ.evaluate(forest_model, Xt, label, resampling=CV(nfolds=3, shuffle=true), measure=MLJ.auc, verbosity=0, check_measure=true)
#         e1.measurement[]
#     end
# end
# plot(ho)

(sorry for the commented code, I'm on my phone)

[Sinansi]

@Sinansi I agree. I don’t even know how to automate HP tuning for a large set of models w/o specifying the hyper parameter ranges for each model.
Do you know how to do this?

@azev77 thanks for asking, but honestly I am stuck with the same issue. I see most analysts check for the default parameter values, then specify an upper and lower boundaries around it. But I couldn't find a rule that specifies the range or search space around the default parameter values. If you are aware of a rule please let me know.

Additionally, you mentioned something about automating HP tuning. Are you referring to the new field of Automated Machine Learning? As far as I know, Auto ML also automate HP tuning, but I haven't explored this new technology yet.

[ablaom]

@azev77 There is currently no way to tune a hyperparameter without specififying a range for that parameter. (To be clear, specifying a one-dimensional range does not mean specifying a one-dimensional grid. It means specifying an upper and lower bound and a scale - and bit more if the range is unbounded.)

Incidentally the MLJ docs on tuning should be up-to-date, with some updates to MLJTutorials imminent.

I have proposed that in the future it be possible to specify only the name of the hyper parameter, in which case a default range is used. Such functionality depends on recording in the model registry default ranges for these parameters. There is already the facility for recording this information (the model interface author implements the hyperparameter_ranges trait) but this has not been done in the case of a single model.

@Sinansi Sorry to hear about the slow progress of your tuning. If you believe this is an MLJ issue (rather than a limitation of grid search in general) then please open a new issue with some more detail (eg, benchmarks) and I will be happy to investigate.

@Sinansi don't believe there is any magic rule that turns a default hyperparemeter value into a range about that value to search. The way in which models depend on a given hyperparameter varies wildly. This is generally decided on a case-by-case basis, and is more art/experience than science. As I say above, we hope to encode some of this experience in the model metadata but this will take some time.

I agree that more tuning options are a priority. At the very least I will try to see if I can't push random search along. An integration of HyperOpt.jl certainly sounds like a good idea. Happy to support efforts in this direction. The new tuning API, which is here, is pretty well documented. As it is complete, I am closing this issue. Please feel free to open new topic-specific issues around further enhancements around this area. See for example this discussion #416 on visualizing tuning results.