Mlj.jl: Add more tuning strategies: bayesian, AD gradient descent

[mik3y64]

We have Grid and RandomSearch baked into MLJ. It would be great for MLJ to implement other tuning strategies including Bayesian optimization algorithms based on Gaussian processes and regression trees and AD-based gradient descent, attracting more users to MLJ and Julia machine learning ecosystem.

For Bayesian-based method, I came across Hyperopt.jl which internally uses BayesianOptimization.jl (which internally uses GaussianProcesses.jl). In deeper stack they utilize well-known Julia optimization libraries NLopt.jl and ForwardDiff.jl. For example BayesianOptimization.jl uses automatic differentiation tools in ForwardDiff.jl to compute gradients of the acquisition function ExpectedImprovement(). All these libraries are pure Julia. It would be great to integrate Hyperopt.jl or utilize the lower level libraries for implementing Bayesian-based optimization to more efficiently tune models in some cases.

I don't have much knowledge about AD-based gradient descent but it would be great to see them implemented in MLJ too.

[azev77]

We have discussed incorporating hyperopt.jl etc
Check out
https://github.com/alan-turing-institute/MLJTuning.jl

[ablaom]

@mik3y64 Thanks for that!

I'm going to close as these enhancements are on the road map and this issue is now cross-linked at the open issue referenced above.