Katib: Add CMA-ES based suggestion service.

[c-bata]

/kind feature

Describe the solution you'd like

The covariance matrix adaptation evolution strategy (CMA-ES) [1] is one of the most promising black-box optimization methods, and it has shown the best performance out of over 100 optimization methods for a variety of BBO problems [2].

In this issue, I propose to add CMA-ES based suggestion service. If approved, I'll send a pull request.

[1] N. Hansen. The CMA Evolution Strategy: A Tutorial. arXiv:1604.00772, 2016.
[2] Loshchilov, I., Schoenauer, M., and Sebag, M. Bi-population CMA-ES Algorithms with Surrogate Models and Line Searches. In Proceedings of the 15th annual conference companion on Genetic and evolutionary computation, pp. 1177–1184, 2013.

Anything else you would like to add:

I implemented CMA-ES in Python and Go.

• CyberAgent/cmaes is pure python CMA-ES implementation which only depends numpy.
  
  
  
  • This library will be adopted by Optuna as its official sampler (see https://github.com/optuna/optuna/pull/920).
  
  
• c-bata/goptuna provides pure Go CMA-ES implementation built on gonum/mat.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| feature | 0.99 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[c-bata]

I found #67 but seems to be inactive.

[gaocegege]

@c-bata Yeah. I'd appreciate it if you could help.

[c-bata]

@gaocegege Thank you for your reply!

Okay! I'm planning to implement CMA-ES based suggestion service by using Goptuna. Could you accept the dependencies with Goptuna?
https://github.com/c-bata/goptuna

Goptuna's CMA-ES sampler is well tested at GitHub Actions. I continuously checked the performance on some benchmark functions. Goptuna's CMA-ES implementation (purple) stands comparison with pycma (blue).

| Himmelblau | Rosenbrock |
| --- | --- |
| | |

See here for details.

[gaocegege]

Sure. Suggestions can be implemented by any language with GRPC support.

[johnugeorge]

Great. SGTM

You could use other algorithm implementations for reference.

[c-bata]

[FYI] I'm working on this at https://github.com/c-bata/katib/pull/1. I'll open a pull request after ready for review.

The branch contains huge file changes because this repository contains vendor/ directory. The main implementation is the following commit.

https://github.com/c-bata/katib/pull/1/commits/1888317abbf4ef30d26573941277a03ab413d3e0

[gaocegege]

Thanks.