Addons: Implement External Optimizers

[seanpmorgan]

Per the RFC, we need to move external optimizers from contrib to addons:

This will involve inheriting from base Keras optimizer, modifying the code to match those APIs, and modifying test cases to run in TF2.

• [ ] ExternalOptimizerInferace
• [ ] ScipyOptimizerInterface

[kyleabeauchamp]

This is partially related, but it might also be good to have a docstring or notebook somewhere that just showcases the most idiomatic way to minimize a tf.function in 2.0. In particular, now that we have a function-first API, it may make sense to showcase some simple "low-level" scipy.optimize use cases. Obviously this is somewhat orthogonal to implementing a Keras-compliant optimizer, but still useful I think. (and it's not clear where else to report this request).

[seanpmorgan]

@facaiy @WindQAQ This is a proposed contrib move that I've been questioning. Its not that I don't find it useful... but to me it seems like it could almost be a separate repository. I think its hard to say that this will fit into a well established API pattern, and it requires us to bring in a new dependancy... but I wanted to get your thoughts.

[facaiy]

At least to me, I don't think it's useful. How about let's implement it in the future if requested by enough users?

[WindQAQ]

I find some references in tensorflow probability. TFP has offered some widely used optimization methods like bfgs, lbfgs and differential evolution, which also lies in scipy.optimize. In this case, I do not think we still have to bridge the scipy interface.
https://www.tensorflow.org/probability/api_docs/python/tfp/optimizer

[seanpmorgan]

Closing this issue unless there is a convincing argument otherwise or a large request from the community