Addons: Feature request : Add more Scipy Optimizer

[nicaogr]

Hi,

Could it be possible to wrap the scipy.optimize.basinhopping and scipy.optimize.anneal the same way the tf.contrib.opt.ScipyOptimizerInterface allow us to use method as LBFGS with Tensorflow to minimize a function ?

Thanks

[elwinmartin]

Hi!

I'd like to work on this if nobody is looking at this right now.

[mirca]

Hey, I'm trying use the 'dogleg' method in tf.contrib.opt.ScipyOptimizerInterface.
I'm able to compute the hessian of my objective functions with tf.hessians (which returns a list of tensors). But then when I pass that hessian to tf.contrib.opt.ScipyOptimizerInterface I get the following error message:

...
~/anaconda3/lib/python3.6/site-packages/scipy/optimize/_trustregion.py in hess(self)
     56         """Value of hessian of objective function at current iteration."""
     57         if self._h is None:
---> 58             self._h = self._hess(self._x)
     59         return self._h
     60 

~/anaconda3/lib/python3.6/site-packages/scipy/optimize/optimize.py in function_wrapper(*wrapper_args)
    290     def function_wrapper(*wrapper_args):
    291         ncalls[0] += 1
--> 292         return function(*(wrapper_args + args))
    293 
    294     return ncalls, function_wrapper

TypeError: 'list' object is not callable

Is the 'dogleg' method supported at all?
Thanks!

[LucaZampieri]

@mirca
I think you should pass a function the computes the Hessian of your loss function at point x to tf.contrib.opt.ScipyOptimizerInterface, not directly the list of tensors. I am trying to use it now as well.

[benjiebob]

I'm very interested in being able to implement a dogleg optimizer in tensorflow. Anyone got some code which I could use as a starting point?

[endolith]

@nicaogr

scipy.optimize.basinhopping and scipy.optimize.anneal

The documentation says "anneal is deprecated", "use basinhopping instead", for the record.

[YeeCY]

Here is another tricky replacement I found on StackOverflow. We can include both Tensorflow GPU acceleration and scipy optimizer at the same time. Hopefully, it will be helpful.

https://stackoverflow.com/questions/50575332/escaping-local-minimum-with-tensorflow/50675655

[dynamicwebpaige]

Though adding more optimizers from SciPy is outside the scope of TensorFlow Core, basinhopping and anneal might be welcome additions to optimizers in TensorFlow Addons.

Transferring this issue to tensorflow/addons; @seanpmorgan for visibility.

[seanpmorgan]

So we had originally planned on moving SciPy optimizers to TFA (#3) but decided against it as it's difficult for us to maintain and there was not a substantial request from the community. Seeing as this issue is quite old I'm going to close this issue pending community requests.

Feel free to re-open if there is a substantial need for this and we can consider possible paths forward. These generally don't fit the Keras/TFA API so we would need to consider how this would fit.

[kyleabeauchamp]

There are some implementations of LBFGS in tensorflow probability that I have been able to use on TF2.0, so that's a good starting point for folks interested in that functionality.