Keras: Is there any advanced optimization methods in keras other than SGD?

[ghost]

I am talking about methods like stochastic average gradient or Stochastic Variance-Reduced Gradient?
did anyone use it before in his projects? How?

• [x] Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

• [ ] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

• [x] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

• [ ] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

[bstriner]

There are many keras optimizers but plenty of research hasn't been implemented yet. Personally, I would like to see LBFGS at some point. If there is an optimizer that seems worthwhile, post the paper and lets talk about how to build it. If the new contrib repository comes together, it should be really easy to add new optimizers.

[ncullen93]

I built a interface to use scipy.optimize.minimize (LBFGS, etc) in keras.. was completely shot down when I tried to submit a pull request lol. Keras lacks a real democratic contribution process and thus code/examples that are useful only in typical computer vision tasks tend to be all that is included. So don't hold your breath on other optimizers being incorporated.

Still needs to some work, and you can't use it exactly like the regular optimizer (since LBFGS has no such thing as "batch size"). I am also quite sure it breaks on non-sequential and/or multi-input/output models. You'll have to read through the code, but it's clean. Unfortunately I cant contribute much more beyond this. Best of luck
custom_optimizers.py.zip

for instance, note that you have to actually pass the model, the data, and nb_epoch into the optimizer - e.g. opt = ScipyOpt(model=model,x=xtrain,y=ytrain, nb_epoch=500) . This is much different of course, but necessary. Then you compile the model as normal and you run fit() as normal, but it will ignore the data and nb_epoch arg passed in to the fit() function.

[ehknight]

To anybody who wants to use an optimizer like L-BFGS, you might want to try wrapping the solutions found here in a TFOptimizer wrapper.

[jbkoh]

Hi, @hyperdo

It's been a while, but do you have any sharable code combining tf.contrib.opt.ScipyOptimizerInterface with TFOptimizer? I am wondering which loss function I should use this with CRF.

[redouanelg]

Hi @jbkoh, did you succeed in using tf.contrib.opt.ScipyOptimizerInterface with TFOptimizer? would be nice if you can share an example

[jbkoh]

@redouanelg Nope, It's been a while, I think I didn't go down the road much. I just used RMSProp for the case.

[zfang-slim]

Has l-BFGS implemented in Keras now? I think it should not be a complicated task.