Addons: Add Novograd optimizer (TF2)

[valendin]

Describe the feature and the current behavior/state.
This is a kind request to add an implementation of the Novograd optimizer as per the NVidia OpenSeq2Seq project: https://nvidia.github.io/OpenSeq2Seq

Relevant information

• Are you willing to contribute it (yes/no): yes (testing)
• Are you willing to maintain it going forward? (yes/no): no
• Is there a relevant academic paper? (if so, where): https://arxiv.org/abs/1805.10387
• Is there already an implementation in another framework? (if so, where): https://github.com/NVIDIA/OpenSeq2Seq/blob/master/open_seq2seq/optimizers/novograd.py (TF 1 only)
• Was it part of tf.contrib? (if so, where): not that I'm aware

Which API type would this fall under (layer, metric, optimizer, etc.) optimizer

Who will benefit with this feature? everyone using seq2seq models

Any other info.

[WindQAQ]

Thanks for bringing this up and feel free to open a PR!

[shreyashpatodia]

I will be happy to take this on @valendin @WindQAQ. Is anyone currently working on this?

[valendin]

@shreyashpatodia not that I’m aware of!

[shreyashpatodia]

I can work on adding this sometime in the next two weeks.

[shreyashpatodia]

@valendin I have opened PR #836 for this.

Could you check the logic and verify my understanding/implementation is correct?

[valendin]

@shreyashpatodia I've imported the standalone novograd.py file into my own project and ran into the following issues:

line 24, in <module> @tf.keras.utils.register_keras_serializable(package="Addons") AttributeError: module 'tensorflow_core.keras.utils' has no attribute 'register_keras_serializable'

Commenting out the decorator makes the code compile but then when calling the fit function I get the following error:

W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Invalid argument: indices[3] = 4 is not in [0, 4) [[{{node Novograd/Novograd/update/GatherV2}}]]

[shreyashpatodia]

@valendin I don't think the decorator is a problem because the code compiles with the decorator locally for me and on the ci test for mac on my PR.

For the second one can you tell me how you run the optimizer and what you supply as parameters? Could you also try running the tests and telling me if they fail with the same error?

I think it would be best if you tried to compile tfa from my PR and see if you can get things to work.

[WindQAQ]

Closes via #836.