Addons: Implement Weight Decay Optimizers

[seanpmorgan]

Per the RFC, we need to move Weight Decay Optimizers from contrib to addons:

• [x] DecoupledWeightDecayExtension
• [x] AdamWOptimizer
• [x] MomentumWOptimizer

This will involve inheriting from base Keras optimizer, modifying the code to match those APIs, and modifying test cases to run in TF2.

[PhilJd]

Hi,
I'd look into this if nobody else has already started working on it ;)
Also, it seems someone added ShampooW in the meantime, should I move that over as well?

[seanpmorgan]

@PhilJd That'd be great... we'd very much appreciate the help!

We're still very early on with setting up contribution guidelines (tests, docs, etc.) so bare with us during review. Also regarding ShampooW, we had not slated the Shampoo optimizer to be moved so that's why the weight decay version is not tracked in this issue.

[tabshaikh]

@PhilJd Are you still working on this issue?

[PhilJd]

@tabshaikh I've postponed this for a while as I wanted to wait until the CI is up and running. I'll take a look at it tomorrow ;)

[lgeiger]

Looks like CI is up and running now. @PhilJd are you still working on this issue?

[armando-fandango]

@lgeiger @PhilJd Best way is to create an issue and assign to yourself, and if you are not working on it anymore then un-assign yourself :-) That way we can resolve who is working on what in a distributed way :-)

[PhilJd]

Sorry, the last few weeks have been extremely busy and to make matters worse my machine broke. Now I'm set up again, I've adapted the optimizer, only tests are still missing. I'll be able to finish and open the pull request tomorrow, finally :)
Sorry again!

[PhilJd]

I'm still fighting import errors of the built addons (_image_ops.so not found), so this might be delayed until tomorrow :(

[seanpmorgan]

@PhilJd so that occurs when trying to import tfa (e.g. running a test) using just the repository source code. You'll need to compile the C++ code ahead of time so those .so files can be found. We do this by testing with bazel. Check out the contributing readme to see the recommended way to run tests.