Addons: Add a tutorial for MovingAverage optimizer

[Squadrick]

We need a tutorial that shows how to use MovingAverage optimizer along with the custom callback for saving: AverageModelCheckpoint. Can alternatively use StochasticAverage.

Currently blocked by #926

[hyang0129]

This sounds interesting.

Is there maybe a paper somewhere showing how it performs better than a regular optimizer?

[LexusH]

Hey @Squadrick, I'm a beginner on this project that would love to help with your issue! Any tips on how to get started would be much appreciated

[Squadrick]

You can take a look at some of the other tutorials we have here. It includes the template (_template.ipynb) that we follow.

You can show a model being trained on MNIST or fashion MNIST with the MovingAverage optimizer and AverageModelCheckpoint callback.

[abhichou4]

Woking on the issue. MovingAverage works fine but AverageModelCheckpoint doesn't seem to be there, even with tfa-nightly.
https://colab.research.google.com/drive/1IxqfhbkpNN6P1guzdW7q5ZEdPoqi7sle
Is something wrong with this?

[LexusH]

I have a MNIST model trained using the MovingAverage optimizer right now so progress is good! Looking at the tutorials for how to do a callback so I can hopefully add that in shortly.

[seanpmorgan]

Woking on the issue. MovingAverage works fine but AverageModelCheckpoint doesn't seem to be there, even with tfa-nightly.
https://colab.research.google.com/drive/1IxqfhbkpNN6P1guzdW7q5ZEdPoqi7sle
Is something wrong with this?

Interesting. It looks like TFA is installed by default on Colab now:
https://colab.research.google.com/drive/18wjcYoNfQi-yc-YFSZvq4pKePy164O29

The version is 0.6.0 which did not have any callbacks (only the stub). This makes it a bit tricky since you'll need to install tfa-nightly and restart the runtime.

@lamberta Do you know a contact for Colab who we could talk to about this. Currently the loaded version 0.6 is pretty out of date and it would be nice to know how we can update this.

[lamberta]

Well ... cool it's there by default now :)

There's probably an internal version that Colab is using. @craigcitro on the Colab site.

@tomerk Is there a better way to regularly sync these?

[craigcitro]

Feel free to file a go/colab-bug on us internally with questions.

[seanpmorgan]

Well ... cool it's there by default now :)

@lamberta Yes this is great news that TFA is installed by default! :smile: Just want to see if we can help keep it up to date.

@craigcitro is https://github.com/googlecolab/colabtools/issues what you're referring to?

[craigcitro]

@seanpmorgan yep, that works.

[failure-to-thrive]

The correct way to upgrade TFA on Colab:
!pip install -t /tensorflow-2.1.0/python3.6/ --upgrade tfa-nightly
There is no need to restart the runtime unless you've already imported TFA.