Addons: Add "ReLU6, HardSwish, KAF, ELiSH, HardELiSH"

[marload]

Describe the feature and the current behavior/state.

These activation functions are required by communities and are commonly used by up-to-date architectures.

• ReLU6
• HardSwish
• KAF
• ELiSH
• HardELiSH

Relevant information

• Are you willing to contribute it (yes/no): YES
• Are you willing to maintain it going forward? (yes/no): YES
• Is there a relevant academic paper? (if so, where): YES

ReLU6
HardSwish
KAF
ELiSH, HardELiSH

• Is there already an implementation in another framework? (if so, where): torch, tf.nn
• Was it part of tf.contrib? (if so, where): NO

Which API type would this fall under (layer, metric, optimizer, etc.)

tfa.activation

Who will benefit with this feature?

Researchers need up-to-date Activation

Any other info.

[AakashKumarNain]

relu6 and swish are already in TF core. I haven't heard about any of the others. Can you point out any papers using them?

[marload]

As far as I know, HardSwish is not in Core. I think the link I attached will help you explain other Activation Function.

Thank You!

[bhack]

HardSwish Is in Tensorflow lite

[AakashKumarNain]

Thanks for the info. IMO, it is okay to add new activation functions but we also need to check the coverage, like how many times the paper for a corresponding activation has been cited. If the coverage for a any functionality is very low, it doesn't add much value to the ecosystem but rather increase the burden of maintenance.

Anyways, that's just my opinion. Let's hear what other maintainers have to say about this. cc: @seanpmorgan @WindQAQ

[bhack]

@AakashKumarNain we was thinking something similar in https://github.com/tensorflow/community/pull/241/files#diff-24cf1a9d17b0988437751c2472f2c296R13 but it is still not expanded.
Probably we could expand on that and just lower the barrier for TFA.

[AakashKumarNain]

@bhack I agree that the barrier for TFA should be low as compared to core TF. But we need to decide that factor IMO to make it more clear to the contributors

[bhack]

Yes what I meant is that if we could clear up this in the upstreaming process it could be easier to define for TFA (or the other way around).