Tacotron-2: Use bias or not as variable?

[begeekmyfriend]

I would like to ask why you did not use bias in attention convolution and dense layer while added one in _location_sensitive_score that in my experience does help convergence https://github.com/keithito/tacotron/issues/170#issuecomment-390873725.

[Rayhane-mamah]

So, about the bias variable that already exists, mathematically speaking: W1 + b1 + W2 + b2 + W3 + b3 = W1 + W2 + W3 + b with b = (b1 + b2 + b3). So disabling bias for all FF layers and making a separate bias vector should do the trick while reducing number of parameters slightly.

As for the bias during convolution layer, I actually have not thought of that, I'll give it a shot and see how it goes! Thanks for pointing that out :)

[begeekmyfriend]

Yes, but you did add bias variable in _location_sensitive_score and it works on convergence, right?

[Rayhane-mamah]

Yes the variable b_a does help convergence, and like I said earlier, it makes the model learn one bias vector instead of 3 separate biases for 3 projection layers so technically usimg b_a is equivalent to using bias for all attention projection layers (memory, query and location features).

It would be interesting to see the effects of bias for the convolution though.

[Rayhane-mamah]

About the bias for the variance, after a first experiment, it appears to be helping alignments, Also noticed that the fmin can affect the model negatively (like you said) if the speaker pitch is low (which masks information from the output and makes the learned distribution wrong).

With both masks, a modified window size, and a decoder with 1/2 the size of current repo, attention gets learned slightly earlier than 10k (I only checkpoint once in 5k so I can't pinpoint the exact step attention started forming):

[begeekmyfriend]

What about the use_bias in Dense layer?

[Rayhane-mamah]

It is useless to set use_bias to True in:
https://github.com/Rayhane-mamah/Tacotron-2/blob/fb5564b7584ae0dc62ffecaa89d463ff24a3c251/tacotron/models/attention.py#L164

This is because the model learns that bias automatically inside:
https://github.com/Rayhane-mamah/Tacotron-2/blob/fb5564b7584ae0dc62ffecaa89d463ff24a3c251/tacotron/models/attention.py#L69-L71