Tacotron2: Adding Constraint on Alignment Saturation

[fatihkiralioglu]

Hi,
I have trained a tacotron2 model on a custom dataset and while testing newly generated model, I have come across some input examples where alignment is maximized way before gate_threshold or max decoder steps conditions are satisfied:

if torch.sigmoid(gate_output.data) > self.gate_threshold:
break
elif len(mel_outputs) == self.max_decoder_steps:
print("Warning! Reached max decoder steps")
break

Mel and alignment output plots are like the following for a specific case:

As you can see, the alignment is maximized at about 3000 ms and the synthesis is completed successfully for the input text, however, because gate output score did not reach the threshold, decoder continues and produces redundant synthesis.

What I wonder is whether it is possible to add a new constraint for alignment so that after alignment saturates, inference loop is stopped.

Thanks.

[CookiePPP]

@fatihkiralioglu
https://github.com/CookiePPP/codedump/blob/master/tacotron2-PPP-1.3.0/loss_function.py#L27
https://pytorch.org/docs/stable/nn.html#torch.nn.BCEWithLogitsLoss

You might fix problem by adding pos_weight

        gate_pos_weight = torch.tensor( 10.0 )
        gate_loss = nn.BCEWithLogitsLoss(pos_weight=gate_pos_weight)(gate_out, gate_target)

or something similar to loss function during training.

At the moment, Tacotron values each 0 on gate with the same value as a single 1.0
Since gate only has to predict a positive once per clip, while predicting hundreds of zeros, there is a noticeable imbalance, and gate will always predict 0 if it is unsure.

[rafaelvalle]

Closing due to inactivity.