Sentence-transformers: Some checks on training with numpy

[alejandrojcastaneira]

Hi, I'm training a STS using this code but over my domain data:

I'm getting these warnings:

/numpy/lib/function_base.py:2534: RuntimeWarning: invalid value encountered in true_divide
c /= stddev[:, None]
/scipy/stats/_distn_infrastructure.py:901: RuntimeWarning: invalid value encountered in greater
return (a < x) & (x < b)
/scipy/stats/_distn_infrastructure.py:901: RuntimeWarning: invalid value encountered in less
return (a < x) & (x < b)
/scipy/stats/_distn_infrastructure.py:1892: RuntimeWarning: invalid value encountered in less_equal
cond2 = cond0 & (x <= _a)
/numpy/lib/function_base.py:2535: RuntimeWarning: invalid value encountered in true_divide

Then the similarities on all the epochs are computed:

Cosine-Similarity : Pearson: nan Spearman: nan
2019-08-30 14:35:20 - Manhattan-Distance: Pearson: nan Spearman: nan
2019-08-30 14:35:20 - Euclidean-Distance: Pearson: nan Spearman: nan
2019-08-30 14:35:20 - Dot-Product-Similarity: Pearson: nan Spearman: nan

The examples with the STSbenchmark works Very Good! I'm just changing the train, dev, set files, I couldn't train on this data, could be associated with the vocabulary of the word embeddings? maybe that it could't contains some words from my corpus.

Bes regards

[alejandrojcastaneira]

I believe I found the cause of this, the thing it's that doing experiments with custom dev and test data, I created synthetic examples all with the cosine similarity score set to the maximum, so the standard deviations on the dev set and test set was 0.

Then the formula for Pearson's product-moment correlation coefficient divides the covariance of the results from training with the gold standard results on the dev and test sets by the product of their standard deviations.

Since my dev set and test set had zero variance, its standard deviation is also zero. That's why I got the true_divide error, then just introducing a small variance or using other data solved the problem. Sorry for my miss appreciation, this a Great library!
Best regards

[olastor]

@alejandrojcastaneira Thank you for your analysis. Having the same problem and your explanation saves me a lot of time investigating I think.

[mhsamavatian]

same problem. Accurate explanation. Thanks

[bierik]

@alejandrojcastaneira Thank you for your explanation.
I have exactly the same problem. I get a /scipy/stats/stats.py:3508: PearsonRConstantInputWarning: An inp ut array is constant; the correlation coefficent is not defined. which results in a nan value for the Cosine-Similarity Pearson and Spearman. I am using the opusparcus german train and dev set to continue the training on an nli trained german model. I tried to understand your answer but my maths skills are not good enough to get your answer. Can you give me an advice what to do to get rid of the nan value.

[Vadbeg]

Yep! Thanks a lot @alejandrojcastaneira. In my case I've used uniform distribution to solve this problem.

label = np.random.uniform(low=0.8, high=1.0)
input_example = InputExample(texts=texts, label=label)