Gensim: subtle errors in gradient descent update step in word2vec.py

[Jianqiang]

I haven't looked in the Cython implementation, but in the Python implementation, it seems that the saved error term In train_sg_pair() line 188 and train_cbow_pair() line 217,

neu1e += dot(gb, l2b)

should be moved to the place _before_ the update for syn1neg (189-190, 215-216) is performed,

    if train_w1:
        model.syn1[word.point] += outer(ga, l1)  # learn hidden -> output

because l2b is a _reference_ to the corresponding values in syn1neg, which will be changed after executing the update of syn1neg.

However, the update of syn0 is supposed to be independent from that of syn1neg.

Of course, as the update for syn1neg itself is very small, this error does not adversely affect the final results much.

[gojomo]

My understanding is that l2b is a copy rather than a reference/view: it's created via numpy "advanced indexing" using the word_indices list, and per http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html that indexing always results in a copy. (Are you sure it's otherwise?)

[tmylk]

@Jianqiang Ping - is this resolved by the reply above?

[Jianqiang]

@gojomo thanks!