Deeplearning4j: SkipGram NullPointerException

[KonceptGeek]

I'm getting the following NullPointerException when calling Word2Vec.fit() after reloading the model.

Exception in thread "VectorCalculationsThread 0" Exception in thread "VectorCalculationsThread 3" java.lang.RuntimeException: java.lang.NullPointerException
    at org.deeplearning4j.models.sequencevectors.SequenceVectors$VectorCalculationsThread.run(SequenceVectors.java:1100)
Caused by: java.lang.NullPointerException
    at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.learnSequence(SkipGram.java:167)
    at org.deeplearning4j.models.sequencevectors.SequenceVectors.trainSequence(SequenceVectors.java:303)
    at org.deeplearning4j.models.sequencevectors.SequenceVectors$VectorCalculationsThread.run(SequenceVectors.java:1075)
java.lang.RuntimeException: java.lang.NullPointerException
    at org.deeplearning4j.models.sequencevectors.SequenceVectors$VectorCalculationsThread.run(SequenceVectors.java:1100)
Caused by: java.lang.NullPointerException
    at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.learnSequence(SkipGram.java:167)
    at org.deeplearning4j.models.sequencevectors.SequenceVectors.trainSequence(SequenceVectors.java:303)
    at org.deeplearning4j.models.sequencevectors.SequenceVectors$VectorCalculationsThread.run(SequenceVectors.java:1075)

The model is loaded using the following command:

Word2Vec word2Vec = WordVectorSerializer.readWord2VecModel(new File(modelPath), true);

[KonceptGeek]

FYI, looks like batches.get() is returning null.

The following code in the function iterateSample is executed:

if (batches.get() == null) {
    batches.set(new ArrayList<Aggregate>());
}

[raver119]

Show full model configuration please.

[KonceptGeek]

@raver119 following was the configuration used:

word2Vec = new Word2Vec.Builder()
                .minWordFrequency(10)
                .layerSize(200)
                .learningRate(0.025)
                .minLearningRate(0.0001)
                .windowSize(5)
                .sampling(0.001)
                .seed(1)
                .workers(8)
                .negativeSample(5)
                .iterations(5)
                .stopWords(new ArrayList<String>())
                .elementsLearningAlgorithm(new SkipGram<VocabWord>())
                .useHierarchicSoftmax(false)
                .allowParallelTokenization(false)
                .tokenizerFactory(tokenizerFactory)
                .iterate(word2VecContentIterator)
                .build();

[raver119]

Thanks, i'll try to reproduce it.

[raver119]

Please take a look into this test, does it looks close enough to what you trying to do?

https://github.com/deeplearning4j/deeplearning4j/blob/70ff36760fbb863cd7622fead582d75935098ef8/deeplearning4j-nlp-parent/deeplearning4j-nlp-uima/src/test/java/org/deeplearning4j/models/word2vec/Word2VecTests.java#L333-L333

[raver119]

TL/DR: We train model, using negSampling. Saving it. Restoring. Attaching new iter/tokenizer, and do training.

For me in both cases i don't see any exceptions.

[raver119]

Here's my output: https://gist.github.com/raver119/6ce0250d1b0b83356e7113c844116925

For Fit 3 i get only one line reported, because iterations = 1 by default, since it's reduced vectors were restored.

[KonceptGeek]

Pretty close, except I'm using default batch size and parallel tokenization is set to false. Another thing, after the reloading of the model, the number of documents provided to the iterator can be less than the batch size. For example, if the batch size is 512, it is possible that a CollectionSentenceIterator was created with list of size 50.

[raver119]

Batch size doesn't matter here, it's basically just a variable that defines runtime partitioning for the corpus AND acts as trigger for op batches aggregation. So even batchSize of 64 or 1 will work fine, just will be a bit slower.

[KonceptGeek]

Another thing, this exception does not occur all the time. Following is the current log of 2 consecutive fit calls:

2016-11-28 10:09:49,941 [pool-4-thread-1] INFO  o.d.m.s.SequenceVectors - Starting learning process...
Exception in thread "VectorCalculationsThread 14" java.lang.RuntimeException: java.lang.NullPointerException
        at org.deeplearning4j.models.sequencevectors.SequenceVectors$VectorCalculationsThread.run(SequenceVectors.java:1100)
Caused by: java.lang.NullPointerException
2016-11-28 10:10:22,340 [pool-4-thread-1] INFO  o.d.m.s.SequenceVectors - Epoch: [1]; Words vectorized so far: [3706980];  Lines vectorized so far: [47365]; learningRate: [1.0E-4]
2016-11-28 10:10:22,340 [pool-4-thread-1] INFO  o.d.m.s.SequenceVectors - Time spent on training: 32399 ms
2016-11-28 10:10:23,392 [pool-4-thread-1] INFO  o.d.m.s.SequenceVectors - Starting learning process...
2016-11-28 10:10:56,848 [pool-4-thread-1] INFO  o.d.m.s.SequenceVectors - Epoch: [1]; Words vectorized so far: [3877685];  Lines vectorized so far: [49925]; learningRate: [1.0E-4]
2016-11-28 10:10:56,848 [pool-4-thread-1] INFO  o.d.m.s.SequenceVectors - Time spent on training: 33456 ms

[raver119]

That's awesome, show full output log please!

[raver119]

Aha, i probably have idea what's wrong.

Is there ANY chance, that your second corpus you're using for training after model restored, contains sentences that have not a single word matching first corpus?

[KonceptGeek]

The full output is pretty huge, but i just took out some of the lines from it: https://gist.github.com/KonceptGeek/20f8683dc5199e23af9ea76a3df3ca13

With regards to the 2nd corpus not have sentences containing a single word in the vocab: Yes, that is totally possible. This test corpus that I am using has approximately 15 million documents. 500k of which were used to build the vocab and initial training, and then chunks of the remaining are being used to update the weights of the model.

[raver119]

Right. Bug found and fixed then, will merge it later today, and it'll be included into 0.7.1 release coming out soon.

Thanks for highlighting this one.

[KonceptGeek]

Any recommended solution for handling this right now on my end?

[raver119]

I'd suggest to add 2 dummy words manually to each document/sentence with custom tokenizer, i.e. "this" word.

As fast workaround that should work.

But main problem here is small vocab you have after initial training. I'd suggest to build proper vocab at initial training, and in subsequent training you won't suffer anymore.

[raver119]

Fix is merged now.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.