Spacy: Spacy 2.2.3 is not predicting any values

[jerilkuriakose]

I have the following code to train ner using spacy for our custom dataset.

spacy.prefer_gpu()
if model_dir:
    print('loading an existing model')
    nlp = spacy.load(model_dir)
else:
    print('creating a new model')
    nlp = spacy.blank('en')  # create blank Language class
# create the built-in pipeline components and add them to the pipeline
# nlp.create_pipe works for built-ins that are registered with spaCy
if 'ner' not in nlp.pipe_names:
    ner = nlp.create_pipe(pipe_name)
    nlp.add_pipe(ner, last=True)
# otherwise, get it so we can add labels
else:
    ner = nlp.get_pipe("ner")
# add labels
for __, annotations in TRAIN_DATA:
    for ent in annotations.get('entities'):
        ner.add_label(ent[2])

# get names of other pipes to disable them during training
other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'ner']
with nlp.disable_pipes(*other_pipes):  # only train NER
    if not model_dir:
        print('Starting a new training')
        optimizer = nlp.begin_training()
    else:
        optimizer = nlp.entity.create_optimizer()
    for itn in range(iterations):
        print("Starting iteration " + str(itn + 1))
        random.shuffle(TRAIN_DATA)
        losses = {}
        batches = minibatch(TRAIN_DATA,
                            size=compounding(4.0, 32.0, 1.001))
        for batch in batches:
            texts, annotations = zip(*batch)
            nlp.update(
                texts,  # batch of texts
                annotations,  # batch of annotations
                drop=0.2,  # dropout - make it harder to memorise data
                sgd=optimizer,  # callable to update weights
                losses=losses)
        print(losses)
        # if losses[pipe_name] < constants.LOSSES_THRESHOLD:
        #     break
    print('Completed training')

And following the code for prediction:

nlp = spacy.load(model_dir)
doc = nlp(page_data)
nlp_output = [(ent.text, ent.label_) for ent in doc.ents]

Now we used the above mentioned code in two spacy versions: spacy 2.1.8 and spacy 2.2.3, and following are the results:

| Tasks | Version 2.1.8 | Version 2.2.3 |
| --- | --- | --- |
| Dataset size | 10 | 10 |
| Iterations | 500 | 500 |
| Training | Success | Success |
| Prediction | Success | Did not predict a single entity |
| Loss | 20.35429 | 1056.54712 |

• Operating System: Windows 7
• Python Version Used: 3.6.5

We have also noticed the losses are decreasing at a good rate in spacy 2.1.8, whereas it takes more iterations in spacy 2.2.3.

To summarise Spacy 2.2.3 did not return any predictions whereas we were getting the expected results in Spacy 2.1.8, what might be the issue, kindly help.

[svlandeg]

Thanks for the detailed report!

I don't see anything wrong with your code and surely the training loss shouldn't be so much higher on the exact same dataset. This could be related to some other problems we've been seeing with GPU. Is there any chance you could try a run on CPU for at least a few epochs, to compare the training loss decrease rate ?

[jerilkuriakose]

Thank you for your reply.

We trained using CPU, in spacy version 2.2.3 for 500 iterations. The model had a loss of 1.568725 and the model was able to predict the labels. The training loss decrease rate is as expected.

It looks like there is a problem with training using GPU for spacy version 2.2.3.

[svlandeg]

Thanks so much for testing! Good to hear it works properly on CPU, at least.
We are currently debugging this so hopefully will find a solution soon for GPU soon.

One more question: did you happen to use the en_core_web_sm model? If so - it might work on GPU with en_core_web_md instead.

[svlandeg]

Also - apologies for the many questions but it's helpful trying to chase this down - do you know whether maybe anything else changed between your runs for 2.1.8 and 2.2.3 ? Did you perhaps change the original model, or switch from CPU to GPU ?

[jerilkuriakose]

Thank you for letting us know.

One more thing that we noticed was, the model that we trained and saved in spacy 2.1.8 did not work with spacy 2.2.3.
i.e., we trained a model for 50000 iterations (it took a week for training) using spacy 2.1.8, and it was predicting with a good accuracy in spacy 2.1.8, but the same model when we tried to use in spacy 2.2.3 for prediction it didnt predict a single entity.

Is there a way to use spacy 2.1.8 trained model with spacy 2.2.3 for predictions? Thank you.

[svlandeg]

No, I'm afraid not. Because we have underlying code changes for 2.2, you need to retrain your models after upgrading ...

[svlandeg]

@jerilkuriakose: we have found the likely culprit, cf PR https://github.com/explosion/thinc/pull/149. We'll close this issue and I hope you won't have any further problems in the future. But if you do, feel free to open a new issue !

[jerilkuriakose]

Thank you for resolving...

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.