Spacy: Spacy 2.2.3 gpu prediction gives less accurate result than spacy 2.0.18 cpu

[pyshahid]

How to reproduce the behaviour

1.Train a model for ner with spacy 2.0.18
2.Check prediction
3.Train a model for ner with spacy 2.2.3 with gpu
4.Check prediction

Your Environment

• Operating System:Ubuntu 18
• Python Version Used:3.6
• spaCy Version Used:2.2.3 and 2.0.18
• Environment Information:Docker
• Pretrained Model used:en_core_web_sm

[svlandeg]

Hm, we have seen regressions on GPU for Windows, but those have been fixed. You're running on Ubuntu? Could you please share some output numbers of a model you've trained and the difference in accuracy you've found?

[pyshahid]

@svlandeg thanks for replay and please forgive because i don't understood the output numbers and
in case of accuracy i don't have numbers but i would say that older spacy gives much better prediction result for example if i train my model with spacy 2.0.18 i would get all of the entities that trained but with spacy 2.2.3 some of the entities are missing but not all of them and i can't avoid that entities because i used spay 2.0.18 model in prodcution but now i want to reduce training time so i updated spacy.

hope i am clear to you

[alejandrojcastaneira]

Hi @svlandeg I had a similar behavior in the ticket [#3937].

[pyshahid]

@alejandrojcastaneira thanks for confirming can please tell me how did you solved that issue or can i use spacy 2.0.18 with gpu i tried it but i got some issues please help

[alejandrojcastaneira]

Actually, I keep using the 2.0.18 version because this one works better for my use case, not necessarily for all, but it works good on GPU for me. Could you share the issues that you had?

[pyshahid]

@alejandrojcastaneira thanks for your comment and your time i really appreciate that so my issue as follows

when try to use spacy 2.0.18 with gpu

i am getting this error

Warning: Unnamed vectors -- this won't allow multiple vectors models to be loaded. (Shape: (0, 0))
Traceback (most recent call last):
File "train_test.py", line 90, in
train()
File "train_test.py", line 81, in train
nlp = _train(train_data)
File "train_test.py", line 56, in _train
nlp.update([text], [annotations], sgd=optimizer)
File "/usr/local/lib/python3.6/site-packages/spacy/language.py", line 421, in update
proc.update(docs, golds, drop=drop, sgd=get_grads, losses=losses)
File "pipeline.pyx", line 479, in spacy.pipeline.Tagger.update
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 61, in begin_update
X, inc_layer_grad = layer.begin_update(X, drop=drop)
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 280, in begin_update
drop=drop)
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 61, in begin_update
X, inc_layer_grad = layer.begin_update(X, drop=drop)
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 374, in uniqued_fwd
Y_uniq, bp_Y_uniq = layer.begin_update(X_uniq, drop=drop)
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 61, in begin_update
X, inc_layer_grad = layer.begin_update(X, drop=drop)
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 176, in begin_update
values = [fwd(X, a, *k) for fwd in forward]
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 176, in
values = [fwd(X, a, *k) for fwd in forward]
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 258, in wrap
output = func(args, *kwargs)
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 176, in begin_update
values = [fwd(X, a, *k) for fwd in forward]
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 176, in
values = [fwd(X, a, *k) for fwd in forward]
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 258, in wrap
output = func(args, *kwargs)
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 176, in begin_update
values = [fwd(X, a, *k) for fwd in forward]
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 176, in
values = [fwd(X, a, *k) for fwd in forward]
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 258, in wrap
output = func(args, *kwargs)
File "/usr/local/lib/python3.6/site-packages/thinc/neural/_classes/hash_embed.py", line 51, in begin_update
keys = self.ops.hash(ids, self.seed) % self.nV
File "ops.pyx", line 821, in thinc.neural.ops.CupyOps.hash
NameError: name 'gpu_ops' is not defined

did you ever faced this issue and i googled same issue then i installed pip install thinc_gpu_ops then i got another error

Traceback (most recent call last):
File "train_test.py", line 90, in
train()
File "train_test.py", line 81, in train
nlp = _train(train_data)
File "train_test.py", line 56, in _train
nlp.update([text], [annotations], sgd=optimizer)
File "/usr/local/lib/python3.6/site-packages/spacy/language.py", line 421, in update
proc.update(docs, golds, drop=drop, sgd=get_grads, losses=losses)
File "nn_parser.pyx", line 589, in spacy.syntax.nn_parser.Parser.update
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 61, in begin_update
X, inc_layer_grad = layer.begin_update(X, drop=drop)
File "/usr/local/lib/python3.6/site-packages/thinc/check.py", line 146, in checked_function
return wrapped(args, *kwargs)
File "/usr/local/lib/python3.6/site-packages/thinc/neural/_classes/affine.py", line 56, in begin_update
output__BO = self.predict(input__BI)
File "/usr/local/lib/python3.6/site-packages/thinc/check.py", line 146, in checked_function
return wrapped(args, *kwargs)
File "/usr/local/lib/python3.6/site-packages/thinc/neural/_classes/affine.py", line 52, in predict
return self.ops.affine(self.W, self.b, input__BI)
File "ops.pyx", line 195, in thinc.neural.ops.Ops.affine
File "ops.pyx", line 183, in thinc.neural.ops.Ops.batch_dot
File "/usr/local/lib/python3.6/site-packages/cupy/linalg/product.py", line 35, in dot
return a.dot(b, out)
File "cupy/core/core.pyx", line 1118, in cupy.core.core.ndarray.__array__
File "cupy/core/core.pyx", line 405, in cupy.core.core.ndarray.astype
File "cupy/core/_kernel.pyx", line 930, in cupy.core._kernel.ufunc.__call__
File "cupy/core/_kernel.pyx", line 953, in cupy.core._kernel.ufunc._get_ufunc_kernel
File "cupy/core/_kernel.pyx", line 676, in cupy.core._kernel._get_ufunc_kernel
File "cupy/core/_kernel.pyx", line 128, in cupy.core._kernel._get_kernel_params
File "cupy/core/_scalar.pyx", line 69, in cupy.core._scalar.get_typename
File "cupy/core/_scalar.pyx", line 74, in cupy.core._scalar.get_typename
KeyError:

so updated to spacy_2_2_3 which lead me here

[alejandrojcastaneira]

Could you try uninstall thinc_gpu_ops: and installed with this command:
pip install --force-reinstall --no-binary :all: thinc-gpu-ops

[pyshahid]

@alejandrojcastaneira thanks for you time and replay but i still get this error

UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.
warnings.warn(msg)
6288
44
Warning: Unnamed vectors -- this won't allow multiple vectors models to be loaded. (Shape: (0, 0))
Traceback (most recent call last):
File "train_test.py", line 90, in
train()
File "train_test.py", line 81, in train
nlp = _train(train_data)
File "train_test.py", line 56, in _train
nlp.update([text], [annotations], sgd=optimizer)
File "/usr/local/lib/python3.6/site-packages/spacy/language.py", line 421, in update
proc.update(docs, golds, drop=drop, sgd=get_grads, losses=losses)
File "nn_parser.pyx", line 589, in spacy.syntax.nn_parser.Parser.update
File "/usr/local/lib/python3.6/site-packages/thinc/api.py", line 61, in begin_update
X, inc_layer_grad = layer.begin_update(X, drop=drop)
File "/usr/local/lib/python3.6/site-packages/thinc/check.py", line 146, in checked_function
return wrapped(args, *kwargs)
File "/usr/local/lib/python3.6/site-packages/thinc/neural/_classes/affine.py", line 56, in begin_update
output__BO = self.predict(input__BI)
File "/usr/local/lib/python3.6/site-packages/thinc/check.py", line 146, in checked_function
return wrapped(args, *kwargs)
File "/usr/local/lib/python3.6/site-packages/thinc/neural/_classes/affine.py", line 52, in predict
return self.ops.affine(self.W, self.b, input__BI)
File "ops.pyx", line 195, in thinc.neural.ops.Ops.affine
File "ops.pyx", line 183, in thinc.neural.ops.Ops.batch_dot
File "/usr/local/lib/python3.6/site-packages/cupy/linalg/product.py", line 35, in dot
return a.dot(b, out)
File "cupy/core/core.pyx", line 1118, in cupy.core.core.ndarray.__array__
File "cupy/core/core.pyx", line 405, in cupy.core.core.ndarray.astype
File "cupy/core/_kernel.pyx", line 930, in cupy.core._kernel.ufunc.__call__
File "cupy/core/_kernel.pyx", line 953, in cupy.core._kernel.ufunc._get_ufunc_kernel
File "cupy/core/_kernel.pyx", line 676, in cupy.core._kernel._get_ufunc_kernel
File "cupy/core/_kernel.pyx", line 128, in cupy.core._kernel._get_kernel_params
File "cupy/core/_scalar.pyx", line 69, in cupy.core._scalar.get_typename
File "cupy/core/_scalar.pyx", line 74, in cupy.core._scalar.get_typename
KeyError:

[pyshahid]

@alejandrojcastaneira hi it will be great help if you provide the versions of cupy,thinc that you are using

[alejandrojcastaneira]

@pyshahid sorry to hear that still not working. The drivers and package versions are:
NVIDIA Driver Version: 430.34 CUDA Version: 10.1
spacy 2.0.18
numpy 1.17.0
thinc 6.12.1
thinc-gpu-ops 0.0.4
cupy-cuda100 6.2.0
cupy-cuda101 6.2.0

[pyshahid]

@alejandrojcastaneira thanks for replaying again and sorry for bothering you one more time can please tell me how did you installed spacy

i installed spacy
like below
pip install spacy[cuda]==2.0.18

is there any other way to install spacy for gpu usage

[alejandrojcastaneira]

Hello @pyshahid I installed in the following way:
pip install spacy[cuda]==2.0.18
pip install --force-reinstall --no-binary :all: thinc-gpu-ops

[svlandeg]

@pyshahid : did those commands work for you?

[pyshahid]

@svlandeg no i tried with this command and i tried several other commands and don't found a solution so we decided to use non gpu version currentl.y so we are using spacy 2.0.18 with out gpu.

[svlandeg]

Ok. It certainly looks like your installation environment may have gone corrupt (Your installed Python is incomplete). This issue started around accuracy on GPU, but I haven't seen specific accuracy numbers or ways to try and reproduce the problem(s). As I don't see an immediate action point on our side, I'll go ahead and close this issue for now. I hope you've found a way to work around the trouble!

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.