Rasa: ValueError: 684830 exceeds max_map_len(32768)

[richwednesday]

Rasa NLU version: 0.13.8

Operating system (windows, osx, ...): Ubuntu

Content of model configuration file:

language: "en"

pipeline: spacy_sklearn

Issue:
This is the full stack trace below from trying to train

2018-12-03 06:45:39 INFO rasa_nlu.utils.spacy_utils - Trying to load spacy model with name 'en'
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.6/dist-packages/rasa_nlu/train.py", line 184, in
num_threads=cmdline_args.num_threads)
File "/usr/local/lib/python3.6/dist-packages/rasa_nlu/train.py", line 148, in do_train
trainer = Trainer(cfg, component_builder)
File "/usr/local/lib/python3.6/dist-packages/rasa_nlu/model.py", line 155, in __init__
self.pipeline = self._build_pipeline(cfg, component_builder)
File "/usr/local/lib/python3.6/dist-packages/rasa_nlu/model.py", line 166, in _build_pipeline
component_name, cfg)
File "/usr/local/lib/python3.6/dist-packages/rasa_nlu/components.py", line 441, in create_component
cfg)
File "/usr/local/lib/python3.6/dist-packages/rasa_nlu/registry.py", line 143, in create_component_by_name
return component_clz.create(config)
File "/usr/local/lib/python3.6/dist-packages/rasa_nlu/utils/spacy_utils.py", line 73, in create
nlp = spacy.load(spacy_model_name, parser=False)
File "/home/ampm/.local/lib/python3.6/site-packages/spacy/__init__.py", line 15, in load
return util.load_model(name, overrides)
File "/home/ampm/.local/lib/python3.6/site-packages/spacy/util.py", line 112, in load_model
return load_model_from_link(name, *overrides)
File "/home/ampm/.local/lib/python3.6/site-packages/spacy/util.py", line 129, in load_model_from_link
return cls.load(overrides)
File "/home/ampm/.local/lib/python3.6/site-packages/spacy/data/en/__init__.py", line 12, in load
return load_model_from_init_py(__file__, *overrides)
File "/home/ampm/.local/lib/python3.6/site-packages/spacy/util.py", line 173, in load_model_from_init_py
return load_model_from_path(data_path, meta, *overrides)
File "/home/ampm/.local/lib/python3.6/site-packages/spacy/util.py", line 156, in load_model_from_path
return nlp.from_disk(model_path)
File "/home/ampm/.local/lib/python3.6/site-packages/spacy/language.py", line 653, in from_disk
util.from_disk(path, deserializers, exclude)
File "/home/ampm/.local/lib/python3.6/site-packages/spacy/util.py", line 511, in from_disk
reader(path / key)
File "/home/ampm/.local/lib/python3.6/site-packages/spacy/language.py", line 641, in
self.vocab.from_disk(p) and _fix_pretrained_vectors_name(self))),
File "vocab.pyx", line 380, in spacy.vocab.Vocab.from_disk
File "vectors.pyx", line 391, in spacy.vectors.Vectors.from_disk
File "/home/ampm/.local/lib/python3.6/site-packages/spacy/util.py", line 511, in from_disk
reader(path / key)
File "vectors.pyx", line 369, in spacy.vectors.Vectors.from_disk.load_key2row
File "vectors.pyx", line 370, in spacy.vectors.Vectors.from_disk.load_key2row
File "/home/ampm/.local/lib/python3.6/site-packages/msgpack_numpy.py", line 205, in unpack
return _unpack(stream, *kwargs)
File "/home/ampm/.local/lib/python3.6/site-packages/msgpack/__init__.py", line 57, in unpack
return unpackb(data, *kwargs)
File "/home/ampm/.local/lib/python3.6/site-packages/msgpack_numpy.py", line 214, in unpackb
return _unpackb(packed, *kwargs)
File "msgpack/_unpacker.pyx", line 187, in msgpack._cmsgpack.unpackb
ValueError: 684830 exceeds max_map_len(32768)

[mvollrath]

Workaround is to freeze msgpack at 0.5.6.

[richwednesday]

This fixed it for me python -m pip install "msgpack<0.6.0"

[libindavis]

When building a docker image via Dockerfile, you can add below to the end of file.
RUN pip install --upgrade msgpack==0.5.6

Possibly, you are loadding the model with a different version of Rasa from which version you use for training the model.

[VickySrinivas]

Getting the below error after i downgraded the msgpack version to 0.5.6

Rasa NLU version: 0.12.3
msgpack-python==0.5.6

Operating system : Macos

Content of model configuration file:
language: "en"
pipeline: spacy_sklearn

Full stacktrace below:

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa_nlu/training_data/training_data.py:194: UserWarning: Entity 'people' has only 1 training examples! minimum is 2, training may fail.
self.MIN_EXAMPLES_PER_ENTITY))
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa_nlu/training_data/training_data.py:194: UserWarning: Entity 'price' has only 1 training examples! minimum is 2, training may fail.
self.MIN_EXAMPLES_PER_ENTITY))
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa_nlu/utils/__init__.py:236: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
return yaml.load(read_file(filename, "utf-8"))
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:521: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:522: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Traceback (most recent call last):
File "nlu_model.py", line 21, in
train_nlu('./data/data.json', 'config_spacy.json', './models/nlu')
File "nlu_model.py", line 12, in train_nlu
trainer = Trainer(config.load(config_file), builder)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa_nlu/model.py", line 149, in __init__
self.pipeline = self._build_pipeline(cfg, component_builder)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa_nlu/model.py", line 160, in _build_pipeline
component_name, cfg)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa_nlu/components.py", line 420, in create_component
cfg)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa_nlu/registry.py", line 140, in create_component_by_name
return component_clz.create(config)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rasa_nlu/utils/spacy_utils.py", line 73, in create
nlp = spacy.load(spacy_model_name, parser=False)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/spacy/__init__.py", line 15, in load
return util.load_model(name, overrides)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/spacy/util.py", line 112, in load_model
return load_model_from_link(name, *overrides)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/spacy/util.py", line 129, in load_model_from_link
return cls.load(overrides)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/spacy/data/en/__init__.py", line 12, in load
return load_model_from_init_py(__file__, *overrides)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/spacy/util.py", line 173, in load_model_from_init_py
return load_model_from_path(data_path, meta, **overrides)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/spacy/util.py", line 156, in load_model_from_path
return nlp.from_disk(model_path)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/spacy/language.py", line 653, in from_disk
util.from_disk(path, deserializers, exclude)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/spacy/util.py", line 511, in from_disk
reader(path / key)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/spacy/language.py", line 649, in
deserializers[name] = lambda p, proc=proc: proc.from_disk(p, vocab=False)
File "pipeline.pyx", line 643, in spacy.pipeline.Tagger.from_disk
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/spacy/util.py", line 511, in from_disk
reader(path / key)
File "pipeline.pyx", line 626, in spacy.pipeline.Tagger.from_disk.load_model
File "pipeline.pyx", line 627, in spacy.pipeline.Tagger.from_disk.load_model
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 352, in from_bytes
copy_array(dest, param[b'value'])
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/thinc/neural/util.py", line 48, in copy_array
dst[:] = src
ValueError: could not broadcast input array from shape (96) into shape (128)