Joblib: struct.error: 'i' format requires -2147483648 <= number <= 2147483647

[enricopal]

Hello,
I am using joblib to parallelize the computation of a feature matrix, a large numpy array of floats (~7k rows and ~10k columns, ~70M values).

My code breaks at this point:

user_item_features = Parallel(n_jobs=n_jobs)(
    delayed(self._compute_features)(data, recommender, users_list)
    for users_list in users_list_chunks
)

with this error:

Traceback (most recent call last):
  File "entity2rec/node2vec_recommender.py", line 138, in <module>
    n_jobs=args.workers, supervised=False)
  File "/home/semantic/Repositories/entity2rec/entity2rec/evaluator.py", line 255, in features
    users_list_chunks, n_jobs)
  File "/home/semantic/Repositories/entity2rec/entity2rec/evaluator.py", line 269, in _compute_features_parallel
    for users_list in users_list_chunks)
  File "/home/semantic/anaconda3/lib/python3.6/site-packages/joblib/parallel.py", line 789, in __call__
    self.retrieve()
  File "/home/semantic/anaconda3/lib/python3.6/site-packages/joblib/parallel.py", line 699, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/semantic/anaconda3/lib/python3.6/multiprocessing/pool.py", line 608, in get
    raise self._value
  File "/home/semantic/anaconda3/lib/python3.6/multiprocessing/pool.py", line 385, in _handle_tasks
    put(task)
  File "/home/semantic/anaconda3/lib/python3.6/site-packages/joblib/pool.py", line 372, in send
    self._writer.send_bytes(buffer.getvalue())
  File "/home/semantic/anaconda3/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/semantic/anaconda3/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
    header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647

I have obtained this error using Linux and different versions of Python:

• python 3.6.6, 3.6.3, 3.6.0
• joblib 0.11

Any help would be appreciated.
Thank you for your work.
Enrico

[ogrisel]

That won't probably change a thing but can you please make sure that you can reproduce the same problem with joblib 0.12.1?

What is the size of data? You can print data.nbytes. Is it really 70e6 * 8 Bytes == 560 MB? If so there should be no problem I cannot reproduce your error with such 'small' datasets.

What is the size of the other arguments recommender and users_list?

[ogrisel]

If you can reproduce the problem with joblib 0.12.1, please post the traceback here so that I get the correct line numbers to investigate further.

[enricopal]

hi @ogrisel, thank you for your reply.

The data matrix that weights ~560MB is returned by the self._compute_features function. The data argument is only a string, user list has a negligible size, but recommender on the other hand weighs > 2.6GB. I suppose that recommender is the culprit here.
I will try to reproduce the error with joblib 0.12.1, but it will take days. Will keep you posted.
Thank you.
Enrico

[ogrisel]

BTW, are you running 32 bit or 64 bit python?

python -c "import struct; print(struct.calcsize('P') * 8)"

[ogrisel]

Is recommender an object that contains numpy array does it use some other kind of data holder object internally?

[enricopal]

it is 64 bit python. In this case, recommender is an object that contains a python dictionary that weighs roughly 2.6GBs of data. Something like:

recommender = Recommender(...)
where:

class Recommender:
      def __init__(self, ...):
            self.model = self._read_scores(...)
      def _read_scores(self, scores_file):
           .....
           return BIG_DICTIONARY

[ogrisel]

Alright, it's caused by a limitation in a low level multiprocessing routine. If your dictionary does not change often, it's better to serialize it on the disk (using pickle.load / pickle.dump from the standard library which is faster than joblib for this kinds of objects) and then load it in your workers in your own code instead of passing it as an argument to the parallel function.

We cannot do anything at the joblib level in this case, so closing.

[gwerbin]

Is there any way to predict when this will happen? Just be careful when returning anything that might conceivably be "big"?

[phesami-zest]

Alright, it's caused by a limitation in a low level multiprocessing routine. If your dictionary does not change often, it's better to serialize it on the disk (using pickle.load / pickle.dump from the standard library which is faster than joblib for this kinds of objects) and then load it in your workers in your own code instead of passing it as an argument to the parallel function.

We cannot do anything at the joblib level in this case, so closing.

Hi @ogrisel, I'm facing a similar issue and can't upgrade to py38 due to other compatibility issues. Can you please elaborate on the workaround you mentioned here a little more? Do you mean to use the multiprocessing backend which uses the native pickle library to serialize data? ref--> https://joblib.readthedocs.io/en/latest/auto_examples/serialization_and_wrappers.html