Joblib: Alignment issue when passing arrays though mmap

[aseyboldt]

Arrays that are transfered to subprocesses though mmap appear not always to be aligned:

import joblib
import numpy as np

def func(x):
    print(type(x))
    print(x.flags)
    print(x.ctypes.data / 8)

if __name__ == '__main__':
    # Make the array larger than 1Mbyte so that mmap is used
    n = 1024 ** 2 // 8 + 1
    x = np.random.randn(n)
    jobs = joblib.Parallel(2)
    print(x.nbytes)
    res = jobs([joblib.delayed(func)(x), joblib.delayed(func)(x)])

This returns

<class 'numpy.core.memmap.memmap'>
1048584
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : False
  ALIGNED : False
  UPDATEIFCOPY : False
570054683.375

Note the last line, the address of the first element of the array is indeed not aligned.

I'm using version 0.11, on OSx sierra.

This came up in https://github.com/pymc-devs/pymc3/issues/2640, and seems to happen on several OSs.

[lesteve]

I can reproduce, this is a problem indeed. I'll try to look at this, but probably not before next week.

[lesteve]

So I looked at this a bit, and it looks like numpy does padding for .npy files in order to ensure alignment of mmap:

https://github.com/numpy/numpy/blob/1e494f1e283340d545b1c7c15dded04a4aaae939/numpy/lib/format.py#L328-L333.

Maybe we should do something similar in joblib.dump? Maybe @aabadie this is something you could look at, since you wrote the single-file pickle?

Implementation details: not sure what character to pad with, and backward-compatibility of pickles without padding needs to be looked at.

[aabadie]

@aseyboldt, FYI I opened #570 to try to fix this issue. The problems seems to be fixed in the snippet you provided. It's still WIP though