Joblib: Parallel hangs when /dev/shm is too small

[Piatachock]

Hey guys, I've encountered hangs on Parallel usage by sklearn, and one thing I've noticed is that hangs are happening when my dataset is big enough.
Debugging print showed me that main process is stuck here, so it looks like main process can never get a response from child process.
After some code analysis I was able to find usage of /dev/shm in joblib. Output of df showed me that this volume is full, so I gave it more space and after that hangs stopped.

So my suspicion is that joblib doesn't work with /dev/shmem fully correct, in sense that maybe we should check if there is enough space available and raise an error instead of hangs.

I ran my experiments in Jupyter Notebook, Ubuntu 14.04 in Docker. All libs are latest versions from pip.

If you need any additional info from me, let me know; I will be glad to help.

P.S. There is a bunch of issues with similar symptom here, but I wasn't sure if they are really related, so I created another one with concrete description.

[Piatachock]

List of possibly-related issues:

125

138

[lesteve]

I am not an expert on /dev/shm but I found this:
http://www.thegeekstuff.com/2008/11/overview-of-ramfs-and-tmpfs-on-linux/

which seems to indicate that when /dev/shm is mounted using tmpfs (as is the case on my Ubuntu box), it will use swap. My guess is that your system was swapping, effectively grinding your computation to a halt.

[Piatachock]

I'm not sure if this is the case, because by "hang" I meant exactly 0% CPU consumption of all subprocesses for like 3 hours minimum (I turned it off after that). IMHO case with swapping should look diffirently.

[ogrisel]

@Piatachock can you please confirm that you do not reproduce the hanging does not occur if you set the temp_folder parameter of Parallel to a filesystem folder such as /tmp for instance?

Do you have any idea why /dev/shm was full in the first place? Is this a bug of joblib with files that are created by joblib and never deleted in /dev/shm? The pid is a part of the filename that should help detect whether all the files from the same execution or if old executions have not be deleted.

[Piatachock]

@ogrisel yes, I never tried to change temp_folder parameter.

I think that fact of /dev/shm was full is no surprise, because I've run Jupyter in Docker container, and it has default size of /dev/shm equal to 64M. After I increased it, hangs stopped.

Why this case (full shared memory) is leading to hangs is interesting, though.

[ogrisel]

Alright thanks for the confirmation. Maybe we should find a way to cheaply check the amount of free space in the temp_folder and raise an error when it's (almost) full.

[Piatachock]

@ogrisel you are welcome.
From my perspective, tho, core problem here is not absence of pre-check, but rather hang of child processes. In that situation main process will set himself on timeout-less wait from IPC, and will never able to recover.
And even worse, only way to stop those processes is to restart entire Jupyter (or kill main process and then find process pids and kill them manually). Common interrupt wont help here because underlying process pool is helpfully launches new processes as old ones die due to interrupt (but that probably worth another issue).

[jmemich]

cheaply check the amount of free space in the temp_folder

This would have been helpful for me too, I had a similar issue with /dev/shm in a Docker container. If no one's actively working on this I am happy to put together a patch with the above proposal.

[GaelVaroquaux]

I had a similar issue with /dev/shm in a Docker container. If no one's
actively working on this I am happy to put together a patch with the
above proposal.

Go for it! In particular if you can reproduce the problem and hence
convince yourself that this approach fixes it.