Singularity: Possible problem accessing /dev/shm path in singularity container

[c-guzman]

Version of Singularity:

2.3.1

Expected behavior

bamCoverage function will run and generate bigWig files

Actual behavior

Getting a weird little error when trying to run bamCoverage for sequencing data. Using Nextflow ... the majority of the pipeline works fine, this is the only step that fails.

bamCoverage -b ${bam} -o ${mergeid}.RPKMnorm.bw -of bigwig -bs ${params.bamcoverage_bs} -p ${params.threads} --normalizeUsingRPKM --smoothLength ${params.bamcoverage_smooth} -e ${params.bamcoverage_e} --centerReads 2> ${mergeid}.bamcoverage_report.txt

Traceback (most recent call last):
  File "/opt/anaconda2/bin/bamCoverage", line 11, in <module>
    main(args)
  File "/opt/anaconda2/lib/python2.7/site-packages/deeptools/bamCoverage.py", line 145, in main
    scale_factor = get_scale_factor(args)
  File "/opt/anaconda2/lib/python2.7/site-packages/deeptools/getScaleFactor.py", line 168, in get_scale_factor
    bam_mapped, bam_mapped_total = get_num_kept_reads(args)
  File "/opt/anaconda2/lib/python2.7/site-packages/deeptools/getScaleFactor.py", line 157, in get_num_kept_reads
    ftk = fraction_kept(args)
  File "/opt/anaconda2/lib/python2.7/site-packages/deeptools/getScaleFactor.py", line 127, in fraction_kept
    verbose=args.verbose)
  File "/opt/anaconda2/lib/python2.7/site-packages/deeptools/mapReduce.py", line 141, in mapReduce
    pool = multiprocessing.Pool(numberOfProcessors)
  File "/opt/anaconda2/lib/python2.7/multiprocessing/__init__.py", line 232, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "/opt/anaconda2/lib/python2.7/multiprocessing/pool.py", line 138, in __init__
    self._setup_queues()
  File "/opt/anaconda2/lib/python2.7/multiprocessing/pool.py", line 234, in _setup_queues
    self._inqueue = SimpleQueue()
  File "/opt/anaconda2/lib/python2.7/multiprocessing/queues.py", line 354, in __init__
    self._rlock = Lock()
  File "/opt/anaconda2/lib/python2.7/multiprocessing/synchronize.py", line 147, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/opt/anaconda2/lib/python2.7/multiprocessing/synchronize.py", line 75, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 2] No such file or directory

This is deeptools version 2.5.3 inside a Singularity container installed using anaconda2.

After asking the deeptools dev, they told me the following:

After a bit of searching around, my only guess is that this is some sort of
issue where the python multiprocessing module has problems accessing
/dev/shm in singularity. That's pretty obscure and not something controlled
by deepTools.

Here is my singularity file:

Bootstrap: docker
From: ubuntu:latest

%labels
Maintainer [email protected]
Version 1.0

%environment
export PATH="/opt/anaconda2/bin:$PATH"

%post

    apt-get update

    apt-get install -y wget bzip2 perl gawk

    ln -s /bin/tar /bin/gtar

    wget https://repo.continuum.io/archive/Anaconda2-4.4.0-Linux-x86_64.sh

    bash Anaconda2-4.4.0-Linux-x86_64.sh -b -p /opt/anaconda2

    /opt/anaconda2/bin/conda install -c r r-base=3.3.2 r-essentials=1.5.2 r-devtools=1.12.0

    /opt/anaconda2/bin/conda install -c bioconda --yes bbmap samtools epic deeptools macs2 bedtools bedops multiqc subread stringtie nextflow bowtie2 bwa hisat2 star fastqc gimmemotifs nucleoatac kallisto r-spp=1.14 bioconductor-biocinstaller=1.24.0 bioconductor-edger=3.16.5 bioconductor-deseq2=1.14.1 bioconductor-chipseeker=1.10.0

    wget -P /opt/anaconda2/bin http://hartleys.github.io/QoRTs/QoRTs.jar

    /opt/anaconda2/bin/R --slave -e 'install.packages("http://hartleys.github.io/QoRTs/QoRTs_LATEST.tar.gz", repos=NULL, type="source")'

Steps to reproduce behavior

Above.

[cclerget]

@c-guzman Can you test by specifying -c :

singularity shell -c container.img

If it fails, /dev/shm is not present on your host system

[c-guzman]

@cclerget I can confirm that the tool works appropriately when specifying the -c flag.

[cclerget]

@c-guzman I thought -c option will failed, but since it's not a binding. It seems you don't have /dev/shm present on your host, if ls /dev/shm from host say "no such file or directory" so it's not a singularity nor bamCoverage problem, but your host system

[c-guzman]

@cclerget this isn't the problem. If I ls /dev/shm from my host, it shows that the directory exists. I can run bamCoverage just fine outside of the container as well.

[c-guzman]

@cclerget i've managed to reproduce this error from inside the singularity container.

Singularity cipher.img:~/projects/soohwan_ctcf> python
Python 2.7.13 |Anaconda custom (64-bit)| (default, Dec 20 2016, 23:09:15) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import multiprocessing
>>> pool = multiprocessing.Pool(15)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/anaconda2/lib/python2.7/multiprocessing/__init__.py", line 232, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "/opt/anaconda2/lib/python2.7/multiprocessing/pool.py", line 138, in __init__
    self._setup_queues()
  File "/opt/anaconda2/lib/python2.7/multiprocessing/pool.py", line 234, in _setup_queues
    self._inqueue = SimpleQueue()
  File "/opt/anaconda2/lib/python2.7/multiprocessing/queues.py", line 354, in __init__
    self._rlock = Lock()
  File "/opt/anaconda2/lib/python2.7/multiprocessing/synchronize.py", line 147, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/opt/anaconda2/lib/python2.7/multiprocessing/synchronize.py", line 75, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 2] No such file or directory

[cclerget]

@c-guzman The only way I can reproduce error is by setting permissions to 700 on /dev/shm with root:root owner/group inside container. Strangely if /dev/shm doesn't exists no error occured, I suspect python module to take a different path if /dev/shm.

Could you send output of ls -ld /dev/shm both inside/outside container ?

[c-guzman]

Outside container:

lrwxrwxrwx 1 root root 8 Jul 15 12:59 /dev/shm -> /run/shm

Inside container:

lrwxrwxrwx 1 root root 8 Jul 15 19:59 /dev/shm -> /run/shm

Inside the container /run/shm does not exist, while outside the container /run/shm does:

Outside:

drwxrwxrwt 2 root root 40 Aug 27 17:08 /run/shm

Inside:

ls: cannot access '/run/shm': No such file or directory

[gmkurtzer]

Maybe try adding /run as a bind path in the configuration.

[celstark]

I'm the one who posted up that issue chrsfilo just linked in here in fmriprep.

I can confirm that passing in -B /run/shm:/run/shm works. It not only fixes my issue but also fixes the python multiprocessing.Pool() sample above that was also a problem for me.

[LunarLanding]

Hi. I want to use singularity build -remote with a definition file, and so can't use the --bind argument fix described above. I also need to test the build locally, so the host is WSL2.
I found a workaround below, I include also my research; it would be good if this issue was reopened with the intent of including handling of /dev/shm being symlinked to /run/shm in singularity tools; it seems like a frequent occurrence. Although I am using an old version so it might no longer be an issue, I see no pull request referenced in this issue.

Singularity version: 3.5.3 (to match cluster version)
Host: WSL2

Research

Expand

When: /dev/shm on the host is symlinked to /run/shm; and: singularity build is called; the container's /dev filesystem as per df -H is None:

$ df -H
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        270G   27G  230G  11% /
none            3.3G     0  3.3G   0% /dev
tmpfs           2.1M  8.2k  2.1M   1% /etc/hosts

The host is WSL2. df -H shows:

$ df -H
/dev/sdb        270G   27G  230G  11% /
tmpfs           3.3G     0  3.3G   0% /mnt/wsl
tools           254G  227G   28G  90% /init
none            3.3G     0  3.3G   0% /dev
none            3.3G  4.1k  3.3G   1% /run
none            3.3G     0  3.3G   0% /run/lock
none            3.3G     0  3.3G   0% /run/shm
none            3.3G     0  3.3G   0% /run/user
tmpfs           3.3G     0  3.3G   0% /sys/fs/cgroup
C:\             254G  227G   28G  90% /mnt/c

From output done by the post section, I conclude (host) /dev-> (live container)/dev
From using the 'debug' flag in singularity build, I find that (live container)/run == (pos-build rootfs)/run .
Meaning (live container)/dev/shm links to (live container)/run/shm which is non-existent.
Linking (setup rootfs)/run/shm to /dev/(some new directory) could work, but would not have the same behavior.

Another solution could be in the setup section mounting run/shm to rootfs/run/shm.

Because I am using singularity build, I cannot bind /run/shm to /run/shm via a command line argument.
I verified that the SINGULARITY_BINDPATH environment variable has no influence on the build command.

Creating /run/shm either in the %setup section or the %post section triggers the same error.

Solution

Well, since df -H on host shows the same filesystem on dev as when it is run on the live container, this works:

%setup
    if [ -L /dev/shm ]; then
        if [ ! -d /run/shm ]; then exit 1; fi
        mkdir -p /dev/manual
        ln -s /dev/manual ${SINGULARITY_ROOTFS}/run/shm
        touch ${SINGULARITY_ROOTFS}/rm_run_shm
    fi
    ...

%post
   ...
   if [ -f /rm_run_shm ]; then 
        rm /run/shm;
        rm /rm_run_shm
   fi