Singularity: Read-only file system error

[GerardBCN]

Dear community,

we are struggling to run Singularity containers in a shared file system. When we try to create a file in an unmounted folder we always get this error.

cp: cannot create regular file '/path/to/my/script': Read-only file system

However, we don't experience this error when we run this singularity in other hosts or file systems, meaning we can correctly create a file in a container's unmounted folder. If I understand correctly how it works, squashfs allows to write up to 1Mb of content until the space runs out, am I right?

We have checked differences between singularity config files but both hosts have the same config. Could it be related to differences in squashfs installation/config?

Thanks a lot for your help!

Version of Singularity:

2.5.2-dist on the host it doesn't work.
2.4.5-dist on the host where it works.

Expected behavior

I'm able to create a file in an unmounted folder.

Actual behavior

I can't do it and I get the error:

cp: cannot create regular file '/path/to/my/script': Read-only file system

Steps to reproduce behavior

I create a new container, e.g. centos docker image and in my recipe file I create a folder in the /

mkdir ${SINGULARITY_ROOTFS}/data

I shell into the container:

singularity shell image.img

I try to create a file in the folder /data

touch /data/foo

it errors:
cp: cannot create regular file '/path/to/my/script': Read-only file system

Thanks!

[jscook2345]

@GerardBCN

I believe we resolved this issue in Slack? Please comment here if you are still having issues. Thanks!

[afortiorama]

What is the solution for benefit of other people who might have a similar problem?

[jmstover]

@afortiorama
IIRC, it ended up being a mix of run OS versions. One of the run OS's not supporting OverlayFS, another doing so. So, on the one _not_ supporting Overlay, a small write to a writable location on the filesystem was failing, which was succeeding on the other host. It was suggested to try setting enable overlay = yes to see if singularity would fail if overlay wasn't supported. I'm not 100% sure it does, but the default of try will attempt it, but if it can't it continues on.

Since a SIF image is read-only, we generally expect writes to fail, but if overlayfs is supported on a host you should be able to do some small writes to locations you would otherwise have write access to (like /var/tmp), or if you set permissions on directories to allow it in the definition file.

[GerardBCN]

Yes, what we found out is that we were running an OS that does not support overlay (to be more exact it was Oracle Linux Server 6.6). By changing overlay = yes we could not shell/run containers anymore because it would complain that the overlay could not be applied correctly.

Thanks a lot for your help.