Singularity: Failing basic recipe for archlinux due to bad filesystem mounts

[VukanJ]

Problem

Basic installation of archlinux system fails. It seems like the filesystem is not mounted as singularity would expect it. My root directory is read-write though.
It is really challenging to find any information on this issue.

1877 may be related to this, I am really not sure. I have tried disabling the CheckSpace option in pacman.conf and tried it on a CentOS host system, but no luck. Is there a way to fix this issue on the user side?

Version of Singularity:

singularity version 3.2.0-rc1.443.gb3e6bd38e
Host system: 5.1.15-arch1-1-ARCH

Recipe

Bootstrap: docker 
FROM: archlinux/base 

%post
pacman -Syu --noconfirm
pacman -S --noconfirm base base-devel make gcc cmake git tmux boost python wget curl

Expected behavior

Successful execution

Actual behavior

(97/97) checking keys in keyring                                  [####################################] 100%
(97/97) checking package integrity                                [####################################] 100%
(97/97) loading package files                                     [####################################] 100%
(97/97) checking for file conflicts                               [####################################] 100%
(97/97) checking available disk space                             [####################################] 100%
error: Partition /etc/resolv.conf is mounted read only
error: Partition /etc/hosts is mounted read only
error: not enough free disk space
error: failed to commit transaction (not enough free disk space)
Errors occurred, no packages were upgraded.
FATAL:   post proc: exit status 1
FATAL:   While performing build: while running engine: exit status 255

The not enough space error is not related to /tmp since I have enough available space there, so I am guessing that it is due to the mountd devices it cannot access.

Steps to reproduce behavior

run sudo singularity build myimage.img myrecipe

[louwers]

I am having the same problem, with enough space available on /tmp. I compiled and used the master brach, version 3.3.0-rc.1.135.g42a714ae1.

Someone on the Arch Forums having the same problem with Docker: https://bbs.archlinux.org/viewtopic.php?id=233736

Quick fix, add this to your recipe:

mount -o remount,rw /etc/resolv.conf
mount -o remount,rw /etc/hosts

[VukanJ]

I eventually solved this problem by myself. If I leave out the packages base and base-devel, singularity is able to complete the installation. Seems like these packages require special permissions, since they setup the base system. Thanks, @louwers, that will be useful at some point for sure! I did not try that out here, though.

[dtrudg]

@VukanJ @louwers - if you're able to test this on 3.5.2 you may find it works without the workaround, as handling of the hosts, resolv.conf files has changed so that they are staged rather than directly bind mounted.

[VukanJ]

I can confirm that the recipe that I posted now works using singularity version 3.5.1-1. I was not able to test the singularity master branch. Thanks, @louwers !