Singularity: Updating CentOS 8 container fails

[olifre]

Version of Singularity:

What version of Singularity are you using? Run:

$ singularity version
3.4.1-1.1.el7

Expected behavior

Using a recipe centos8.def:

BootStrap: docker
From: centos:centos8

%post
yum -y update

I expect to be able to build a container being up-to-date with the most recent security updates doing:

singularity build centos8.sif centos8.def

Actual behavior

Errors out during yum -y update:

...
  Upgrading        : setup-2.12.2-2.el8.noarch                                                                                                                                                                                         21/38 
Error unpacking rpm package setup-2.12.2-2.el8.noarch
error: unpacking of archive failed on file /etc/hosts: cpio: utime

  Upgrading        : bind-export-libs-32:9.11.4-17.P2.el8_0.1.x86_64                                                                                                                                                                   22/38 
  Running scriptlet: bind-export-libs-32:9.11.4-17.P2.el8_0.1.x86_64                                                                                                                                                                   22/38 
error: setup-2.12.2-2.el8.noarch: install failed

  Cleanup          : systemd-udev-239-13.el8.x86_64                                                                                                                                                                                    23/38 
...
Error: Transaction failed
FATAL:   failed to execute %post proc: exit status 1
FATAL:   While performing build: while running engine: exit status 255

What OS/distro are you running

$ lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.7.1908 (Core) 
Release:        7.7.1908
Codename:       Core

How did you install Singularity

EPEL 7

[dtrudg]

Tried replicating this on my Debian 10 host. I cannot replicate with singularity 3.4.1 or current release-3.4 branch.

This needs to be checked on a RHEL / CentOS host.

[olifre]

@dctrud: https://github.com/containers/fuse-overlayfs/issues/108 made me think this might be selinux interfering, but interestingly, I can also reproduce in SELinux permissive mode.

[DrDaveD]

@olifre I tried reproducing on an SL7.7 host with 3.4.1-1.2.el7 and SElinux permissive mode. First, it requires running as root or with --fakeroot, right? I tried both and did not reproduce the error.

  ...
  Upgrading        : setup-2.12.2-2.el8.noarch                            21/38 
  Running scriptlet: setup-2.12.2-2.el8.noarch                            21/38 
warning: /etc/shadow created as /etc/shadow.rpmnew

  Upgrading        : bind-export-libs-32:9.11.4-17.P2.el8_0.1.x86_64      22/38 
  Running scriptlet: bind-export-libs-32:9.11.4-17.P2.el8_0.1.x86_64      22/38 
  Cleanup          : systemd-udev-239-13.el8.x86_64                       23/38 
  ...

[olifre]

@DrDaveD Interestingly, it works fine with --fakeroot for me - but fails consistently when running as root, not matter what the SELinux state of the machine is. I'm on CentOS 7.7, but it would be strange if SL7.7 and CentOS 7.7 would show a difference here, and I reproduced on two differently installed nodes.

[olifre]

Since it seems I'm the only one observing this (on whatever machine I try...) I collected a bit more information now.

This is using singularity version 3.4.1-1.2.el7 with the following recipe:

BootStrap: docker
From: centos:centos8

%post
mount
stat /etc/hosts
yum -y update || stat /etc/hosts

Here's the interesting parts, first trying non-fakeroot:

$ singularity build centos8.sif centos8.def
...
+ mount
/dev/vda1 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
...
/dev/vda1 on /etc/hosts type xfs (ro,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,noquota)
...
+ stat /etc/hosts
  File: /etc/hosts
  Size: 158             Blocks: 8          IO Block: 4096   regular file
Device: fd01h/64769d    Inode: 10479       Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-10-15 21:01:01.439202358 +0000
Modify: 2018-09-10 11:51:03.000000000 +0000
Change: 2019-10-04 19:11:30.526934217 +0000
 Birth: -
+ yum -y update
...
  Upgrading        : setup-2.12.2-2.el8.noarch                                                                                                                  26/48 
Error unpacking rpm package setup-2.12.2-2.el8.noarch
error: unpacking of archive failed on file /etc/hosts: cpio: utime

  Upgrading        : bind-export-libs-32:9.11.4-17.P2.el8_0.1.x86_64                                                                                            27/48 
  Running scriptlet: bind-export-libs-32:9.11.4-17.P2.el8_0.1.x86_64                                                                                            27/48 
error: setup-2.12.2-2.el8.noarch: install failed
...
Error: Transaction failed
+ stat /etc/hosts
  File: /etc/hosts
  Size: 158             Blocks: 8          IO Block: 4096   regular file
Device: fd01h/64769d    Inode: 10479       Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-10-15 21:01:01.439202358 +0000
Modify: 2018-09-10 11:51:03.000000000 +0000
Change: 2019-10-04 19:11:30.526934217 +0000
 Birth: -

What strikes me is that /etc/hosts is bind-mounted into the build environment read-only, so expectedly, yum inside the container can not update it.
@DrDaveD Any idea why it is working for you? Could you also check with mount if /etc/hosts is mounted ro in non-fakeroot mode?

In comparison, here's what I get in fakeroot mode:

$ singularity build -f centos8b.sif centos8.def
...
+ mount
/dev/vda1 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
...
/dev/vda1 on /etc/hosts type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
...
+ stat /etc/hosts
  File: /etc/hosts
  Size: 158             Blocks: 8          IO Block: 4096   regular file
Device: fd01h/64769d    Inode: 10479       Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-10-15 21:01:01.439202358 +0000
Modify: 2018-09-10 11:51:03.000000000 +0000
Change: 2019-10-04 19:11:30.526934217 +0000
 Birth: -
+ yum -y update
...
  Upgrading        : python3-hawkey-0.22.5-5.el8_0.x86_64                                                                                                       25/48 
  Upgrading        : setup-2.12.2-2.el8.noarch                                                                                                                  26/48 
  Running scriptlet: setup-2.12.2-2.el8.noarch                                                                                                                  26/48 
warning: /etc/shadow created as /etc/shadow.rpmnew

  Upgrading        : bind-export-libs-32:9.11.4-17.P2.el8_0.1.x86_64                                                                                            27/48 
...
Complete!
+ stat /etc/hosts
  File: /etc/hosts
  Size: 158             Blocks: 8          IO Block: 4096   regular file
Device: fd01h/64769d    Inode: 10479       Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-09-10 11:51:03.000000000 +0000
Modify: 2018-09-10 11:51:03.000000000 +0000
Change: 2019-10-15 23:08:29.536470136 +0000
 Birth: -
INFO:    Creating SIF file...

Expectedly, with the rw bindmount, things work fine.

Is this difference in bind-mounting expected?

[olifre]

The difference apparently comes from this code snippet:
https://github.com/sylabs/singularity/blob/316f0dc971b3547756bed29d919b7dcf50b9c6b0/internal/pkg/runtime/engine/imgbuild/create_linux.go#L80-L84
@dctrud Can you tell my if you can also see this on Debian? If /etc/hosts is mounted read-only for you (and the code suggests it should) I can't see how yum can ever do something to it which would change the utime, i.e. it should also fail on a Debian build machine - right?

[dtrudg]

@olifre

Building with sudo, yes the /etc/hosts is ro:

/dev/mapper/piran--vg-root on /etc/hosts type ext4 (ro,nosuid,nodev,noexec,relatime,errors=remount-ro)

... but the build completes without any error.

In the --fakeroot situation then the mount is rw but I won't have any ability to modify it... it's mapped down to nobody:nobody in the container - and besides, any operations as root inside the --fakeroot user namespace are going to be mapped to my dave user id on the host fs.

/dev/mapper/piran--vg-root on /etc/hosts type ext4 (rw,relatime,errors=remount-ro)
...
+ stat /etc/hosts
  File: /etc/hosts
  Size: 185         Blocks: 8          IO Block: 4096   regular file
Device: fd01h/64769d    Inode: 1573143     Links: 1
Access: (0644/-rw-r--r--)  Uid: (65534/  nobody)   Gid: (65534/  nobody)
Access: 2019-10-16 15:30:01.461885094 +0000
Modify: 2019-09-14 17:44:52.053989393 +0000
Change: 2019-09-14 17:44:52.053989393 +0000

Again the build completes without any issue.

[dtrudg]

@olifre - when you ran:

$ singularity build -f centos8b.sif centos8.def

.... were you root or an unpriv user there? Thinking about it, the root ownership instead of nobody on the stat /etc/hosts output looks like maybe you were root?

Could you use $ / # unpriv/root prompts or show whoami in your examples to make it easier for me to spot?

[olifre]

@dctrud Both commands were indeed run as root, I'll use # and $ consistently in the future :wink: .

The strange thing here is that yum should check the utime of /etc/hosts before and after the installation and check if it was updated to match the RPM contents. So clearly, for me it complains in the ro case, since the utime does not change. Now the million dollar question is why for you it works...

[dtrudg]

Here's a thought - on debian, maybe my /etc/hosts that is bound in is different enough from the file yum is expecting to update, so it leaves it alone and doesn't try to update it at all? That might explain why @DrDaveD couldn't reproduce either if his /etc/hosts is much customized.

[olifre]

That's a really good thought indeed!
In my case, it was an almost vanilla CentOS 7 installation (installed with Puppet, no modifications to /etc/hosts) and a fresh CentOS 7 VM on CERN's OpenStack, which also has an unmodified /etc/hosts.
So in your case and the case of @DrDaveD , yum probably just created an .rpmnew file and left the /etc/hosts untouched.

Now the question is:

• Is this a bug? It means building on different OS, or differently modified OS, may lead to a success or failure of the build.
• If it is a bug (and my feeling is that it is), how to "fix" it? I don't have a clean, "better" solution at hand - you could technically create a copy of /etc/hosts to a temporary FS and bind mount that instead of the hosts' /etc/hosts, which would fix the failure, but is cumbersome, and also, resulting containers would still differ by the presence of the .rpmnew file (that's not a real problem in practice, but rather a reproducibility issue).

I think the same could happen for etc/resolv.conf at least.
Do you have a good idea on this?

[dtrudg]

Okay - yes it is this - I can trigger it on the CentOS7 singularity vagrant box - which has a stock hosts file... and if I edit the hosts file then run the build I don't get the error.

Interestingly CentOS doesn't make an /etc/hosts.rpmnew - I can see it making other .rpmnew files though in the yum output.

We will have to think about the 'bug' status of this a bit. I should probably ask @cclerget what he thinks here.

[olifre]

Okay - yes it is this - I can trigger it on the CentOS7 singularity vagrant box - which has a stock hosts file... and if I edit the hosts file then run the build I don't get the error.

Perfect, that's it then!

Interestingly CentOS doesn't make an /etc/hosts.rpmnew - I can see it making other .rpmnew files though in the yum output.

Now that I think about it that might be "as advertised":
https://www.cl.cam.ac.uk/~jw35/docs/rpm_config.html
i.e. if the file did not actually change from the old rpm to the new one, and the file on disk was "edited", nothing is done at all if it is a %config or %config(noreplace) file.
However, if the file did not change between the old rpm and the new one, but the file on-disk is unedited, it is overwritten with the version from the RPM (i.e. the utime will change and RPM checks that).
So essentially, all the table cells in the link marked with File from update will trigger the issue for a config file bind-mounted into the host.

We will have to think about the 'bug' status of this a bit. I should probably ask @cclerget what he thinks here.

Agreed - let me know :smile:.

[dtrudg]

That's a great reference link. Thanks @olifre

[dtrudg]

Conferred with Cedric. The thinking is we should stage /etc/hosts and resolv.conf to tmpfs before binding into the container.

[dtrudg]

This is probably less of a critical bug than #4532 - but I think the staging to tmpfs and binding in rw can fix that too... and that one is definitely going to need to be fixed in master for a 3.5 release.

[cclerget]

Hey @olifre , can you give a try to #4629 ?

[olifre]

@dctrud Sorry for the delay!
I saw it's already in master, so I just tried with master and indeed the problem is fixed, as expexted :+1: .
Naturally, something like:

mv /etc/hosts /etc/hosts.foo

will still fail, but I don't see how that can be fixed nor why it would be needed (at least not with the way yum works right now).