Buildah: USERADD errors with lock file already used

[TidyData]

USERADD errors with lock file already used

Steps to reproduce the issue:

1. Run the below Bash Buildah Script

#!/bin/bash
set -x
fahcont=$(buildah from registry.access.redhat.com/ubi8/ubi:latest)
buildah run "$fahcont" -- bash -c 'useradd --create-home --user-group --shell /usr/bin/bash fahclient'
buildah run "$fahcont" -- bash -e <<- EOF
    useradd -ms /bin/bash fahclient
EOF
buildah commit --rm "$fahcont" "test"

Describe the results you received:
It is producing errors with regards to passwd file lock

++ buildah from registry.access.redhat.com/ubi8/ubi:latest
+ fahcont=ubi-working-container-2
+ buildah run ubi-working-container-2 -- bash -c 'useradd --create-home --user-group --shell /usr/bin/bash fahclient'
useradd: /etc/passwd.1: lock file already used
useradd: cannot lock /etc/passwd; try again later.
ERRO exit status 1                                
+ buildah run ubi-working-container-2 -- bash -e
useradd: lock /etc/passwd.lock already used by PID 1
useradd: cannot lock /etc/passwd; try again later.
ERRO exit status 1                                
+ buildah commit --rm ubi-working-container-2 test

Describe the results you expected:

I expected a user fahclient to be created within the container (not on the base OS).

$rpm -q buildah
buildah-1.11.6-4.module+el8.1.1+5259+bcdd613a.x86_64

$ buildah version
Version:         1.11.6
Go Version:      go1.12.12
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.1-dev
CNI Spec:        0.4.0
libcni Version:  
image Version:   5.0.0
Git Commit:      
Built:           Wed Dec 31 16:00:00 1969
OS/Arch:         linux/amd64

```bash
$ cat /etc/*release
NAME="Red Hat Enterprise Linux"
VERSION="8.1 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.1"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.1 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.1:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.1
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.1"
Red Hat Enterprise Linux release 8.1 (Ootpa)
Red Hat Enterprise Linux release 8.1 (Ootpa)

```bash
$ uname -a
Linux dev-0 4.18.0-147.5.1.el8_1.x86_64 #1 SMP Tue Jan 14 15:50:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

$ cat /etc/containers/storage.conf
# This file is is the configuration file for all tools
# that use the containers/storage library.
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
#mount_program = "/usr/bin/fuse-overlayfs"

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

# mountopt specifies comma separated list of extra mount options
# mountopt = "nodev,metacopy=on"

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs.  Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base.
# device.
# mkfsarg = ""

# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver 
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"

# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

# If specified, use OSTree to deduplicate files with the overlay backend
ostree_repo = ""

# Set to skip a PRIVATE bind mount on the storage home directory.  Only supported by
# certain container storage drivers
skip_mount_home = "false"

[rhatdan]

I ran your script on Fedora 32 and I get

$ /tmp/build.sh 
++ buildah from registry.access.redhat.com/ubi8/ubi:latest
+ fahcont=ubi-working-container-1
+ buildah run ubi-working-container-1 -- bash -c 'useradd --create-home --user-group --shell /usr/bin/bash fahclient'
+ buildah run ubi-working-container-1 -- bash -e
useradd: user 'fahclient' already exists
ERRO exit status 9                                
+ buildah commit --rm ubi-working-container-1 test
Getting image source signatures
Copying blob 92b864bfcfaa skipped: already exists  
Copying blob 92af68d64d0b skipped: already exists  
Copying blob d78247e208c9 done  
Copying config 2d113f5494 done  
Writing manifest to image destination
Storing signatures
2d113f54945361d9e293f5f47ff3046b5c7c8f27d45946a303ea3c6f6144bd02
[dwalsh@localhost f31]$ 

$ rpm -q buildah fuse-overlayfs
buildah-1.14.3-1.fc32.x86_64
fuse-overlayfs-0.7.8-1.fc32.x86_64

This might be an issue in fuse-overlayfs, if you are running it in rootless mode.
@giuseppe WDYT

[TidyData]

yes I'm running in rootless mode within main base RHEL 8 environment. I've toggled SELINUX and it still occurs. I put two ways to do useradd, so only use one of them...

your user 'faclient' already exists error shows that it worked for the first useradd. How can I replace fuse-overlayfs to see if that is the problem?

[lazy-developer]

Even I'm facing the same issue in the rootless mode within Red Hat Enterprise Linux release 8.1 (Ootpa)

useradd: /etc/passwd.8: lock file already used
useradd: cannot lock /etc/passwd; try again later.

P.S. - I've already ran _buildah unshare_

_fuse-overlayfs_ details

fuse-overlayfs -V
fuse-overlayfs: version 0.7.2
FUSE library version 3.2.1
using FUSE kernel interface version 7.26

@rhatdan Can you please throw some light on this, thanks.

[rhatdan]

Can you try this with the RHEL8.2 beta?

[lazy-developer]

Can you try this with the RHEL8.2 beta?

We're being partnered with RedHat and I don't think so proceeding with beta OS would be good idea for production! that's why using RHEL 8.1 and thereby subscribing with our credentials.

Also, in rootless mode, I'm facing the following error while using podman build

podman build -f containerfile
STEP 12: COMMIT quay.io/xyz/xyz:latest
e60fe50bc2fa384199e9515b51bbefe7ac20f8af9316c5f7e3d380e21fe93f66
ERRO[0245] unable to close namespace: "close /proc/43003/ns/user: bad file descriptor"

Reading ample blogs/forums, I guess it is related to _fuse-overlays_

fuse-overlayfs -V
fuse-overlayfs: version 0.7.2
FUSE library version 3.2.1
using FUSE kernel interface version 7.26

I tried upgrading to 0.7.6, but it says architecture not supported.
Is this somehow related? https://github.com/containers/fuse-overlayfs/issues/183
Would downgrading help?
Please guide, thank you.

[rhatdan]

I am just looking to make sure the issue is fixed in rhel8.2 release.

BTW There is a RHEL8.1.1 version available, Not sure if this is what you are testing.

There were lots of fixes in fuse-overlayfs which comes in RHEL8.2.

[TidyData]

@rhatdan I have confirmed my buildah script works as initially expected (adds user appropriately) in the RHEL 8.2 beta within a virtual machine.

[<redacted> ~]$ rpm -q buildah
buildah-1.11.6-4.module+el8.2.0+5169+e8b0c9ba.x86_64
[<redacted> ~]$ buildah version
Version:         1.11.6
Go Version:      go1.13.4
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.1-dev
CNI Spec:        0.4.0
libcni Version:  
image Version:   5.0.0
Git Commit:      
Built:           Wed Dec 31 16:00:00 1969
OS/Arch:         linux/amd64
[<redacted> ~]$ cat /etc/*release
NAME="Red Hat Enterprise Linux"
VERSION="8.2 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.2"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.2 Beta (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.2:beta"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.2 Beta"
Red Hat Enterprise Linux release 8.2 Beta (Ootpa)
Red Hat Enterprise Linux release 8.2 Beta (Ootpa)
[<redacted> ~]$ uname -a
Linux <redacted> 4.18.0-167.el8.x86_64 #1 SMP Sun Dec 15 01:24:23 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

looking forward to the 8.2 release!

[lazy-developer]

I am just looking to make sure the issue is fixed in rhel8.2 release.

BTW There is a RHEL8.1.1 version available, Not sure if this is what you are testing.

There were lots of fixes in fuse-overlayfs which comes in RHEL8.2.

Is RHEL 8.1.1/8.2 is available on Azure/AWS, I'm spinning up the server there, we've a project plan for production and that's why we partnered with RedHat.

Current Version:-

cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.1 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.1"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.1 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.1:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.1
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.1"

Podman Version:-

podman version
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.12.12
OS/Arch:            linux/amd64

Buildah Version:-

buildah version
Version:         1.11.6
Go Version:      go1.12.12
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.1-dev
CNI Spec:        0.4.0
libcni Version:
image Version:   5.0.0
Git Commit:
Built:           Thu Jan  1 00:00:00 1970
OS/Arch:         linux/amd64

Performing:-
podman build -f containerfile
NOTE:- there are many builds for different containerfiles as such

We are facing following 2 issues while building images via podman build in rootless mode:-

1. The following error occur every time while committing image:-
   ERRO[0245] unable to close namespace: "close /proc/43003/ns/user: bad file descriptor"
2. One of the containerfile has "useradd" command which fails with following error:-

useradd: /etc/passwd.8: lock file already used
useradd: cannot lock /etc/passwd; try again later.

Please guide, thanks.

[rhatdan]

I believe you can get access to RHEL8.2 beta, but it is not released yet.

[TidyData]

@lazy-developer @rhatdan I have already verified the issue is resolved in 8.2 beta.

• Shall we close this issue as the 8.2 release will resolve it?
• Any guesses on 8.2 release date?

[rhatdan]

No later then the end of May, I believe, Releases usually happen around the time of Red Hat summit.

[TidyData]

that feels a long time to wait, any suggestions on updating my fuse-overlayfs on 8.1.1?

@rhatdan I see lots of people clammering for your attention, appreciate what you do!

podman and buildah are great!

[rhatdan]

Probbaly just check out out the repo and compile it is best.

[saprette]

@TidyData did you manage to have it working? Are you using buildah from a container?

@rhatdan I switched to RHEL 8.2 for the USERADD issue

I still have this issue with GROUPADD or USERADD

I use this image for buildah build in a Tekton task on OpenShift 4.4.5
registry.redhat.io/rhel8/buildah:8.2

[build-buildah] STEP 1: FROM registry.access.redhat.com/ubi8/ubi:8.2
[build-buildah] Getting image source signatures
[build-buildah] Copying blob sha256:58e1deb9693dfb1704ccce2f1cf0e4d663ac77098a7a0f699708a71549cbd924
[build-buildah] Copying blob sha256:78afc5364ad2c981e4a4919f535aaefef9ac2f990837be01c766764e025b1f31
[build-buildah] Copying config sha256:8e0b0194b7e1142c76aa26d71264d3561fab1b039fd3c49d9083dc9d3f42068f
[build-buildah] Writing manifest to image destination
[build-buildah] Storing signatures
[build-buildah] STEP 2: ENV EAP_HOME="/opt/eap/EAP-7.2.2/jboss-eap-7.2/" JAVA_HOME="/etc/alternatives/java_sdk_${JAVA_MAIN_VERSION}_openjdk/" ANT_HOME="/usr/share/ant"
[build-buildah] 31ad7c84b7248986d0fc7f75e739b16eaaf2df5266d0a4c47c3a90db65c44c8d
[build-buildah] STEP 3: RUN rm -f /etc/localtime && ln -s /usr/share/zoneinfo/Europe/Luxembourg /etc/localtime
[build-buildah] bdc2b553ddf6cd13c02ceff7e20cb358b6146900ccd1b0ef6f7f6cb377edd4a3
[build-buildah] STEP 4: RUN /usr/sbin/groupadd aserver &&      /usr/sbin/useradd -g aserver --no-create-home --shell /bin/bash asowner 
[build-buildah] groupadd: /etc/group.246: lock file already used
[build-buildah] groupadd: cannot lock /etc/group; try again later.
[build-buildah] subprocess exited with status 10
[build-buildah] subprocess exited with status 10
[build-buildah] error building at STEP "RUN /usr/sbin/groupadd aserver &&      /usr/sbin/useradd -g aserver --no-create-home --shell /bin/bash asowner": exit status 10

And same with USERADD

[build-buildah] STEP 1: FROM registry.access.redhat.com/ubi8/ubi:8.2
[build-buildah] Getting image source signatures
[build-buildah] Copying blob sha256:78afc5364ad2c981e4a4919f535aaefef9ac2f990837be01c766764e025b1f31
[build-buildah] Copying blob sha256:58e1deb9693dfb1704ccce2f1cf0e4d663ac77098a7a0f699708a71549cbd924
[build-buildah] Copying config sha256:8e0b0194b7e1142c76aa26d71264d3561fab1b039fd3c49d9083dc9d3f42068f
[build-buildah] Writing manifest to image destination
[build-buildah] Storing signatures
[build-buildah] STEP 2: ENV EAP_HOME="/opt/eap/EAP-7.2.2/jboss-eap-7.2/" JAVA_HOME="/etc/alternatives/java_sdk_${JAVA_MAIN_VERSION}_openjdk/" ANT_HOME="/usr/share/ant"
[build-buildah] d9a0368770797df24e637258a37f632fd04be174e75e0588eaa1b5a2055b05c1
[build-buildah] STEP 3: RUN rm -f /etc/localtime && ln -s /usr/share/zoneinfo/Europe/Luxembourg /etc/localtime
[build-buildah] fc0da5651ab986363d05522651bab35e01e989052f7a3a5cb2588e14140a3727
[build-buildah] STEP 4: RUN /usr/sbin/useradd --no-create-home --shell /bin/bash asowner
[build-buildah] useradd: /etc/passwd.244: lock file already used
[build-buildah] useradd: cannot lock /etc/passwd; try again later.
[build-buildah] subprocess exited with status 1
[build-buildah] subprocess exited with status 1
[build-buildah] error building at STEP "RUN /usr/sbin/useradd --no-create-home --shell /bin/bash asowner": exit status 1

[saprette]

Note that I don't have any issue using USERADD with buildah from a container if that container is ubuntu, but I do have the issue reported above when using RHEL 8.2.

So this works for me:

• Containers running in OpenShift 4.4.5
• A Tekton task with a step using an image created like so

sudo buildah bud -f Dockerfile -t private-repo/ocp/cs-buildah-build:0.0.1

Dockerfile

FROM docker.io/ubuntu:20.10

RUN apt update && apt-get install --yes wget gnupg2 \
    && . /etc/os-release \
    && sh -c "echo 'deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/ /' > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list" \
    && wget -nv https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_20.04/Release.key -O Release.key \
    && apt-key add - < Release.key \
    && apt-get update -qq \
    && apt-get -qq -y install buildah

This is the Tekton task

    - command:
        - buildah
        - bud
        - '--tls-verify=$(params.sslVerify)'
        - '--layers'
        - '-f'
        - "$(workspaces.source.path)/$(params.subdirectory)/eap_generic_build/Dockerfile"
        - '-t'
        - $(params.output_image_url)
        - .
      image: private-repo/ocp/cs-buildah-build:0.0.1
      name: build-buildah
      resources: {}
      securityContext:
        privileged: true
      volumeMounts:
        - mountPath: /var/lib/containers
          name: varlibcontainers

With this task, I can build an image with this dockerfile that containers a USERADD

FROM registry.access.redhat.com/ubi8/ubi:8.2

ENV EAP_HOME="/opt/eap/EAP-7.2.2/jboss-eap-7.2/" JAVA_HOME="/etc/alternatives/java_sdk_${JAVA_MAIN_VERSION}_openjdk/" ANT_HOME="/usr/share/ant"

RUN rm -f /etc/localtime && ln -s /usr/share/zoneinfo/Europe/Luxembourg /etc/localtime

RUN /usr/sbin/useradd --no-create-home --shell /bin/bash asowner

Build logs

[build-buildah] STEP 1: FROM registry.access.redhat.com/ubi8/ubi:8.2
[build-buildah] Getting image source signatures
[build-buildah] Copying blob sha256:58e1deb9693dfb1704ccce2f1cf0e4d663ac77098a7a0f699708a71549cbd924
[build-buildah] Copying blob sha256:78afc5364ad2c981e4a4919f535aaefef9ac2f990837be01c766764e025b1f31
[build-buildah] Copying config sha256:8e0b0194b7e1142c76aa26d71264d3561fab1b039fd3c49d9083dc9d3f42068f
[build-buildah] Writing manifest to image destination
[build-buildah] Storing signatures
[build-buildah] STEP 2: ENV EAP_HOME="/opt/eap/EAP-7.2.2/jboss-eap-7.2/" JAVA_HOME="/etc/alternatives/java_sdk_${JAVA_MAIN_VERSION}_openjdk/" ANT_HOME="/usr/share/ant"
[build-buildah] --> da30c6b5e4a
[build-buildah] STEP 3: RUN rm -f /etc/localtime && ln -s /usr/share/zoneinfo/Europe/Luxembourg /etc/localtime
[build-buildah] --> 5e3f8e80e5a
[build-buildah] STEP 4: RUN /usr/sbin/useradd --no-create-home --shell /bin/bash asowner
[build-buildah] --> dd99a66c641

So building an image with a USERADD from ubuntu:20.10 works fine with buildah but does not work with ubi:8.2.

We'll research for any matching known issue reported in RedHat tickets, and if we find none, we'll raise a RedHat support case.

[rhatdan]

@giuseppe Do you think this issue is caused because of fuse-overlayfs?
@saprette What is the version of fuse-overlayfs on RHEL8.2?

[saprette]

@rhatdan in registry.redhat.io/rhel8/buildah:8.2

sh-4.4$ yum list fuse*
2020-06-02 15:21:26,705 [ERROR] yum:106:MainThread @logutil.py:194 - [Errno 13] Permission denied: '/var/log/rhsm/rhsm.log' - Further logging output will be written to stderr
Not root, Subscription Management repositories not updated
Last metadata expiration check: 0:01:48 ago on Tue Jun  2 15:19:38 2020.
Installed Packages
fuse-overlayfs.x86_64                                                                       0.7.2-5.module+el8.2.0+6368+cf16aa14                                                                       @RHEL-8.2.0-updates-20200423.0-AppStream-1
fuse3-libs.x86_64                                                                           3.2.1-12.el8                                                                                               @RHEL-8.2.0-updates-20200423.0-BaseOS-1   
Available Packages
fuse.x86_64                                                                                 2.9.7-12.el8                                                                                               ubi-8-baseos                              
fuse-common.x86_64                                                                          3.2.1-12.el8                                                                                               ubi-8-baseos                              
fuse-libs.i686                                                                              2.9.7-12.el8                                                                                               ubi-8-baseos                              
fuse-libs.x86_64                                                                            2.9.7-12.el8                                                                                               ubi-8-baseos                              
fuse3.x86_64                                                                                3.2.1-12.el8                                                                                               ubi-8-baseos                              
fuse3-devel.i686                                                                            3.2.1-12.el8                                                                                               ubi-8-baseos                              
fuse3-devel.x86_64                                                                          3.2.1-12.el8                                                                                               ubi-8-baseos                              
fuse3-libs.i686                                                                             3.2.1-12.el8                                                                                               ubi-8-baseos               

For buildah

sh-4.4$ yum list buildah*
2020-06-02 15:23:53,925 [ERROR] yum:159:MainThread @logutil.py:194 - [Errno 13] Permission denied: '/var/log/rhsm/rhsm.log' - Further logging output will be written to stderr
Not root, Subscription Management repositories not updated
Last metadata expiration check: 0:04:15 ago on Tue Jun  2 15:19:38 2020.
Installed Packages
buildah.x86_64                                                                          1.11.6-8.module+el8.2.0+6368+cf16aa14                                                                          @RHEL-8.2.0-updates-20200423.0-AppStream-1

[rhatdan]

RHEL8.2 I believe has fuse-overlayfs from January. RHEL8.2.1 will just to fuse-overlayfs 1.0

[giuseppe]

It is an issue in the RHEL kernel and that got fixed. We have also added a custom patch to fuse-overlays to workaround it.

[rhatdan]

So this will be fixed in RHEL8.2.1 then correct?

[giuseppe]

Yes, correct

[nnachefski]

FYI, when doing tekton builds using the buildah ClusterTask, you have to override the storage driver to get it to work. You have to add '--storage-driver=overlay'

You can use this patch:

# work-around bug in buildah ClusterTask by manually setting storage-driver to overlay
$OC_CLIENT patch clustertask buildah --type merge -p '{"spec":{"steps":[{"command":["buildah","bud","--storage-driver=overlay","--format=$(params.FORMAT)","--tls-verify=$(params.TLSVERIFY)","--layers","-f","$(params.DOCKERFILE)","-t","$(resources.outputs.image.url)","$(params.CONTEXT)"],"image":"$(params.BUILDER_IMAGE)","name":"build","resources":{},"securityContext":{"privileged":true},"volumeMounts":[{"mountPath":"/var/lib/containers","name":"varlibcontainers"}],"workingDir":"/workspace/source"}]}}'