buildah bud failing on COPY where docker build works

[msmigiel]

Description

I am trying to create a container with the following Docker file:

FROM websphere-liberty:webProfile7
ARG FILE
ARG DB_DRIVER_JAR
USER root
RUN mkdir -p /opt/ibm/jars /opt/ssfs/runtime
RUN chown -R default:root /opt/ibm/jars /opt/ssfs/runtime
USER default
COPY --chown=default:root $FILE /config/dropins
ADD --chown=default:root $DB_DRIVER_JAR /opt/ibm/jars

and the following command:

buildah bud -t mycluster.icp:8500/ibmcom/om-app-svt:ubi1 --build-arg FILE=ear.dir --build-arg DB_DRIVER_JAR=db2jcc4.jar -f /opt/ssfs/runtime/docker/docker-app/Dockerfile.om-app-os /opt/ssfs/runtime/docker/docker-app

I have also tried with podman build -t and that also fails.

Steps to reproduce the issue:

1. Using the Docker file above, try to build a container with buildah bud or podman build. Note the failure.
2. Try the same with docker build, note the success.

Describe the results you received:

The output from the above command is below. I have tried both with and without cache and it doesn't matter.

STEP 1: FROM websphere-liberty:webProfile7
STEP 2: ARG FILE
STEP 3: ARG DB_DRIVER_JAR
STEP 4: USER root
STEP 5: RUN mkdir -p /opt/ibm/jars /opt/ssfs/runtime
STEP 6: RUN chown -R default:root /opt/ibm/jars /opt/ssfs/runtime
STEP 7: USER default
STEP 8: COPY --chown=default:root $FILE /config/dropins
error building at step {Env:[PATH=/opt/ibm/wlp/bin:/opt/ibm/helpers/build:/opt/ibm/java/jre/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin JAVA_VERSION=1.8.0_sr5fp31 JAVA_HOME=/opt/ibm/java/jre IBM_JAVA_OPTIONS=-XX:+UseContainerSupport LIBERTY_VERSION=19.0.0_03 LOG_DIR=/logs WLP_OUTPUT_DIR=/opt/ibm/wlp/output RANDFILE=/tmp/.rnd JVM_ARGS=-Xshareclasses:name=liberty,nonfatal,cacheDir=/output/.classCache/ KEYSTORE_REQUIRED=true FILE=ear.dir DB_DRIVER_JAR=db2jcc4.jar] Command:copy Args:[ear.dir /config/dropins] Flags:[--chown=default:root] Attrs:map[] Message:COPY --chown=default:root $FILE /config/dropins Original:COPY --chown=default:root $FILE /config/dropins}: error creating directory "/var/lib/containers/storage/overlay/9119a7a96ed918dd6d3ad94a785cda5c1bc06b7e1b23f8ddb15ae6c9269c92d1/merged/config/dropins": chown /var/lib/containers/storage/overlay/9119a7a96ed918dd6d3ad94a785cda5c1bc06b7e1b23f8ddb15ae6c9269c92d1/merged/config/dropins: no such file or directory
[root@dub385002 docker]# echo ${COMMAND} -t mycluster.icp:8500/ibmcom/om-app-svt:ubi1 --build-arg FILE=ear.dir --build-arg DB_DRIVER_JAR=db2jcc4.jar -f ${ROOT_FOLDER}/docker-app/Dockerfile.om-app-os ${ROOT_FOLDER}/docker-app
buildah bud -t mycluster.icp:8500/ibmcom/om-app-svt:ubi1 --build-arg FILE=ear.dir --build-arg DB_DRIVER_JAR=db2jcc4.jar -f /opt/ssfs/runtime/docker/docker-app/Dockerfile.om-app-os /opt/ssfs/runtime/docker/docker-app

Describe the results you expected:

When I run the same command but with docker build, it works and I see this:

Sending build context to Docker daemon  718.4MB
Step 1/8 : FROM websphere-liberty:webProfile7
webProfile7: Pulling from library/websphere-liberty
Digest: sha256:ac7d913863021465f4a4370bfbff79eb47cb2f2dbd6efda4447637df31cb6a46
Status: Image is up to date for websphere-liberty:webProfile7
 ---> e6995314a475
Step 2/8 : ARG FILE
 ---> Running in ac5a4d00d3ea
Removing intermediate container ac5a4d00d3ea
 ---> 65488246bf3a
Step 3/8 : ARG DB_DRIVER_JAR
 ---> Running in a95d1f6884d8
Removing intermediate container a95d1f6884d8
 ---> b565260297df
Step 4/8 : USER root
 ---> Running in d14f6f30839e
Removing intermediate container d14f6f30839e
 ---> f158eeb65055
Step 5/8 : RUN mkdir -p /opt/ibm/jars && mkdir -p /opt/ssfs/runtime && chown -R default:root /opt/ibm/jars && chown -R default:root /opt/ssfs/runtime
 ---> Running in be66c3b5ea55
Removing intermediate container be66c3b5ea55
 ---> 1d6402cb2d0f
Step 6/8 : USER default
 ---> Running in 63f06e9ffda6
Removing intermediate container 63f06e9ffda6
 ---> 2b21ed712685
Step 7/8 : COPY --chown=default:root $FILE /config/dropins
 ---> b70a46bb60ea
Step 8/8 : ADD --chown=default:root $DB_DRIVER_JAR /opt/ibm/jars
 ---> 58c6e94fb346
Successfully built 58c6e94fb346
Successfully tagged mycluster.icp:8500/ibmcom/om-app-svt:test1

Output of rpm -q buildah or apt list buildah:

buildah-1.5-2.gite94b4f9.el7.x86_64

Output of buildah version:

Version:         1.5
Go Version:      go1.10.3
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  v0.7.0-alpha1
Git Commit:      e94b4f9
Built:           Mon Dec 10 12:15:42 2018
OS/Arch:         linux/amd64

Output of podman version if reporting a podman build issue:

Version:       0.12.1.2
Go Version:    go1.10.3
OS/Arch:       linux/amd64

Output of cat /etc/*release:

LSB_VERSION=core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.6"
Red Hat Enterprise Linux Server release 7.6 (Maipo)
Red Hat Enterprise Linux Server release 7.6 (Maipo)

Output of uname -a:

Linux hostname.com 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs.  Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""

# mountopt specifies extra mount options used when mounting the thin devices.
# mountopt = ""

# use_deferred_removal Marking device for deferred removal
# use_deferred_removal = "True"

# use_deferred_deletion Marking device for deferred deletion
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

[TomSweeneyRedHat]

@msmigiel Thanks for the issue.

Well, shoot. We've just very recently updated --build-args handling, as in yesterday, and I was hoping to tell you to try upstream given you're using an older Bulidah. However, it looks like we now have an issue with mkdir and the target directory if the directory exists. Need to run that down.

buildah --debug bud -t mycluster.icp:8500/ibmcom/om-app-svt:ubi1 --build-arg FILE=temp.txt --build-arg DB_DRIVER_JAR=db2jcc4.jar -f ~/Dockerfile .

error building at STEP "COPY --chown=default:root $FILE /config/dropins": error copying "/root/tsweeney/workspaces/containers/buildah/src/github.com/containers/buildah/temp.txt" to "/var/lib/containers/storage/overlay/7fc5d63d25db4cb1744f1b004b214d2bea077b37f42695d620ab9eaf34f3e22b/merged/config/dropins": mkdir /var/lib/containers/storage/overlay/7fc5d63d25db4cb1744f1b004b214d2bea077b37f42695d620ab9eaf34f3e22b/merged/config: file exists

[msmigiel]

You're welcome - please let me know if I can help out any further.

I also have a similar issue with another Docker file
Dockerfile.agent.txt

which is throwing this error in buildah but works fine in Docker:

STEP 17: RUN useradd -g 0 -u 1000 -m -d /opt/ssfs -s /bin/bash omsuser && echo "omsuser:omsuser" | /usr/sbin/chpasswd && echo "omsuser ALL=(ALL) NOPASSWD:ALL" | tee -a /etc/sudoers.d/omsuser && su omsuser -c "mkdir -p /opt/ssfs/runtime"
tee: /etc/sudoers.d/omsuser: No such file or directory
omsuser ALL=(ALL) NOPASSWD:ALL
error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin FILE=om-agent.tar.gz] Command:run Args:[useradd -g 0 -u 1000 -m -d /opt/ssfs -s /bin/bash omsuser && echo "omsuser:omsuser" | /usr/sbin/chpasswd && echo "omsuser ALL=(ALL) NOPASSWD:ALL" | tee -a /etc/sudoers.d/omsuser && su omsuser -c "mkdir -p /opt/ssfs/runtime"] Flags:[] Attrs:map[] Message:RUN useradd -g 0 -u 1000 -m -d /opt/ssfs -s /bin/bash omsuser && echo "omsuser:omsuser" | /usr/sbin/chpasswd && echo "omsuser ALL=(ALL) NOPASSWD:ALL" | tee -a /etc/sudoers.d/omsuser && su omsuser -c "mkdir -p /opt/ssfs/runtime" Original:RUN useradd -g 0 -u 1000 -m -d /opt/ssfs -s /bin/bash omsuser && echo "omsuser:omsuser" | /usr/sbin/chpasswd && echo "omsuser ALL=(ALL) NOPASSWD:ALL" | tee -a /etc/sudoers.d/omsuser && su omsuser -c "mkdir -p /opt/ssfs/runtime"}: error while running runtime: exit status 1

[TomSweeneyRedHat]

Thanks @msmigiel will take a look at the second Dockerfile too.

[msmigiel]

Hi @TomSweeneyRedHat , were you able to find anything more about this potential defect?

[TomSweeneyRedHat]

@msmigiel honestly I'd not looked at it specifically since my last comment due to other fires. However, a number of those fires were related to code in this same area of processing and I just ran your example now using the latest from upstream and it's working for me. Can you try pulling the latest from upstream and trying yourself?

[msmigiel]

Thanks @TomSweeneyRedHat for the reply. I did try version 1.8.0 and the same problems occurred. Is there a newer version I should try? Attached here are my 3 Docker files and 3 errors I am getting. Two of the three are the same and they are all about files not being found.

tee: /etc/sudoers.d/omsuser: No such file or directory

error building at STEP "COPY --chown=default:root $FILE /config/dropins": error creating directory

DockerFilesAndErrors.zip

[TomSweeneyRedHat]

@msmigiel if you could try 1.9-dev, that would be awesome. That just got spun up very, very recently.

[msmigiel]

@TomSweeneyRedHat I've also tried the build buildah-1.9.0-0.1.dev.gitddbd805.fc31.x86_64.rpm from rpmfind.net and it is having the same issue.

[TomSweeneyRedHat]

@msmigiel thanks for trying, will try to pop it up higher on the list then.

[TomSweeneyRedHat]

I've a suspicion this and #1578 are related.

[TomSweeneyRedHat]

@msmigiel FWIW, I tried to test this today, but every time I ran your Dockerfile my VM ran out of disk space. I'll see if I can create a Dockerfile that's less disk intensive tomorrow.

[vbashyamus]

@TomSweeneyRedHat Based on the comments in #1578 is our issue because of #1583

[TomSweeneyRedHat]

@vbashyamus definite maybe. I'm hopeful, but as I've not been able to recreate your issue due to unrelated system problems, I can not say for sure. But my fingers are crossed.

[TomSweeneyRedHat]

@msmigiel if you've the time and patience, can I ask you to try a couple of things for me.

I used this Dockerfile as the last line from yours was filling up the disk on my small VM's and the error you were running into was in the next to last line.

# cat ~/Dockerfile.badcopyfull
FROM websphere-liberty:webProfile7
ARG FILE
ARG DB_DRIVER_JAR
USER root
RUN mkdir -p /opt/ibm/jars /opt/ssfs/runtime
RUN chown -R default:root /opt/ibm/jars /opt/ssfs/runtime
USER default
COPY --chown=default:root $FILE /config/dropins

and used this command:

buildah bud -t tom -f ~/Dockerfile.badcopyfull --build-arg FILE=newfakefile.txt .

Which worked for me on the latest Buildah from upstream, but failed for buildah v1.8.2 and earlier. Also if I changed the FROM command in the Dockerfile from websphere-liberty:webProfiel7 to "Fedora" and default:root to bin:root, that worked for me with both upstream AND Buildah 1.8.*. So I think something with the way we handled file systems in Ubuntu, which is what the websphere-liberty is based on changed recently but I've not chased the root cause.

So if you have the time to test again after pulling the absolute latest Buildah from GitHub, and/or try changing your Dockerfile to Fedora and bin:root as an experiment, that would be great info to have. If you don't have time to do so, no worries.

[msmigiel]

Hi @TomSweeneyRedHat , I'm still getting file exists errors when I try, even with buildah 1.9:

# buildah version
Version:         1.9.0-dev
Go Version:      go1.11.5
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  v0.7.0-rc2
Git Commit:      0e30da6
Built:           Tue Apr 30 08:42:59 2019
OS/Arch:         linux/amd64

# cat Dockerfile.badcopyfull
FROM websphere-liberty:webProfile7
ARG FILE
ARG DB_DRIVER_JAR
USER root
RUN mkdir -p /opt/ibm/jars /opt/ssfs/runtime
RUN chown -R default:root /opt/ibm/jars /opt/ssfs/runtime
USER default
COPY --chown=default:root $FILE /config/dropins

# buildah bud -t tom -f ~/Dockerfile.badcopyfull --build-arg FILE=newfakefile.txt .
STEP 1: FROM websphere-liberty:webProfile7
STEP 2: ARG FILE
STEP 3: ARG DB_DRIVER_JAR
STEP 4: USER root
STEP 5: RUN mkdir -p /opt/ibm/jars /opt/ssfs/runtime
STEP 6: RUN chown -R default:root /opt/ibm/jars /opt/ssfs/runtime
STEP 7: USER default
STEP 8: COPY --chown=default:root $FILE /config/dropins
error building at STEP "COPY --chown=default:root $FILE /config/dropins": error copying "/root/newfakefile.txt" to "/var/lib/containers/storage/overlay/428f2932f1731ec81100d2e2e6992046083fa9e2fe4547731e8c2b6303aaddfe/merged/config/dropins": mkdir /var/lib/containers/storage/overlay/428f2932f1731ec81100d2e2e6992046083fa9e2fe4547731e8c2b6303aaddfe/merged/config: file exists

However, the Fedora edits you suggest do work. If that is the case, does this mean something is wrong with Ubuntu based images or something is wrong with buildah?

# cat Dockerfile.fedora
FROM fedora
ARG FILE
ARG DB_DRIVER_JAR
USER root
RUN mkdir -p /opt/ibm/jars /opt/ssfs/runtime
RUN chown -R bin:root /opt/ibm/jars /opt/ssfs/runtime
USER default
COPY --chown=bin:root $FILE /config/dropins

# buildah bud -t tom -f ~/Dockerfile.fedora --build-arg FILE=newfakefile.txt .
STEP 1: FROM fedora
STEP 2: ARG FILE
STEP 3: ARG DB_DRIVER_JAR
STEP 4: USER root
STEP 5: RUN mkdir -p /opt/ibm/jars /opt/ssfs/runtime
STEP 6: RUN chown -R bin:root /opt/ibm/jars /opt/ssfs/runtime
STEP 7: USER default
STEP 8: COPY --chown=bin:root $FILE /config/dropins
STEP 9: COMMIT tom
Getting image source signatures
Copying blob b7ba3be6a0d6 skipped: already exists
Copying blob 126eccbcb087 done
Copying config cc3afc9b40 done
Writing manifest to image destination
Storing signatures
--> cc3afc9b4074a8749d38b00702e7f6474632eb8e63806502efc87f33316e6780

[TomSweeneyRedHat]

@msmigiel thanks for the info. I decided to try a brand new VM and I too am seeing the error now with v1.9-dev. I'm not sure why I was not seeing it before. I also tried replacing websphere-liberty:webProfile7 wiith ubuntu and that also completed without an issue, just like the fedora test. If you've time could you try just ubuntu too and see if that works for you?

I don't know much about the websphere-liberty image. Can you tell me where that comes from and who creates it? I noticed that that paricular image uses an older variant of Ubuntu:

buildah run websphere-liberty-working-container cat /etc/*release
/bin/cat: /etc/fedora-release: No such file or directory
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

while the must current Ubuntu image that I found has:

# buildah run ubuntu-working-container cat /etc/*release
/bin/cat: /etc/fedora-release: No such file or directory
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Would it be possible to upgrade Unbuntu in the websphere image? @vrothberg is this ringing any Ubuntu bells in your head?

[msmigiel]

Hi @TomSweeneyRedHat - Liberty is an application server for J2EE based programs, similar to Weblogic or Jboss. Liberty is similar to IBM's own Websphere but smaller and easier to deploy. I work in testing a J2EE based application that runs on it. I don't know who maintains the Liberty Docker image but I'll ask around and see if there is a reason why it is not on the most recent Ubuntu LTS.

https://hub.docker.com/_/websphere-liberty/?tab=description

When I try to use a Dockerfile with ubuntu instead of fedora, that works too, so it's possible there is something off with images based on older versions of Ubuntu.

Edit:
It turns out that Liberty itself builds from ibmjava:8-jre, which uses an old version of Ubuntu.

https://github.com/ibmruntimes/ci.docker/blob/c7b157236cc7b590b85c0498ea1c7cbe114382d3/ibmjava/8/jre/ubuntu/Dockerfile

[msmigiel]

Hi @TomSweeneyRedHat - has there been any progress on this? We would like to make use of builah and podman but unfortunately they aren't working for us in any of our use cases.

[TomSweeneyRedHat]

@msmigiel sorry, I've been pulled into a couple other fire fights and haven't made much progress on this in particular. However, I 'm crossing my fingers that #1630 might fix this, but I've not had a chance to test it.

[TomSweeneyRedHat]

@msmigiel I finally got my head above water and tried testing this again today. I'm not able to reproduce again. However we've made a number of changes to Buildah and it looks like webspher-liberty:webProfile7 has been updated as of June 21. Can I ask you to retry once again with Buildah v1.9.2 please? That should be the latest available version.

[smith-kyle]

Can confirm this is still an issue in Buildah v1.11.2.

Here's a simple example
Output of ls

Dockerfile  a_file.txt

Output of cat Dockerfile

FROM node:slim

ARG my_arg
COPY ${my_arg} /file.txt

Output of buildah bud --build-arg my_arg=a_file.txt .

STEP 1: FROM node:slim
STEP 2: ARG my_arg
STEP 3: COPY ${my_arg} /file.txt
error dry-running "COPY ${my_arg} /file.txt": no files found matching "/temp/${my_arg}": no such file or directory

Now I'll change my Dockerfile to copy the file using a static path rather than a build arg and output the build arg to a file just to show that it was passed in correctly
Output of cat Dockerfile

FROM node:slim
ARG my_arg
COPY a_file.txt /file.txt
RUN echo ${my_arg} > output

Output of buildah bud --build-arg my_arg=a_file.txt .

STEP 1: FROM node:slim
STEP 2: ARG my_arg
STEP 3: COPY a_file.txt /file.txt
STEP 4: RUN echo ${my_arg} > output
STEP 5: COMMIT
Getting image source signatures
Copying blob e9dc98463cd6 skipped: already exists
Copying blob 0e0d79e2c080 skipped: already exists
Copying blob bfb6455a0ec4 skipped: already exists
Copying blob b157e0da2944 skipped: already exists
Copying blob 0173bef8938f skipped: already exists
Copying blob 0f1e25f55fb8 done
Copying config 29d704d3cc done
Writing manifest to image destination
Storing signatures
29d704d3ccf0c1cfb217a97f1ae4bd96b77a345b00a8f21783a988ea5cf947e5

Now I check that the build arg was written to the output file

[root@1fa24373aab7 temp]# podman run 29d704d3ccf0c1cfb217a97f1ae4bd96b77a345b00a8f21783a988ea5cf947e5 cat output
a_file.txt

Output of cat /etc/containers/libpod.conf

cgroup_manager="cgroupfs"
events_logger="file"

Output of cat /etc/containers/storage.conf

# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "vfs"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs.  Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""

# mountopt specifies extra mount options used when mounting the thin devices.
# mountopt = ""

# use_deferred_removal Marking device for deferred removal
# use_deferred_removal = "True"

# use_deferred_deletion Marking device for deferred deletion
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

Output of buildah --version

buildah version 1.11.2 (image-spec 1.0.1, runtime-spec 1.0.1-dev)

[jduncan-rva]

+1
Same behavior with buildah-1.11.3-2.fc30.x86_64

At least the scenario of using a build-arg in a COPY command fails because the argument isn't evaluated.

COPY requirements.txt entrypoint.sh workshops/${workshop_name} /opt/docs/ Error: error dry-running "COPY requirements.txt entrypoint.sh workshops/${workshop_name} /opt/docs/": no files found matching "/home/jduncan/Code/labguide/workshops/${workshop_name}": no such file or directory

[TomSweeneyRedHat]

@jduncan-rva hmm, 1.11.3 should not be doing that still. Can you do a 'rpm -qa | grep buildah' please? Also if you could share your Dockerfile and buildah bud command that would be great.

[TomSweeneyRedHat]

FWIW, #1874 should have fixed the --build-arg issue and that landed (supposedly) in v1.11.3.

[jduncan-rva]

rpm -qa | grep buildah
buildah-1.11.3-2.fc30.x86_64

Confirmed this worked cleanly with buildah bud. Thanks!

[TomSweeneyRedHat]

@jduncan-rva probably a lack of tea in my system. So this is working for you now in Buildah 1.11.3 right?

[sgallagher]

@TomSweeneyRedHat I am still seeing this when doing podman build on

buildah-1.11.3-2.fc31.x86_64
podman-1.6.1-5.fc31.x86_64

I don't know if it matters, but I'm encountering it from an ADD command rather than COPY. (I need to use ADD because I'm having it auto-extract a tarball.)

[TomSweeneyRedHat]

Thanks @sgallagher and @jduncan-rva I'll take a look from the Podman side. I did a quick run with Buildah 1.11.3 directly and it seemed OK.

[sgallagher]

@TomSweeneyRedHat If you want to reproduce it, you can clone https://github.com/fedora-modularity/libmodulemd, install podman-docker so that it owns the /usr/bin/docker command, and run TRAVIS_JOB_NAME="Fedora 31" ./.travis/travis-fedora.sh in the cloned directory to see what I see.

[jduncan-rva]

A little more thorough answer:

buildah bud works:

rpm -qa | grep buildah
buildah-1.11.3-2.fc30.x86_64

relevant lines from Dockerfile:

ARG workshop_name=example-workshop
COPY requirements.txt entrypoint.sh workshops/$workshop_name /opt/docs/

Buildah bud works just fine.

 buildah bud --build-arg workshop_name=ansible-for-devops -t quay.io/jduncan/operator-workshop-lab-guide-ansible-for-devops .
STEP 1: FROM centos:7
STEP 2: ARG workshop_name=example-workshop
...
STEP 9: COPY requirements.txt entrypoint.sh workshops/$workshop_name /opt/docs/

However, podman doesn't seem to work properly yet:

 rpm -qa | grep podman
podman-docker-1.6.1-2.fc30.noarch
podman-1.6.1-2.fc30.x86_64
podman-manpages-1.6.1-2.fc30.noarch

 podman build --build-arg workshop_name=ansible-for-devops -t quay.io/jduncan/operator-workshop-lab-guide-ansible-for-devops .
STEP 1: FROM centos:7
STEP 2: ARG workshop_name=example-workshop
...
STEP 9: COPY requirements.txt entrypoint.sh workshops/$workshop_name /opt/docs/
Error: error dry-running "COPY requirements.txt entrypoint.sh workshops/$workshop_name /opt/docs/": no files found matching "/home/jduncan/Google Drive/Code/creynolds-labguide/workshops/$workshop_name": no such file or directory

[rhatdan]

Could this have something to do with layers.
podman build defaults to layers==true. buildah bud layers==false.

If you attempt to build with podman build --layers=false, does it succeed? Similarly buildah bud --layers=false, does it fail?

[sgallagher]

Could this have something to do with layers.
podman build defaults to layers==true. buildah bud layers==false.

If you attempt to build with podman build --layers=false, does it succeed? Similarly buildah bud --layers=false, does it fail?

It still fails with podman build --layers=false and still succeeds with buildah bud --layers=true.

[rhatdan]

Have you checked to make sure you are running the same version of buildah in both cases.

buildah version
podman info | grep -i Buildah

[sgallagher]

OK, it appears that podman is still using BuildahVersion: 1.11.2 while this issue was apparently fixed in 1.11.3.

Why is podman bundling buildah rather than using the system version?

[sgallagher]

OK, podman 1.6.2 which bundles buildah 1.11.3 works properly for me. (Just pulled it from https://koji.fedoraproject.org/koji/packageinfo?packageID=26289 )

[TomSweeneyRedHat]

@sgallagher @jduncan-rva thanks for all the digging on this issue.

Under the covers of Podman, we use Buildah's API for the 'podman bulid' command. That command calls the 'buildah bud' api to do that work for it as both tools share the same container images. To make that happen we have to vendor a version of Buildah into Podman.

This should be fixed then in the next release of Podman, I'm not sure what the version will be atm though.

[rhatdan]

Fixed in podman 1.6.2 so closing.

[jduncan-rva]

Confirmed this works in 1.6.2