Quarkus: Quarkus NativeImageBuildStep fails with perm denied with podman 2.x

[jerboaa]

Describe the bug
This bug originated here:
https://bugzilla.redhat.com/show_bug.cgi?id=1854738
and here:
https://groups.google.com/g/quarkus-dev/c/8LVoqk4G5uc/m/M5MAVopLAwAJ

Basically with podman 1.x --userns=keep-id contained a bug which got fixed with 2.x which results in quarkus build failures with options -Pnative -Dquarkus.native.container-build=true -Dquarkus.native.container-runtime=podman in rootless mode.

Expected behavior
Native image build passes. No "Permission Denied" error.

Actual behavior
Native image build in the container with mounted volume fails due to "Permission Denied".

To Reproduce
Steps to reproduce the behavior:

1. Install podman 2.0 (for example in F32)
2. mvn package -Pnative -Dquarkus.native.container-build=true -Dquarkus.native.container-runtime=podman

Additional context
Looking at this podman bug:
https://github.com/containers/podman/issues/3990

... it seems podman won't help with getting permissions correctly set up for volume mounts.

My experiments showed that doing some podman unshare tricks prior the native image build might work. Something like this:

$ CONTAINER_UID=$(podman run --rm --entrypoint /bin/bash -ti quay.io/quarkus/ubi-quarkus-native-image:19.3.1-java11 -c 'id -u' | sed 's/\r//g')
$ podman unshare chown -R $CONTAINER_UID target/getting-started-1.0-SNAPSHOT-native-image-source-jar
$ podman run -v getting-started/target/getting-started-1.0-SNAPSHOT-native-image-source-jar:/project:z --env LANG=C --rm quay.io/quarkus/ubi-quarkus-native-image:19.3.1-java11 -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -J-Duser.language=en -J-Dfile.encoding=UTF-8 --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy\$BySpaceAndTime -H:+JNI -jar getting-started-1.0-SNAPSHOT-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:-IncludeAllTimeZones -H:EnableURLProtocols=http --no-server -H:-UseServiceLoaderFeature -H:+StackTrace getting-started-1.0-SNAPSHOT-runner

Getting the UID the container runs as that way is important as it might change.

[rmarting]

Same failure error with quay.io/quarkus/ubi-quarkus-native-image:20.1.0-java11 image and podman version 2.0.3.

[galderz]

Assigning this to myself.

[galderz]

I'm unassigning myself from this because I won't be able to work on in for the next week or so. I've tried to replicate it on macOS but that won't work because you can't run podman on top of docker. Also tried centos 7 server, but podman version there is only 1.x. Finally tried fedora 32 aarch64 but the quarkus images don't support that arch (see https://github.com/quarkusio/quarkus-images/issues/83). This should be better handled by someone with amd64 env with a more recent linux version.

[zakkak]

Unfortunately podman unshare chown -R $CONTAINER_UID path will leave you with a path that the host user won't be able to edit/delete and will need podman unshare rm -rf path to achieve it. So after the build is complete we need to chown the path back to the host user.

[zakkak]

Note that if the image is using the root user this issue doesn't appear:

$ mkdir project
$ podman run -it --rm -v $PWD/project:/project:z --entrypoint /bin/bash fedora:32 -c 'id; touch /project/lala'
uid=0(root) gid=0(root) groups=0(root)
$ ls -la project/lala 
-rw-r--r--. 1 zakkak zakkak 0 Jul 30 16:41 project/lala

vs

$ mkdir project
$ podman run -it --rm -v $PWD/project:/project:z --entrypoint=/bin/bash quay.io/quarkus/ubi-quarkus-mandrel:20.1.0.1.Alpha2-java11 -c 'id; touch /project/lala'

uid=1001(quarkus) gid=1001(quarkus) groups=1001(quarkus)
touch: cannot touch '/project/lala': Permission denied
$ ls -la project/lala
ls: cannot access 'project/lala': No such file or directory

So an alternative would be to generate images that use the root user instead of the quarkus user

[zakkak]

@cescoffier would using the root user in the images cause any issues?

[gsmet]

@geoand do you know who could look at that one?

[geoand]

@gastaldi maybe? I have never used podman, otherwise I would take a look myself

[gsmet]

@Ladicek maybe?

[gastaldi]

I've played with Podman once, I can have a look if no one beats me to it

[geoand]

That's better than me, none and I don't think I can even install it :)

[gastaldi]

I wonder if https://github.com/tqvarnst/quarkus/commit/0b52b074b5ab95f264331dd60ab580d0d51182a6 actually fixes this. @tqvarnst can you confirm?

[zakkak]

I wonder if tqvarnst@0b52b07 actually fixes this. @tqvarnst can you confirm?

It looks like it applies @jerboaa 's suggestion, but it still needs to reset the ownership at the end (see https://github.com/quarkusio/quarkus/issues/10637#issuecomment-665648359)

[Ladicek]

Just like @geoand, I have exactly zero experience with Podman.

[zakkak]

I think I've got a solution (we essentially need to add -u=$(id -u):$(id -g)):

$ mkdir project
$ podman run -it --rm -v $PWD/project:/project:z --userns=keep-id -u=$(id -u):$(id -g) --entrypoint=/bin/bash quay.io/quarkus/ubi-quarkus-mandrel:20.1.0.1.Alpha2-java11 -c 'id; touch /project/lala'
uid=1000(1000) gid=1000 groups=1000
$ ls -la project/lala 
-rw-r--r--. 1 zakkak zakkak 0 Jul 30 17:34 project/lala
$ rm project/lala

[geoand]

@zakkak if that works, please open a PR and ask @rsvoboda to test

[gastaldi]

@zakkak is it OK if I assign this issue to you?

[zakkak]

@gastaldi Yes. I do have something working, just polishing it and I will soon open a PR.

[raffaelespazzoli]

hello, I'm having this issue with 1.8.3.

[zakkak]

@raffaelespazzoli can you please share info on how to reproduce it? Using code.quarkus.io/api.download and editing the pom.xml file to use 1.8.3 works as expected for me.

[raffaelespazzoli]

the link doesn't work can you give me the link of the app?
I was using the following tutorial:
http://web-m3-quarkus-bjrvk-guides.apps.cluster-952wq.952wq.sandbox763.opentlc.com/workshop/quarkus-lab/lab/springstrangle

but I am convinced it can be reproduced with any project. It seems to me
the issue depends on the combination of OS and podman.
I my case:
FC 33
podman version 2.1.1

and podman is aliased to docker.

On Fri, Nov 27, 2020 at 11:27 AM Foivos [e-mail] wrote:

@raffaelespazzoli https://github.com/raffaelespazzoli can you please
share info on how to reproduce it? Using code.quarkus.io/api.download and
editing the pom.xml file to use 1.8.3 works as expected for me.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/quarkusio/quarkus/issues/10637#issuecomment-734908624,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABPERXDOESYSKC5OODGQRELSR7HOZANCNFSM4OWOM6YQ
.

--
ciao/bye
Raffaele

[zakkak]

I did

curl -O -J  https://code.quarkus.io/api/download
unzip code-with-quarkus.zip
cd code-with-quarkus
sed -i 's/1.9.2.Final/1.8.3.Final/g' pom.xml
./mvnw package -Pnative -Dquarkus.native.container-runtime=podman -Dquarkus.native.container-build=true

$ ls target        
classes/                              code-with-quarkus-1.0.0-SNAPSHOT-native-image-source-jar/  generated-sources/  maven-status/
code-with-quarkus-1.0.0-SNAPSHOT.jar  code-with-quarkus-1.0.0-SNAPSHOT-runner*                   maven-archiver/     quarkus-app/
$ ./target/code-with-quarkus-1.0.0-SNAPSHOT-runner 
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2020-11-27 18:55:38,086 INFO  [io.quarkus] (main) code-with-quarkus 1.0.0-SNAPSHOT native (powered by Quarkus 1.8.3.Final) started in 0.012s. 
2020-11-27 18:55:38,087 INFO  [io.quarkus] (main) Profile prod activated. 
2020-11-27 18:55:38,087 INFO  [io.quarkus] (main) Installed features: [cdi]
hello commando
2020-11-27 18:55:38,087 INFO  [io.quarkus] (main) code-with-quarkus stopped in 0.000s

I am on Fedora 33 with podman 2.1.1 as well

[raffaelespazzoli]

that server has been retired, hashidoc of the tutorial can be found here:
https://github.com/RedHat-Middleware-Workshops/quarkus-workshop-m1m2/tree/rhtr2020/docs

On Fri, Nov 27, 2020 at 11:46 AM raffaele spazzoli <
[email protected]> wrote:

the link doesn't work can you give me the link of the app?
I was using the following tutorial:

http://web-m3-quarkus-bjrvk-guides.apps.cluster-952wq.952wq.sandbox763.opentlc.com/workshop/quarkus-lab/lab/springstrangle

but I am convinced it can be reproduced with any project. It seems to me
the issue depends on the combination of OS and podman.
I my case:
FC 33
podman version 2.1.1

and podman is aliased to docker.

On Fri, Nov 27, 2020 at 11:27 AM Foivos [e-mail] wrote:

@raffaelespazzoli https://github.com/raffaelespazzoli can you please
share info on how to reproduce it? Using code.quarkus.io/api.download
and editing the pom.xml file to use 1.8.3 works as expected for me.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/quarkusio/quarkus/issues/10637#issuecomment-734908624,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABPERXDOESYSKC5OODGQRELSR7HOZANCNFSM4OWOM6YQ
.

--
ciao/bye
Raffaele

--
ciao/bye
Raffaele

[raffaelespazzoli]

it worked with this option -Dquarkus.native.container-runtime=podman, perhaps it should be made more apparent in the docs.

[zakkak]

Hi @raffaelespazzoli, this is documented in https://github.com/quarkusio/quarkus/blob/master/docs/src/main/asciidoc/building-native-image.adoc#creating-a-linux-executable-without-graalvm-installed
Improvements (through PR) or suggestions are welcome :)

[gsmet]

Do we have a way to detect if docker is in fact podman and automatically enable the podman specific bits?

[geoand]

That's a good idea. I am sure it should be possible. @gastaldi any ideas? Perhaps just docker --version when backed by podman is enough?

[gastaldi]

Yeah docker --version should work AFAIK. I'm on PTO until Dec 2nd but I can provide a PR with this auto-detection algorithm after that

[zakkak]

While at it, could we make -Dquarkus.native.container-runtime=podman optional, i.e., only required when both docker and podman are available?

If docker is not present and podman is available Quarkus should automatically chose it and vice versa.
If both docker and podman are available, Quarkus should default to one of them and check if it's an alias to the other, to enable the specific bits.
If -Dquarkus.native.container-runtime=podman is used Quarkus should use podman and fail if podman is an alias to docker, similarly for -Dquarkus.native.container-runtime=docker, Quarkus should use docker and fail if docker is an alias to podman.

WDYT?

[gastaldi]

@zakkak sure, the auto-detection should kick in only if the -Dquarkus.native.container-runtime is not specified

[gastaldi]

I think we can even simplify the auto-detection algorithm by returning podman if podman --version returns something like podman version 2.1.1 and fallback to docker otherwise, because if you have a docker alias to podman, you'll certainly have the podman executable available