Kaniko: Our images now fail to run with OCI error

Created on 4 Feb 2020 · 19Comments · Source: GoogleContainerTools/kaniko

Actual behavior
The new images return Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"/usr/bin/java\": stat /usr/bin/java: no such file or directory": unknown

Expected behavior
Previously it run without issue. No OCI runtime

To Reproduce

Build image with Kaniko
Try to run image docker run
Additional Information

FROM openjdk:8-jre-slim

# Expose ports to enable running the service
# Ports should be standardized to make it easier to debug
# Exposing two services in the same port can create conflicts

ENV PORT 8080
EXPOSE 8080

# List of ARGS input from Kaniko Build
ARG IMAGE_DATE
ARG VCS_REVISION
ARG VCS_SEMVER
ARG PKG_WORKDIR

# Labeling based on https://github.com/opencontainers/image-spec/blob/master/annotations.md
LABEL org.opencontainers.image.created="${IMAGE_DATE}"              \
      org.opencontainers.image.revision="${VCS_REVISION}"           \
      org.opencontainers.image.version="${VCS_SEMVER}"              \
      org.opencontainers.image.title="mytitle"                      \
      org.opencontainers.image.description="mydescription"          \
      org.opencontainers.image.authors="myauthors"                  \
      org.opencontainers.image.vendor="myvendor"                    \
      org.opencontainers.image.url="myurl"                          \
      org.opencontainers.image.documentation="mydocumentationlink"  \
      org.opencontainers.image.source="mygitrepourl"

# Copy of distribution/target folder artifacts
# In case additional Artifacts are required

# All containers should run in least privileged mode, meaning not ROOT.
# NOTE: On OpenShift there is a warning when you try to run as ROOT
RUN addgroup -g 1001 -S cc && \
    adduser -u 1001 -S -G cc cc && \
    chown -R 1001:0 /home/cc && \
    chmod -R g=u /home/cc

COPY --chown=1001:0 ${PKG_WORKDIR}/target/*.jar /home/cc/service.jar

USER 1001
# Command to initialize the service
CMD ["/usr/bin/java", "-jar", "home/cc/service.jar"]

arefilesystems in progress kinbug prioritp0 regression

Source

webmutation

👍1

All 19 comments

I have similar problems, but they already occur when performing a docker pull on a freshly build image with kaniko debug-v0.17.0.

The build seems to go fine without any errors, but when pulling the image, I get random errors related to missing files, what the error exactly details depends on the image.

For now, I have gone back to debug-v0.16.0 which builds the same images fine... No errors when pulling or running.

RemcodM on 4 Feb 2020

We get a different error and cannot build the image in Google Cloud Build error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/docker.sock: device or resource busy.

liemdo on 4 Feb 2020

👍1

@liemdo are you mounting docker.sock in your Kaniko build? Can you please specify your Dockerfile and kaniko command

tejal29 on 4 Feb 2020

@liemdo PR in progress. #1025

Patch fix coming soon.

tejal29 on 4 Feb 2020

👍1

Indeed fix for us was force 0.16 on pipeline. Only a dozen microservices affected.

webmutation on 4 Feb 2020

@webmutation Unfortunately i am not able to reproduce your error. Would it be possible for you to give us -v=trace logs?

tejal29 on 5 Feb 2020

We get a different error and cannot build the image in Google Cloud Build error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/docker.sock: device or resource busy.

@liemdo That error should be fixed by v0.17.1 release due to #1025

tejal29 on 5 Feb 2020

I have similar problems, but they already occur when performing a docker pull on a freshly build image with kaniko debug-v0.17.0.

The build seems to go fine without any errors, but when pulling the image, I get random errors related to missing files, what the error exactly details depends on the image.

For now, I have gone back to debug-v0.16.0 which builds the same images fine... No errors when pulling or running.

@RemcodM Sorry to hear about that. Were your issues related to files in /var/run Is yes we fixed that.
Please let me know if you are having other issues.

tejal29 on 5 Feb 2020

@tejal29 I will update the pipeline to 0.17.1 and see if the problem is related and solved by the patch.

EDIT: Same issue still, this only happens with FROM openjdk:8-jre-alpine not sure why. For the time being we are stuck on 0.16.

webmutation on 5 Feb 2020

I have similar problems, but they already occur when performing a docker pull on a freshly build image with kaniko debug-v0.17.0.
The build seems to go fine without any errors, but when pulling the image, I get random errors related to missing files, what the error exactly details depends on the image.
For now, I have gone back to debug-v0.16.0 which builds the same images fine... No errors when pulling or running.

@RemcodM Sorry to hear about that. Were your issues related to files in /var/run Is yes we fixed that.
Please let me know if you are having other issues.

https://github.com/GoogleContainerTools/kaniko/issues/1028 for this one I think

afirth on 5 Feb 2020

@afirth Indeed, #1028 seems like the problem I am experiencing with 0.17.0 (and thus also with 0.17.1). So if this issue is unrelated it can be closed.

RemcodM on 5 Feb 2020

Bad image creation with OCI error continues with v0.17.1.

webmutation on 6 Feb 2020

👍1

I suspect this is related to #1039

cvgw on 8 Feb 2020

We've committed a change which I believe will fix this. If anyone feels like testing tags a1af057f997316bfb1c4d2d82719d78481a02a79 and debug-a1af057f997316bfb1c4d2d82719d78481a02a79 have the new code

cvgw on 25 Feb 2020

experiencing the same

bitsofinfo on 4 Mar 2020

only reverting to 0.15.0 lets me actually get working images

bitsofinfo on 4 Mar 2020

@webmutation i verfied your Dockerfile on latest master build
I changed the base image to openjdk:8

FROM openjdk:8

# Expose ports to enable running the service
# Ports should be standardized to make it easier to debug
# Exposing two services in the same port can create conflicts

ENV PORT 8080
EXPOSE 8080

# List of ARGS input from Kaniko Build
ARG IMAGE_DATE
ARG VCS_REVISION
ARG VCS_SEMVER
ARG PKG_WORKDIR

# Labeling based on https://github.com/opencontainers/image-spec/blob/master/annotations.md
LABEL org.opencontainers.image.created="${IMAGE_DATE}"              \
      org.opencontainers.image.revision="${VCS_REVISION}"           \
      org.opencontainers.image.version="${VCS_SEMVER}"              \
      org.opencontainers.image.title="mytitle"                      \
      org.opencontainers.image.description="mydescription"          \
      org.opencontainers.image.authors="myauthors"                  \
      org.opencontainers.image.vendor="myvendor"                    \
      org.opencontainers.image.url="myurl"                          \
      org.opencontainers.image.documentation="mydocumentationlink"  \
      org.opencontainers.image.source="mygitrepourl"

# Copy of distribution/target folder artifacts
# In case additional Artifacts are required

# All containers should run in least privileged mode, meaning not ROOT.
# NOTE: On OpenShift there is a warning when you try to run as ROOT
RUN addgroup -gid 1001 -system cc && \
    adduser cc -u 1001 -system -gid 1001 && \
    chown -R 1001:0 /home/cc && \
    chmod -R g=u /home/cc

COPY --chown=1001:0 target/*.jar /home/cc/service.jar

USER 1001
# Command to initialize the service
CMD ["/usr/bin/java", "-jar", "home/cc/service.jar"]

I ran the following command

 docker run -v /usr/local/google/home/tejaldesai/.config/gcloud:/root/.config/gcloud -v /usr/local/google/home/tejaldesai/workspace/kaniko/integration:/workspace gcr.io/tejal-test/executor:debug -f dockerfiles/Dockerfile1 --context=dir://workspace --destination=gcr.io/tejal-test/test_10241
INFO[0014] Applying label org.opencontainers.image.created= 
INFO[0014] Applying label org.opencontainers.image.revision= 
INFO[0014] Applying label org.opencontainers.image.version= 
INFO[0014] Applying label org.opencontainers.image.title=mytitle 
INFO[0014] Applying label org.opencontainers.image.description=mydescription 
INFO[0014] Applying label org.opencontainers.image.authors=myauthors 
INFO[0014] Applying label org.opencontainers.image.vendor=myvendor 
INFO[0014] Applying label org.opencontainers.image.url=myurl 
INFO[0014] Applying label org.opencontainers.image.documentation=mydocumentationlink 
INFO[0014] Applying label org.opencontainers.image.source=mygitrepourl 
INFO[0014] RUN addgroup -gid 1001 -system cc &&     adduser cc -u 1001 -system -gid 1001 &&     chown -R 1001:0 /home/cc &&     chmod -R g=u /home/cc 
INFO[0014] cmd: /bin/sh                                 
INFO[0014] args: [-c addgroup -gid 1001 -system cc &&     adduser cc -u 1001 -system -gid 1001 &&     chown -R 1001:0 /home/cc &&     chmod -R g=u /home/cc] 
Adding group `cc' (GID 1001) ...
Done.
Adding system user `cc' (UID 1001) ...
Adding new user `cc' (UID 1001) with group `cc' ...
Creating home directory `/home/cc' ...
INFO[0014] Taking snapshot of full filesystem...        
INFO[0014] Resolving paths                              
INFO[0016] Resolving srcs [target/*.jar]...             
INFO[0016] COPY --chown=1001:0 target/*.jar /home/cc/service.jar 
INFO[0016] Resolving srcs [target/*.jar]...             
INFO[0016] Resolving paths                              
INFO[0016] Taking snapshot of files...                  
INFO[0016] USER 1001                                    
INFO[0016] cmd: USER                                    
INFO[0016] CMD ["/usr/bin/java", "-jar", "home/cc/service.jar"]

I docker ran the new image

tejaldesai@@kaniko (r-v0.18.0)$ docker run gcr.io/tejal-test/test_10241
no main manifest attribute, in home/cc/service.jar

and as expected it complains no main in manifest.

tejal29 on 6 Mar 2020

This is still happening even post 1.0, we've been stuck at 0.16 for while now

imranismail on 23 Oct 2020

We're seeing something similar trying to use kaniko to build docker images as part of a gitlab-runner pipeline:

root@dev:~# docker --version
Docker version 20.10.0, build 7287ab3

root@dev:~# gitlab-runner --version
Version:      12.9.0
Git revision: 4c96e5ad
Git branch:   12-9-stable
GO version:   go1.13.8
Built:        2020-03-20T13:01:56+0000
OS/Arch:      linux/amd64

This is a fragment from our gitlab-ci.yml:

build-dockerhub:
  stage: build
  image:
    # TODO: use latest instead of debug once we get to the bottom of issue using latest tag
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  variables:
    DOCKER_HUB_AUTH: $DOCKER_HUB_AUTH
  script:
    - if [ "$CI_COMMIT_REF_NAME" == "master" ]; then IMAGE_TAG="latest"; else IMAGE_TAG=$CI_COMMIT_REF_SLUG; fi
    - echo $CI_COMMIT_REF_NAME > $CI_PROJECT_DIR/src/CI_COMMIT_REF_NAME
    - echo $CI_COMMIT_SHA > $CI_PROJECT_DIR/src/CI_COMMIT_SHA
    - echo $IMAGE_TAG > $CI_PROJECT_DIR/src/IMAGE_TAG
    - echo "{\"auths\":{\"https://index.docker.io/v1/\":{\"auth\":\"$DOCKER_HUB_AUTH\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/docker/Dockerfile --destination flagsmith/flagsmith-api:$IMAGE_TAG

For some reason, using debug in place of latest fixes the issue.