Kaniko: Node.js binary not included in built image

[micthiesen]

Actual behavior
We are using kaniko to build an image within GitLab CI using GitLab runners. The image is built successfully and pushed to our registry. When attempting to use the image, running a command installed via npm, it fails with the following error: /usr/bin/env: ‘node’: No such file or directory

If we open a bash shell inside the image and look for the node or npm binaries, they are not found:

root@12fb111175f7:/# which node
root@12fb111175f7:/# npm
bash: npm: command not found

The binaries are not found in /usr/bin/ which is usually where they are installed.

Expected behavior
The node and npm binaries should be installed inside the image. When using Docker on our local machine they are.

To Reproduce
Steps to reproduce the behavior:

1. Use kaniko to build an image which installs Node.js (e.g. the one included below)
2. Look for the node and npm binaries inside the container. They will be missing

Additional Information

• Dockerfile
  Please provide either the Dockerfile you're trying to build or one that can reproduce this error:

FROM python:3.7.3

# Add an APT source for LTS Node.js. Now includes npm
# https://github.com/nodesource/distributions/blob/master/README.md#installation-instructions
RUN curl --silent --location https://deb.nodesource.com/setup_12.x | bash

RUN apt-get update --quiet --yes \
 && apt-get install \
    --no-install-recommends \
    --yes \
    libmemcached-dev \
    nodejs
RUN npm install --global snyk

• Build Context
  Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands): The Dockerfile only contains a single FROM and 3 RUN commands
• Kaniko Image (fully qualified with digest): sha256:260ca7bf7866d0d59c8ce83b66af231e888f44b00c7c738fe5dc9207524cc68c for gcr.io/kaniko-project/executor:debug

[micthiesen]

Enabling caching seems to bypass this bug. This is the command we now use:

/kaniko/executor
    --cache true
    --cache-ttl 12h
    --context ${CI_PROJECT_DIR}
    --dockerfile ${CI_PROJECT_DIR}/${DOCKERFILE_NAME}
    --destination ${DESTINATION}

[tejal29]

@micthiesen Thanks for providing the work-around. We will definitely investigate and fix this.

[micthiesen]

@tejal29 thanks. I’d like to add that I’ve had inconsistent results trying to reproduce this. I’m uncertain now whether or not it is the caching that fixed things. Good luck with the investigation

[rustymunkey]

We stumbled upon a similar issue when installing the "graphviz" debian package. The kaniko build succeeds, however in the resulting image several files are missing.

Our Dockerfile:

ARG CONFLUENCE_VERSION=6.15.9
FROM atlassian/confluence-server:${CONFLUENCE_VERSION}-ubuntu

RUN apt-get update -y \
    && DEBIAN_FRONTEND="noninteractive" apt-get install -y tzdata ttf-dejavu graphviz \
    && apt-get clean

The result is not deterministic. Sometimes newly installed libraries are missing, sometimes the "/usr/bin/dot" binary is not in the image.

We appended an ls command at the end of the RUN to see if the libraries were installed, and - yes - all required libs were at the correct place. Thus, the problem must be sometimes after the RUN commands were executed.

We are running kaniko in a docker container, as we intend to use it in our CI pipeline do build docker images without the need of accessing the docker server.

We tried several workarounds, including caching, --single-snapshot or --snapshotMode time.

[donmccasland]

Can one of you add a log of a debug run of kaniko that reproduces the issue?

[rustymunkey]

At least on our side it was the problem with our kaniko docker image. We took great care in creating it (yes, we have seen the warning on the readme!), as it would better fit to our build pipeline.
However, when we switched back to the official docker image, these weird non-deterministic problems disappeard and never came back.
After several weeks of testing and building various images with kaniko, I think the problem can be seen as solely from our custom docker image

[tejal29]

@micthiesen and @rustymunkey I think we might have identified the root cause for this behavior. The reason why this is intermittent is,

• apt get ends up running child process in other process groups.
• kaniko isnt aware of these and does not wait for them to complete at times.

We are working on investigating this further. I will update this ticket with progress. Hopefully you will be game for switching back to kaniko when this issue is fixed.

Thanks! And sorry for the inconvenience.
Tejal.

[drptbl]

@tejal29 just encountered this issue after installing yarn and building this docker container with kaniko. All files from yarn unpacked below are missing.

FROM node:10.19.0-buster
ENV YARN_VERSION 1.22.0
# install latest yarn
RUN curl -fSLO --compressed "https://yarnpkg.com/downloads/$YARN_VERSION/yarn-v$YARN_VERSION.tar.gz" \
    && tar -xzf yarn-v$YARN_VERSION.tar.gz -C /opt/ \
    && ln -snf /opt/yarn-v$YARN_VERSION/bin/yarn /usr/local/bin/yarn \
    && ln -snf /opt/yarn-v$YARN_VERSION/bin/yarnpkg /usr/local/bin/yarnpkg \
    && rm yarn-v$YARN_VERSION.tar.gz

@scone

[tejal29]

Thanks @drptbl . this is the next on our list. Will update the progress

[tejal29]

@drptbl I tried your dockerfile and am not able to reproduce this error.

tejaldesai@tejaldesai-macbookpro2 ~docker run -it  gcr.io/tejal-test/test /bin/sh
tejaldesai@tejaldesai-macbookpro2 ~docker run -it  gcr.io/tejal-test/test /bin/sh
# which npm 
/usr/local/bin/npm
# ls /usr/local/bin/yarnpkg
/usr/local/bin/yarnpkg
# ls -al opt/yarn-v1.22.0/lib/
total 5200
drwxr-xr-x 2 root root    4096 Feb  5 12:54 .
drwxr-xr-x 4 root root    4096 May  6 05:44 ..
-rwxr-xr-x 1 root root 5300709 Feb  5 12:54 cli.js
-rw-r--r-- 1 root root    9910 Feb  5 12:54 v8-compile-cache.js
# ls -al /opt/yarn-v1.22.0/bin/
total 28
drwxr-xr-x 2 root root 4096 Feb  5 12:54 .
drwxr-xr-x 4 root root 4096 May  6 05:44 ..
-rwxr-xr-x 1 root root 1025 Feb  5 12:54 yarn
-rwxr-xr-x 1 root root   34 Feb  5 12:54 yarn.cmd
-rwxr-xr-x 1 root root 1015 Feb  5 12:54 yarn.js
-rwxr-xr-x 1 root root   42 Feb  5 12:54 yarnpkg
-rwxr-xr-x 1 root root   30 Feb  5 12:54 yarnpkg.cmd
# 

But it could be due to its non-deterministic nature.