Kaniko: Kaniko's --build-arg accepts only literal values

[anastasiosv]

Actual behavior
Using encrypted variables is working fine with cloud build image.
Docker also supports copying a value from the environment by just specifying NAME[1].
But if I switch to kaniko, it doesn't pass the secret values as a build argument. Kaniko works only when I pass constant value, eg NAME=value.

Expected behavior
Be able to use the same way as docker does, the environment variables on Kaniko

To Reproduce

command:
gcloud builds submit --config=cloudbuild-kaniko.yaml .

Dockerfile:
FROM busybox
ARG THE_SECRET
RUN echo "::${THE_SECRET}::"

cloudbuild-kaniko.yaml:
steps:

• id: bash
  name: 'gcr.io/cloud-builders/docker'
  entrypoint: "bash"
  args: ['-c', 'echo "$$THE_SECRET"']
  secretEnv: ['THE_SECRET']
• id: docker
  name: 'gcr.io/kaniko-project/executor:latest'
  args:
  
  
  
  • --destination=gcr.io/[project-id]/image
  
  
  • --build-arg=THE_SECRET=$$THE_SECRET
  
  
  • --build-arg=THE_SECRET=secrettttttt
    
    secretEnv: ['THE_SECRET']
    
    secrets:
  
  
• kmsKeyName: projects/[project-id]/locations/global/keyRings/tinkering/cryptoKeys/cloud-build-demo
  secretEnv:
  THE_SECRET: XXXXXX

---

[osnagovskyi]

Looks like we're facing same issue - kaniko executor does not expand environment variables passed through --build-arg.

[jasonjho]

Running into the same issue as well. @osnagovskyi @tassosv how have you worked around this?

[diegosucaria]

Same issue here, any workaround?

[etheleon]

- name: 'gcr.io/cloud-builders/docker'
  entrypoint: 'bash'
  args: [ 
          '-c',
          'docker build -t gcr.io/$PROJECT_ID/appengine/ts-cloudbuild-secrets-example:latest -f Dockerfile --build-arg REDIS_PASS=$$REDIS_PW .' 
        ]
  secretEnv: ['REDIS_PW']    

maybe something like this? using bash as the entrypoint and the -c flag

[diegosucaria]

I'm in fact running kaniko in docker in a bash script like this:
(using remote builder)

#!/bin/bash

set -eo pipefail

IMAGE_NAME=$2
GCR_PATH=$1
REPO=$3

echo $REPO .  <-- this works and echoes the data correctly.

docker run gcr.io/kaniko-project/executor:latest \
    --destination $GCR_PATH/$IMAGE_NAME
    --cache=true
    --build-arg=REPO=$REPO

So it is not related to cloud build...

[inhumantsar]

came across this thread while researching kaniko use on GitLab CI. was concerned this would impact me, but i'm unable to reproduce using the debug image.

i've posted what i'm using below. hopefully it helps someone

some job name:
    image:
      name: gcr.io/kaniko-project/executor:debug
      entrypoint: [""]
    variables:
        JAR_SUFFIX: $VERSION
        JKS_SECRET: $PROD_JKS_SECRET
        JKS_VERSION: prod
    ...
    script:
      ...
      - |
        /kaniko/executor \
          --context $CI_PROJECT_DIR \
          --dockerfile $CI_PROJECT_DIR/Dockerfile \
          --destination $DOCKER_BASE_URL/$DOCKER_GROUP/$DOCKER_NAME:latest \
          --destination $DOCKER_BASE_URL/$DOCKER_GROUP/$DOCKER_NAME:$VERSION \
          --build-arg JAR_SUFFIX=$JAR_SUFFIX \
          --build-arg JKS_VERSION=$JKS_VERSION \
          --build-arg JKS_SECRET=$JKS_SECRET

[ncri]

Same issue here, prevents us from using kaniko at the moment.

[marcosimioni]

Same here, only way I could find in order to pass secrets via environment variables is via sh on kaniko:debug for now. Here's how I'm doing it for example, both for using envvars for authenticating against gitlab for my context, but also as buildargs:

....
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    env:
      - name: GIT_USERNAME
        valueFrom:
          secretKeyRef:
            name: git
            key: username
      - name: GIT_TOKEN
        valueFrom:
          secretKeyRef:
            name: git
            key: password
    command: [
              "/busybox/sh",
              "-ec",
              "mkdir /workspace && cd /workspace && /kaniko/executor \
              --verbosity=debug \
              --dockerfile=Dockerfile \
              --context=git://oauth2:${GIT_TOKEN}@********************.git \
              --destination=index.docker.io/**********:latest \
              --build-arg=GIT_USERNAME=${GIT_USERNAME} \
              --build-arg=GIT_TOKEN=${GIT_TOKEN} \
              --build-arg=GIT_BRANCH=*******"
              ]
....

[Sexual]

Any update regarding this?

[stevenaldinger]

We're running into this issue as well, preventing us from using kaniko with cloudbuild securely without major changes to use google's secrets options or something which we don't want to do since we already have a secrets management system.

The tool itself is awesome, any updates on whether or not it's being worked would be great.

This is essentially what we need to work and its confirmed working with a hardcoded token.

steps:
- name: 'gcr.io/kaniko-project/executor:latest'
  env: ['GITHUB_OAUTH_TOKEN=token-here']
  args:
  - --dockerfile=Dockerfile.prod
  - --context=.
  - --build-arg=GITHUB_OAUTH_TOKEN
  - --destination=us.gcr.io/$PROJECT_ID/app:$COMMIT_SHA
  - --cache=true
  - --cache-ttl=24h
tags: ['docker_image']

[cvgw]

I not sure if I'm understanding this issue correctly but here's what I've found

test Dockerfile

FROM ubuntu:16.04
ARG TEST=undefined
RUN echo $TEST > log.txt
ENTRYPOINT ["echo"]

test script

#!/bin/bash
ISSUE_NUMBER=713
context=~/kaniko-dev/issues/${ISSUE_NUMBER}/context

docker run \
  -v "$HOME"/.config/gcloud:/root/.config/gcloud \
  -v "$context":/workspace \
  gcr.io/kaniko-project/executor:v0.16.0 \
        --context dir:///workspace/ \
        --dockerfile Dockerfile \
        --destination gcr.io/kaniko-dev/issues:${ISSUE_NUMBER}-kaniko \
        -v trace \
        --cache=false \
        --build-arg=TEST=$ISSUE_NUMBER

test command
docker run -it --entrypoint="" --rm gcr.io/kaniko-dev/issues:713-kaniko cat log.txt

output
713

To me, that seems like it is working.

docker build also supports this cool short reference form where if there is some ENV variable FOO you can do docker build ..... --build-arg FOO and it will essentially expand to FOO=$FOO. Kaniko does not support this at this time, but seems like something reasonable to consider.

In general; I think build arg expansion would have to be done by whatever shell is invoking docker run .... kaniko... or the ENV variable would have to be explicitly passed to the kaniko docker container and then referenced in the arguments.

[antechrestos]

@cvgw Build arg in kaniko is working as posix argument.

The fact is, docker provides argument expansion. When you specify an argument such as --build-arg MY_ARGUMENT, and if MY_ARGUMENTis a valued environment variable, then then the build argument will be valued to the value in environment.

That can be useful in the following context: consider you use kaniko in a ci context (such as gitlab) with a variable set in group/project, containing something like a ssh key as it is mentionned here, either you will have to escape invalid characters, or you could rely on a build argument expansion.

I can do a PR to implement it as I've started to do it to use tes it locally. See my commit

[cvgw]

@antechrestos I think adding the short form (argument expansion) that docker build has is a reasonable feature addition, but I think a separate ticket should be open for that.

[sourcec0de]

Confirmed working ❤️

--build-arg=NPM_TOKEN

[sisp]

Are you aware that the values of build args are stored in the Docker image history? Anyone with access to the Docker image can inspect the history and extract secrets passed as build args.

[ncri]

@sisp That is why secrets are encrypted, see original post.

[sisp]

@ncri Do you mean this one? https://github.com/GoogleContainerTools/kaniko/issues/713#issue-466867494 I don't see anything about encrypted secrets. How would encrypted secrets passed via build args work? Could you give an example?

[ncri]

@sisp yes, that one, you can see it here:

kmsKeyName: projects/[project-id]/locations/global/keyRings/tinkering/cryptoKeys/cloud-build-demo
secretEnv:
THE_SECRET: XXXXXX

So the secret is not passed directly, but encrypted using googles Cloud Key Management Service. The XXXXXX hold the encrypted secret value. You can find info here: https://cloud.google.com/cloud-build/docs/securing-builds/use-encrypted-secrets-credentials

[sisp]

@ncri I still don't see how the secret provided to the Docker/Kaniko build process as a build arg should not end up in the Docker image history.

To clarify what I mean, let's consider the following example:

1. A simple Dockerfile:

   FROM alpine:latest
   ARG TOKEN
   RUN echo $TOKEN
   

2. Build the Docker image (not using Kaniko here, but the result should be the same with Kaniko):

   $ docker build --build-arg TOKEN=t0k3n -t test:latest .
   

3. Check the Docker image history:

   $ docker history test:latest
   IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
   0651b2a0e146        4 seconds ago       |1 TOKEN=t0k3n /bin/sh -c echo $TOKEN           0B                  
   57e6a9f30499        6 seconds ago       /bin/sh -c #(nop)  ARG TOKEN                    0B                  
   a24bb4013296        2 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B                  
   <missing>           2 weeks ago         /bin/sh -c #(nop) ADD file:c92c248239f8c7b9b…   5.57MB
   

   Notice the TOKEN=t0k3n statement in the top-most Docker image layer. I don't see how Google Cloud Build can fix this problem because the problem is that the values of Docker build args become part of the Docker image history.

Or am I missing something?

[ncri]

@sisp you might be right, yes, the docker history image will show the decrypted secret. So you still need to take care that you limit access to the images, that are usually placed on the google container registry. The purpose of encrypting the secrets is only to prevent having to check in sensitive data to your code repository. Anyway, might not be the right place to discuss this here. ;-)

[sisp]

@ncri Okay, thanks for clarifying this. I'll open a separate issue about this problem.

[sisp]

I just checked the Docker image history of an image built with Kaniko and it seems Kaniko does _not_ add the build arg value to the history. I wonder whether this is intended. I'll open an issue about this difference to docker build instead (https://github.com/GoogleContainerTools/kaniko/issues/1327).

[Ahmed3laa]

this worked for me

cloudbuild.yaml

steps:
name: 'gcr.io/kaniko-project/executor:latest'
env: ['ENV=$_ENV']
args:
--destination=gcr.io/$PROJECT_ID/frontend:$COMMIT_SHA
--cache=true
--cache-ttl=24h
--build-arg=ENV

and in docker

ARG ENV 
RUN npm run prerender:${ENV}