Kaniko: Possibility to pull-tag-push without build

[daHaimi]

We are using docker in a gitlab CI/CD environment with unprivileged runners and successfully build docker images to be stored into gitlab.

Our build pipeline needs to use the previously built images from one registry, retag them for another registry and finally push them there for deployment.

As different stages use different environments, every deployment has its own registry and therefore its own build step.

So: Is there a possibility to run kaniko in a way, that no build is done, but a source image is pulled, retagged and pushed to another registry?

[tholok97]

We have a similar use-case. We build, test and push images in separate GitLab CI stages and pass the built image to later stages using GitLab artifiacts. Kaniko would allow us to do image building in unprivileged containers, but we'd love for it to be able to push as well. For us the built image is passed as a tarball, so ideally (in our case) Kaniko would take a tarball as input (the same way it can generate a tarball as output).

Pushing with Kaniko without building would be nice, and compliments building without pushing well, as we would be able to test the built image before pushing.

[HaveFun83]

This feature will be really helpful

[jonjohnsonjr]

FWIW, in google/go-containerregistry (a package that kaniko uses), I maintain a tool that does this (among other things) called crane.

We can copy an image from dockerhub to GCR with crane cp:

$ go get github.com/google/go-containerregistry/cmd/crane

$ crane cp ubuntu gcr.io/jonjohnson/ubuntu
2019/07/17 17:43:55 Pulling index.docker.io/library/ubuntu:latest
2019/07/17 17:43:55 No matching credentials were found, falling back on anonymous
2019/07/17 17:43:56 Pushing gcr.io/jonjohnson/ubuntu:latest
2019/07/17 17:43:58 existing blob: sha256:a31c3b1caad473a474d574283741f880e37c708cc06ee620d3e93fa602125ee0
2019/07/17 17:43:58 existing blob: sha256:b054a26005b7f3b032577f811421fab5ec3b42ce45a4012dfa00cf6ed6191b0f
2019/07/17 17:43:58 existing blob: sha256:5b7339215d1d5f8e68622d584a224f60339f5bef41dbd74330d081e912f0cddd
2019/07/17 17:43:58 existing blob: sha256:14ca88e9f6723ce82bc14b241cda8634f6d19677184691d086662641ab96fe68
2019/07/17 17:43:59 existing blob: sha256:4c108a37151f54439950335c409802e948883e00c93fdb751d206c9a9674c1f6
2019/07/17 17:43:59 gcr.io/jonjohnson/ubuntu@sha256:eb70667a801686f914408558660da753cde27192cd036148e58258819b927395: digest: sha256:eb70667a801686f914408558660da753cde27192cd036148e58258819b927395 size: 1152
2019/07/17 17:44:02 pushed blob: sha256:e37fc27e0a1c7033d1d6725b271b9fd6de1286028cab7c0e3bdbf4411aa8b1e7
2019/07/17 17:44:02 pushed blob: sha256:0af11110095b961ce4addaca6fbf8afe199d233b510f48cdd196b4c0b4029576
2019/07/17 17:44:03 pushed blob: sha256:69dd89700a6c256d7ddae21c28adf3ded21abb73838795b0a1a5c3b12556da0b
2019/07/17 17:44:03 pushed blob: sha256:d7984ec58db25dba3b1eb769446c440c608e2a35869a13b6a57ba069134f28e1
2019/07/17 17:44:06 pushed blob: sha256:890bdf70a444971b59240478f3073e23d165e3ba40106d3d4e94a57e52f9f715
2019/07/17 17:44:06 gcr.io/jonjohnson/ubuntu@sha256:09a066942d6dc6b5d9f88c707f1197748cb620bd0b322484ed327253cab2ac1a: digest: sha256:09a066942d6dc6b5d9f88c707f1197748cb620bd0b322484ed327253cab2ac1a size: 1152
2019/07/17 17:44:09 pushed blob: sha256:3c1de3d39dd66d71ef688133f7ffc8b4757ed9f8383adf7dc0c84b547ef7bd85
2019/07/17 17:44:09 pushed blob: sha256:c99ec04f469874c69bd01d057f54593e5346d90cfba712152008d76624217ddb
2019/07/17 17:44:09 pushed blob: sha256:dbc2e75663b5f54850089251d728dea5cb0b29b1e95e1bc0785c801bd2dc3092
2019/07/17 17:44:09 pushed blob: sha256:8627bf1c6512dd26e72f564465b94d24232e3221dc649211b6c2169cd9bae0f7
2019/07/17 17:44:12 pushed blob: sha256:85df13e07ac866f0749412605b4d04aef859d2d116979e1dad9da5093585ce3c
2019/07/17 17:44:13 gcr.io/jonjohnson/ubuntu@sha256:fcf80b2fae669da66e29b272d768451f7d2b1641c26a250b7adf83bf0dafb452: digest: sha256:fcf80b2fae669da66e29b272d768451f7d2b1641c26a250b7adf83bf0dafb452 size: 1152
2019/07/17 17:44:16 pushed blob: sha256:3f40a38fc4ff69c71a6542dfb6acbdf6e5e3be25f0f94659ee85f2e120e37d82
2019/07/17 17:44:16 pushed blob: sha256:cf5ff47c2b80aec3ebd29b66a23fe67c95ef99fa1e6371f7044638b61e3dac83
2019/07/17 17:44:16 pushed blob: sha256:fe0423e7d050e37d4651190300ea074bd25a8ac7977ab897205fd1df44889605
2019/07/17 17:44:16 pushed blob: sha256:d0d75e9a49d317af603cf3ed3fd0ac05c131074504b43af468073d51021f9bcd
2019/07/17 17:44:19 pushed blob: sha256:3245b5a2a588d6e4eef6ef141d6a38071aa4d1617100cc2feace80b721e75274
2019/07/17 17:44:20 gcr.io/jonjohnson/ubuntu@sha256:30fe694bccc16b8d61c21f92ebc24065ef3ff7e87a8891eb9a7d3657fe1f3d94: digest: sha256:30fe694bccc16b8d61c21f92ebc24065ef3ff7e87a8891eb9a7d3657fe1f3d94 size: 1152
2019/07/17 17:44:23 pushed blob: sha256:4dc4a19dc1271abc8daffce96513470c1cb3a909784bdc92f838bd4ffb6f92ac
2019/07/17 17:44:23 pushed blob: sha256:03367c790f847a4fb9da497ba924c531b08ed840ced5ddebd576b1e617fdeabc
2019/07/17 17:44:23 pushed blob: sha256:7a0dfc04432356cee71730dc1476c35948bcaa233371a0bee11cf8be333b28da
2019/07/17 17:44:23 pushed blob: sha256:5fb5943989879ea71f39058451b18a9cfe7c8d63d5e98bea0f8ce8398b662cfd
2019/07/17 17:44:26 pushed blob: sha256:ef5b5b197566d2f3081c54f8e5ce315ef1bb409a622708e21f26f8a316324dc9
2019/07/17 17:44:27 gcr.io/jonjohnson/ubuntu@sha256:b11b1467edefe08a02b4eb3799cb7df8f19a25f0b2da2ef837b993ad2c750c3c: digest: sha256:b11b1467edefe08a02b4eb3799cb7df8f19a25f0b2da2ef837b993ad2c750c3c size: 1152
2019/07/17 17:44:29 pushed blob: sha256:7ebc7802c8d928c19cc7c3ea9a2f1e3e582ae8100aa515c74ec5dbe432ab3ad0
2019/07/17 17:44:30 pushed blob: sha256:c9dae6e8c4d6bcdbd01ec8e9e91d209bc0d0fb0e392fdb4ef5075d1e322fd3ac
2019/07/17 17:44:30 pushed blob: sha256:756f46ae05c783e37b80e6d459322ed9643102a8da1ca620233c21a601cce8e4
2019/07/17 17:44:30 pushed blob: sha256:8641916778260cb4957d491680516450d36f7590247405e9657afbd7ab582a71
2019/07/17 17:44:32 pushed blob: sha256:3e0ab3b7390d02875ba6655704ba528f120405db6e19a6c8014bacd27a2a2b07
2019/07/17 17:44:33 gcr.io/jonjohnson/ubuntu@sha256:2250c49a818ed4162862b23d6c7e470e64d6a83d68a38615c1666126bd42993c: digest: sha256:2250c49a818ed4162862b23d6c7e470e64d6a83d68a38615c1666126bd42993c size: 1152
2019/07/17 17:44:34 gcr.io/jonjohnson/ubuntu:latest: digest: sha256:9b1702dcfe32c873a770a32cfd306dd7fc1c4fd134adfb783db68defc8894b3c size: 1418

The reason you see a bunch of images copied is that the ubuntu image is a manifest list. We can confirm this with crane manifest:

$ crane manifest ubuntu | jq .
2019/07/17 17:45:41 No matching credentials were found, falling back on anonymous
{
  "manifests": [
    {
      "digest": "sha256:eb70667a801686f914408558660da753cde27192cd036148e58258819b927395",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 1152
    },
    {
      "digest": "sha256:09a066942d6dc6b5d9f88c707f1197748cb620bd0b322484ed327253cab2ac1a",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      },
      "size": 1152
    },
    {
      "digest": "sha256:fcf80b2fae669da66e29b272d768451f7d2b1641c26a250b7adf83bf0dafb452",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm64",
        "os": "linux",
        "variant": "v8"
      },
      "size": 1152
    },
    {
      "digest": "sha256:30fe694bccc16b8d61c21f92ebc24065ef3ff7e87a8891eb9a7d3657fe1f3d94",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "386",
        "os": "linux"
      },
      "size": 1152
    },
    {
      "digest": "sha256:b11b1467edefe08a02b4eb3799cb7df8f19a25f0b2da2ef837b993ad2c750c3c",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "ppc64le",
        "os": "linux"
      },
      "size": 1152
    },
    {
      "digest": "sha256:2250c49a818ed4162862b23d6c7e470e64d6a83d68a38615c1666126bd42993c",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "s390x",
        "os": "linux"
      },
      "size": 1152
    }
  ],
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}

And we can confirm they are identical with crane digest:

$ crane digest ubuntu
sha256:9b1702dcfe32c873a770a32cfd306dd7fc1c4fd134adfb783db68defc8894b3c

$ crane digest gcr.io/jonjohnson/ubuntu
sha256:9b1702dcfe32c873a770a32cfd306dd7fc1c4fd134adfb783db68defc8894b3c

If I want to tag the linux image specifically, I can use crane cp again:

$ crane cp ubuntu@sha256:eb70667a801686f914408558660da753cde27192cd036148e58258819b927395 gcr.io/jonjohnson/ubuntu:linux
2019/07/17 17:50:55 Pulling index.docker.io/library/ubuntu@sha256:eb70667a801686f914408558660da753cde27192cd036148e58258819b927395
2019/07/17 17:50:55 No matching credentials were found, falling back on anonymous
2019/07/17 17:50:57 Pushing gcr.io/jonjohnson/ubuntu:linux
2019/07/17 17:50:57 existing blob: sha256:14ca88e9f6723ce82bc14b241cda8634f6d19677184691d086662641ab96fe68
2019/07/17 17:50:57 existing blob: sha256:5b7339215d1d5f8e68622d584a224f60339f5bef41dbd74330d081e912f0cddd
2019/07/17 17:50:57 existing blob: sha256:a31c3b1caad473a474d574283741f880e37c708cc06ee620d3e93fa602125ee0
2019/07/17 17:50:57 existing blob: sha256:b054a26005b7f3b032577f811421fab5ec3b42ce45a4012dfa00cf6ed6191b0f
2019/07/17 17:50:57 existing blob: sha256:4c108a37151f54439950335c409802e948883e00c93fdb751d206c9a9674c1f6
2019/07/17 17:50:58 gcr.io/jonjohnson/ubuntu:linux: digest: sha256:eb70667a801686f914408558660da753cde27192cd036148e58258819b927395 size: 1152

Hope that helps.

[qwinkler]

You can do something like this:

echo "FROM registry.a/project/name:tag-a" | \
/kaniko/executor --dockerfile /dev/stdin --destination registry.b/project/name:tag-b

[donmccasland]

Sounds like this functionality is already supported.

[JeremyCraigMartinez]

@donmccasland how is this already supported? Please provide links to the documentation explaining how this is supported

[daHaimi]

If the workaround by @smoulderme should be seen as "suported functionality" it should be in the official documentation for this use case.

[jeunii]

Any official documentation on this ? I have a Jenkins job that needs to clone images and I do not want to introduce dind containers just so that I can run docker pull/push.

We already use kaniko to build images and push them to our internal repo. Would be great of Kaniko did something like pull-tag-push.

[daHaimi]

As we have started using @smoulderme 's "piped" functionality and migrated now all projects with retagging needs, I can give a short explanation how we did it with gitlab-ci:

Step 1: Create "Origin"

We created out own build step and pushed it to "registry1/project:tag". This wasperformed by using the kaniko runner in the default configuration.

Step 2: Credential merging

Then, we needed to retag the image. So the first step was to add the seond registry's credentials to kaniko's auth JSON. This means, in this step, the auth JSON contains both, the source and target repository's credentials. This may only apply to you, if your source and target tag are on two different registries.

Step 3: Retag

The last change is the most simple. Change the --dockerfile param of the kaniko command to /dev/stdin. The result looks like

script:
  - echo "${DOCKER_AUTH_COMBINED}" > /kaniko/.docker/config.json
  - echo "FROM  ${TAG_ORIGIN}" | /kaniko/executor --context $CI_PROJECT_DIR --dockerfile /dev/stdin --destination ${TAG_TARGET}

[meglory]

i just verified @daHaimi 's solution，but i found that the image hash has changed，and i also get some error，like this:

Error while retrieving image from cache: getting file info: stat ... : no such file or directory

but it pushed succeed，kind of weird.

[fhoeben]

@daHaimi solution works great, but it is a real pity the hash changes. This is a real difference compared to the 'normal' pull/tag/push approach.

I tried using kaniko's cache, but that did help. (Actually I was quite surprised that did not work. As I would have expected a cache hit, even though the 'FROM' image name was different than one I used in a previous run, the source image hash was the same)

Anyone has any idea how to keep the original hash?

[fhoeben]

I found the solution for my GitLab pipeline to retag using the original hash. @jonjohnsonjr 's 'crane' tool works like a charm! Thanks!

retag_image:
  image:
    name: gcr.io/go-containerregistry/crane:debug
    entrypoint: [""]
  variables:
    GIT_STRATEGY: none
  script:
    - crane auth login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - crane cp ${IMAGE}:temp-${CI_PIPELINE_ID} ${IMAGE}:latest

[KyrietS]

To extend @fhoeben 's great answer: if you use kaniko with custom cretificate to tag your latest successfull build as _latest_, then you'll need to specify two more flags for crane.

tag_latest_build:
    stage: post_deploy
    only:
        - tags
    when: on_success
    image:
        name: gcr.io/go-containerregistry/crane:debug
        entrypoint: [""]
    variables:
        GIT_STRATEGY: none
        GODEBUG: x509ignoreCN=0
    script:
        - SSL_CERT_FILE=$REGISTRY_CERTIFICATE
        - crane auth login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
        - crane tag $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG latest

Where:

• GODEBUG suppresses an error: _certificate relies on legacy Common Name field_
• $REGISTRY_CERTIFICATE is your custom GitLab file-type variable. It means that this var holds path to a file with your certificate, not certificate itself, eg. /builds/project/your-repo.tmp/REGISTRY_CERTIFICATE
• Use crane tag instead of crane cp as mentioned above.

For some reason in GitLab CI you can't do:

variables:
    SSL_CERT_FILE: $REGISTRY_CERTIFICATE

Because GitLab will then put content of a file from path $REGISTRY_CERTIFICATE into env variable SSL_CERT_FILE instead of assigning a path to it.

[kumprj]

You can do something like this:

echo "FROM registry.a/project/name:tag-a" | \
/kaniko/executor --dockerfile /dev/stdin --destination registry.b/project/name:tag-b

I am currently doing this in our CI solution, but the downside is the SHA changes and therefore you can't validate its the same thing sent to each environment.

[jonjohnsonjr]

but the downside is the SHA changes and therefore you can't validate its the same thing sent to each environment

I would recommend using a separate tool for this, e.g. skopeo copy or crane copy.

[kumprj]

@jonjohnsonjr I'm experiencing issues trying to authenticate ECR with crane and skopeo. I'm thinking its an issue with how I'm configuring my local ~/.docker/config.json but I can't seem to get it right.

For example, I can log into jfrog artifactory (our docker image repository) with crane auth login jfrogurl.com -u username -p password and then pull down with crane pull docker-registry-url.com/image tarname

However, with ECR I can run echo "acctid.dkr.ecr.us-east-2.amazonaws.com" | crane auth get | jq .password | sed 's/"//g'. It retrieves the password successfully. I then copy and paste the password into a crane auth login, but the crane login followed by a pull from ECR does not work (401 unauthorized). If I take the same password that is echoed out and do a docker login to ECR, it works, so I know its not the password.

I also tried piping it out using --password-stdin instead of a manual copy/paste.

Any tips?

[jonjohnsonjr]

If I take the same password that is echoed out and do a docker login to ECR, it works, so I know its not the password.

This is really surprising to me! If you don't mind opening an issue against go-containerregistry, I can try to help debug further.

If you include your ~/.docker/config.json (please redact any strings that aren't obviously safe to share over the internet), that would be helpful as well.

[kumprj]

Steps to reproduce on macOS catalina bash. For what its worth I'm behind an enterprise proxy:

$ echo "acctid.dkr.ecr.us-east-2.amazonaws.com" | crane auth get | jq .password | sed 's/"//g'

long string password is output

$ crane auth login acctid.dkr.ecr.us-east-2.amazonaws.com -u AWS -p stringfromabove
$ crane pull acctid.dkr.ecr.us-east-2.amazonaws.com/reponame:tag tarball.tar

Error: 2020/12/07 13:27:15 GET https://acctid.dkr.ecr.us-east-2.amazonaws.com/v2/repo-name/manifests/tag-name: unexpected status code 401 Unauthorized: Not Authorized

$ docker login acctid.dkr.ecr.us-east-2.amazonaws.com -u AWS -p longstringfromabove
warning about --password via cli followed by Login Succeeded
docker pull succeeds

Config.json:

$ vim ~/.docker/config.json 
{
        "auths": {
                "acctid-that-I-want-to-pull-from.dkr.ecr.us-east-2.amazonaws.com": {},
                "different-actid.dkr.ecr.us-east-2.amazonaws.com": {},
                "jfrog.company.com": {},
                "jfrogdev.company.com": {}
        },
        "HttpHeaders": {
                "User-Agent": "Docker-Client/19.03.13 (darwin)"
        },
        "credsStore": "desktop",
        "experimental": "disabled",
        "stackOrchestrator": "swarm"
}

edit:
I did try adding a

"credHelpers": {
                "acct-id.dkr.ecr.us-east-1.amazonaws.com": "ecr-login"
        },

and setting the credsStore to ecr-login. But that didn't help my crane pull either.

[jonjohnsonjr]

"credsStore": "desktop",

So it looks like you're configured to use this global credsStore, docker-credential-desktop, which doesn't seem to be open source or have any docs (or my google-fu is failing me). Unfortunately, I don't have a macOS machine and haven't played with Docker Desktop for Mac enough to know what else might be going wrong. I _think_ docker and crane will always use this global credsStore if you have it configured.

I'd try just deleting that credsStore line (will probably break some things, but would help with diagnosing what's happening) and adding the credHelpers object:

"credHelpers": {
        "acct-id.dkr.ecr.us-east-1.amazonaws.com": "ecr-login"
},

If _that_ doesn't work, maybe docker-credential-ecr-login isn't configured correctly? You shouldn't need to log in at all if the ecr-login credential helper is configured.

[kumprj]

If _that_ doesn't work, maybe docker-credential-ecr-login isn't configured correctly? You shouldn't need to log in at all if the ecr-login credential helper is configured.

Yeah the latter is the behavior I expected too. If you see my edit, I did actually try that unfortunately with no avail.

[jonjohnsonjr]

I would guess there's some mismatch between your home directory's docker config file and the docker for desktop virtual machine configuration :/

For what its worth I'm behind an enterprise proxy

I missed this part. I'm not sure if that's what's causing the issue, but if you've done anything special to get docker for desktop working, you might need to do something equivalent on your host with HTTPS_PROXY etc?

Using crane pull -v to dump http requests can be helpful here, but I wouldn't know exactly what to look for.

I'd also be curious what the docker-credential-desktop binary is doing. On mac, I think dtruss -f docker login (or docker pull) and comparing to dtruss -f crane auth login (or crane pull) would be interesting, but again not sure what I'd be looking for, exactly.

[kumprj]

I'll play with it a bit more with this information and open an issue to get more eyes if needed. Thanks @jonjohnsonjr