Skaffold: Add support for unique-per-image tag (was DIGEST_HEX)

[elisiano]

In #1482 the tagging mechanism changed.

Expected behavior

with a tagging policy like this:

[...]
  tagPolicy:
    envTemplate:
      template: '{{.IMAGE_NAME}}:something-'
[...]

I was expecting to see the digest appended automatically to the name.

Actual behavior

The digest is not appended

Information

• Skaffold version: v0.29
• Operating system: macOS/Linux
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta10
kind: Config
build:
  artifacts:
  - image: quay.io/somecompany/someimage
    context: ./docker/path
    docker:
      dockerfile: Dockerfile
  tagPolicy:
    envTemplate:
      template: '{{.IMAGE_NAME}}:something-'
  local: {}
deploy:
  kustomize:
    path: somepath

Steps to reproduce the behavior

1. start minikube
2. make sure it's the default context
3. eval $(minikube docker-env)
4. skaffold build
5. docker images

[briandealwis]

For example, I applied the following change to examples/getting-started to use {{.DIGEST_HEX}}:

--- a/examples/getting-started/skaffold.yaml
+++ b/examples/getting-started/skaffold.yaml
@@ -1,6 +1,9 @@
 apiVersion: skaffold/v1beta10
 kind: Config
 build:
+  tagPolicy:
+    envTemplate: 
+      template: "{{.IMAGE_NAME}}:{{.USER}}{{.DIGEST_HEX}}"
   artifacts:
   - image: gcr.io/k8s-skaffold/skaffold-example
 deploy:

skaffold dev then produces a warning

Generating tags...
 - gcr.io/k8s-skaffold/skaffold-example -> WARN[0000] {{.DIGEST}}, {{.DIGEST_ALGO}} and {{.DIGEST_HEX}} are deprecated, image digest will now automatically be appended to image tags 
gcr.io/k8s-skaffold/skaffold-example:bsd_DEPRECATED_DIGEST_HEX_

But the images are tagged as my username and do not have a digest appended.

[nkubala]

this is the intended behavior of the tagger. the envTemplate tagger will generate a tag based exactly on what is provided in the skaffold.yaml: it won't append anything else on there without being explicitly specified by the user.

we compute the tags for image BEFORE we build them, so there's no image ID for us to use when generating the tag. we can't use the image digest for local images anyway, because we don't actually have the digest (this would require pushing the image, then retrieving the image manifest from the remote registry where we pushed it).

[dgageot]

Skaffold builds the image then tags it with {{.IMAGE_NAME}}:something- with, in your case resolves to quay.io/somecompany/someimage:something-.

If you run skaffold dev or skaffold run, it won't put this value in the yaml files though because that would be the same value no matter what the content of the image is -> Kubernetes would never update the running container.

So here's what Skaffold does. It depends wether images are pushed or not:

• If images are pushed (working with a remote k8s), then skaffold will push the quay.io/somecompany/someimage:something- tag and will use the fully qualified images name in the k8s manifests. Something like quay.io/somecompany/someimage:something-@sha2546:4ac032ecafcf2a6c3ed9f40cdd997028f42d1afb9e3eeda9c7345504308a7e1a
• If images are not pushed, there's no image digest so skaffold fallsback to tagging the image with it's local imageID and use that tag in the k8s manifests. So the same image locally has two tags. One that's mutable: quay.io/somecompany/someimage:something- and one that's immutable: quay.io/somecompany/someimage:IMAGE_ID

I wish it wouldn't be two different behaviours but since this is due to Docker's behaviour, there's little Skaffold can do.

[elisiano]

what I'm looking for is an alternative to the old {{DIGEST_HEX}} (available before pushing).
It would need to be something unique per image (dependent on the content). Doesn't matter if it's a real digest or not.

@dgageot how would I go about using the immutable tag in skaffold config? I tried {{.IMAGE_ID}} but it doesn't contain any value.

Error parsing reference: "quay.io/somecompany/someimage:something-<no value>" is not a valid repository/tag: invalid reference format

if interested, our use case is to have a common repository used as a playground for users to test their images before finalizing the projects and creating the repos.
As such we need a way to differentiate images with a custom prefix in the tag.
example:

• quay.io/somecompany/experimental:project1-$DIGEST
• quay.io/somecompany/experimental:project2-$DIGEST
• and so on

The old DIGEST vars would allow us to achieve the above. Now them being removed leaves us in a place where we cannot update skaffold.

Is there any alternative mechanism with which we can achieve a similar result as the one discussed above?

[dgageot]

Hi @elisiano, I wonder if there's any way you can test https://github.com/GoogleContainerTools/skaffold/pull/2772 and see if solves your issue.

[elisiano]

@dgageot if I read that correctly, I should use the {{.DIGEST}} var in the template?

[dgageot]

@elisiano yes. That's more a Fully Qualified Name than a digest. We kept the name for compatibility reasons but we might want to revisit that.

[morozov]

This is what I tried:

$ git diff
diff --git a/examples/bazel/skaffold.yaml b/examples/bazel/skaffold.yaml
index ef6b3b5b..3151111e 100644
--- a/examples/bazel/skaffold.yaml
+++ b/examples/bazel/skaffold.yaml
@@ -5,3 +5,6 @@ build:
   - image: gcr.io/k8s-skaffold/skaffold-bazel
     bazel:
       target: //:skaffold_example.tar
+  tagPolicy:
+    envTemplate:
+      template: '{{.IMAGE_NAME}}:{{.DIGEST}}'

$ ../../out/skaffold version
v0.37.0-82-g4899ae57

$ ../../out/skaffold build
Generating tags...
 - gcr.io/k8s-skaffold/skaffold-bazel -> WARN[0000] {{.DIGEST}}, {{.DIGEST_ALGO}} and {{.DIGEST_HEX}} are deprecated, image digest will now automatically be appended to image tags
gcr.io/k8s-skaffold/skaffold-bazel:_DEPRECATED_DIGEST_
Tags generated in 133.374µs
Checking cache...
 - gcr.io/k8s-skaffold/skaffold-bazel: Not found. Building
Cache check complete in 134.010476ms
Starting build...
Building [gcr.io/k8s-skaffold/skaffold-bazel]...
INFO: Analyzed target //:skaffold_example.tar (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:skaffold_example.tar up-to-date:
  bazel-bin/skaffold_example.tar
INFO: Elapsed time: 0.085s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
FATA[0000] build failed: build failed: building [gcr.io/k8s-skaffold/skaffold-bazel]: build artifact: writing image "gcr.io/k8s-skaffold/skaffold-bazel:_DEPRECATED_DIGEST_": UNAUTHORIZED: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

It seems like the digest is not replaced in the image name.

[dgageot]

Yeah, this whole envTemplate/DIGEST/DIGEST_ALGO/DIGEST_HEX thing is a mess.

envTemplate used to make sense and be useful. I'm not sure it's very useful now.

• If you remove the tagPolicy config, everything should work out of the box.
• One thing that envTemplate can do for you is to add a prefix/suffix to images. Fixed ones or from env variables. Something like:

tagPolicy:
    envTemplate:
        template: 'prefix-{{.IMAGE_NAME}}-{{.SUFFIX}}:{{.TAG}}'

The digest will then be automatically be appended without you having to use {{.DIGEST}}

[morozov]

If you remove the tagPolicy config, everything should work out of the box.

If I remove it entirely and use the same example as above, I get:

$ ../../out/skaffold build
Generating tags...
 - gcr.io/k8s-skaffold/skaffold-bazel -> gcr.io/k8s-skaffold/skaffold-bazel:v0.37.0-82-g4899ae57
Tags generated in 20.19983ms
Checking cache...
 - gcr.io/k8s-skaffold/skaffold-bazel: Not found. Building
Cache check complete in 125.299096ms
Starting build...
Building [gcr.io/k8s-skaffold/skaffold-bazel]...
INFO: Analyzed target //:skaffold_example.tar (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:skaffold_example.tar up-to-date:
  bazel-bin/skaffold_example.tar
INFO: Elapsed time: 0.143s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
FATA[0001] build failed: build failed: building [gcr.io/k8s-skaffold/skaffold-bazel]: build artifact: writing image "gcr.io/k8s-skaffold/skaffold-bazel:v0.37.0-82-g4899ae57": UNAUTHORIZED: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

The image gets tagged with git-sourced attributes which is not what I want. In the case when I build multiple images from the same repo, I want images re-tagged only if their content has been changed by a given git commit (hence, I'm using the sha256 tagger with skaffold v0.21).

One thing that envTemplate can do for you is to add a prefix/suffix to images. Fixed ones or from env variables. Something like:

tagPolicy:
    envTemplate:
        template: 'prefix-{{.IMAGE_NAME}}-{{.SUFFIX}}:{{.TAG}}'

Patching the same example with the above yields:

../../out/skaffold build
Generating tags...
 - gcr.io/k8s-skaffold/skaffold-bazel -> prefix-gcr.io/k8s-skaffold/skaffold-bazel-<no value>:<no value>
Tags generated in 76.244µs
Checking cache...
 - gcr.io/k8s-skaffold/skaffold-bazel: Not found. Building
Cache check complete in 62.95956ms
Starting build...
Building [gcr.io/k8s-skaffold/skaffold-bazel]...
INFO: Analyzed target //:skaffold_example.tar (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:skaffold_example.tar up-to-date:
  bazel-bin/skaffold_example.tar
INFO: Elapsed time: 0.150s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
FATA[0000] build failed: build failed: building [gcr.io/k8s-skaffold/skaffold-bazel]: build artifact: parsing tag "prefix-gcr.io/k8s-skaffold/skaffold-bazel-<no value>:<no value>": tag can only contain the runes `abcdefghijklmnopqrstuvwxyz0123456789_-.ABCDEFGHIJKLMNOPQRSTUVWXYZ`: <no value> 

Specifically, the {{.TAG}} attribute doesn't contain a value.

@dgageot could you try using examples/bazel/skaffold.yaml as the baseline and make a working example of what could be used as an alternative to the sha256 tagger?

[dgageot]

@morozov Sorry, I should have said that in my envTemplate usage example, SUFFIX and TAG are two env variables.

These 3 examples should work:

apiVersion: skaffold/v1beta13
kind: Config
build:
  tagPolicy:
    envTemplate:
      template: "{{.IMAGE}}"
  artifacts:
    - image: gcr.io/k8s-skaffold/skaffold-bazel
      bazel:
        target: //:skaffold_example.tar

apiVersion: skaffold/v1beta13
kind: Config
build:
  tagPolicy:
    gitCommit: {}
  artifacts:
    - image: gcr.io/k8s-skaffold/skaffold-bazel
      bazel:
        target: //:skaffold_example.tar

apiVersion: skaffold/v1beta13
kind: Config
build:
  tagPolicy:
    sha256: {}
  artifacts:
    - image: gcr.io/k8s-skaffold/skaffold-bazel
      bazel:
        target: //:skaffold_example.tar

[morozov]

Thank you @dgageot. Unfortunately, neither of the examples above works as I'd expect:

1. ``yaml apiVersion: skaffold/v1beta13 kind: Config build: tagPolicy: envTemplate: template: "{{.IMAGE}}" artifacts: <ul> <li>image: gcr.io/k8s-skaffold/skaffold-bazel<br /> bazel:<br /> target: //:skaffold_example.tar<br /> <code>md5-c64a6676194c373cbedf6c9bd768bdb0</code><br /> $ ../../out/skaffold build<br /> Generating tags...<br /> <ul><br /> <li>gcr.io/k8s-skaffold/skaffold-bazel -> <no value><br /><br /> Tags generated in 489.458µs<br /><br /> Checking cache...</li><br /> <li>gcr.io/k8s-skaffold/skaffold-bazel: Not found. Building<br /><br /> Cache check complete in 508.219369ms<br /><br /> Starting build...<br /><br /> Building [gcr.io/k8s-skaffold/skaffold-bazel]...<br /><br /> INFO: Analyzed target //:skaffold_example.tar (1 packages loaded, 2 targets configured).<br /><br /> INFO: Found 1 target...<br /><br /> Target //:skaffold_example.tar up-to-date:<br /><br /> bazel-bin/skaffold_example.tar<br /><br /> INFO: Elapsed time: 1.238s, Critical Path: 0.66s<br /><br /> INFO: 12 processes: 12 linux-sandbox.<br /><br /> INFO: Build completed successfully, 19 total actions<br /><br /> FATA[0002] build failed: build failed: building [gcr.io/k8s-skaffold/skaffold-bazel]: build artifact: parsing tag "<no value>": repository can only contain the runes <code>md5-340ff582d6ec1327cff9478e0463c32b</code>: <no value><br /><br /> <code>md5-1c60785977e2e59ad7b09a33ad2663e0</code><no value>. Probably some values/configuration are still missing.




2. ``yaml apiVersion: skaffold/v1beta13 kind: Config build: tagPolicy: gitCommit: {} artifacts: <ul> <li>image: gcr.io/k8s-skaffold/skaffold-bazel<br /> bazel:<br /> target: //:skaffold_example.tar<br /> <code>md5-c64a6676194c373cbedf6c9bd768bdb0</code><br /> $ ../../out/skaffold build<br /> Generating tags...<br /> <ul><br /> <li>gcr.io/k8s-skaffold/skaffold-bazel -> gcr.io/k8s-skaffold/skaffold-<a href="bazel:v0.37.0-82-g4899ae57-dirty">bazel:v0.37.0-82-g4899ae57-dirty</a><br /><br /> Tags generated in 27.503497ms<br /><br /> Checking cache...</li><br /> <li>gcr.io/k8s-skaffold/skaffold-bazel: Not found. Building<br /><br /> Cache check complete in 159.692667ms<br /><br /> Starting build...<br /><br /> Building [gcr.io/k8s-skaffold/skaffold-bazel]...<br /><br /> INFO: Analyzed target //:skaffold_example.tar (0 packages loaded, 0 targets configured).<br /><br /> INFO: Found 1 target...<br /><br /> Target //:skaffold_example.tar up-to-date:<br /><br /> bazel-bin/skaffold_example.tar<br /><br /> INFO: Elapsed time: 0.106s, Critical Path: 0.00s<br /><br /> INFO: 0 processes.<br /><br /> INFO: Build completed successfully, 1 total action<br /><br /> FATA[0004] build failed: build failed: building [gcr.io/k8s-skaffold/skaffold-bazel]: build artifact: writing image "gcr.io/k8s-skaffold/skaffold-bazel:v0.37.0-82-g4899ae57-dirty": UNAUTHORIZED: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication<br /><br /> <code>md5-daa9c79c340eff4e46eee0330ea9433d</code>v0.37.0-82-g4899ae57-dirty), not from the file contents.




3. ``yaml apiVersion: skaffold/v1beta13 kind: Config build: tagPolicy: sha256: {} artifacts: <ul> <li>image: gcr.io/k8s-skaffold/skaffold-bazel<br /> bazel:<br /> target: //:skaffold_example.tar<br /> <code>md5-c64a6676194c373cbedf6c9bd768bdb0</code><br /> $ ../../out/skaffold build<br /> Generating tags...<br /> <ul><br /> <li>gcr.io/k8s-skaffold/skaffold-bazel -> gcr.io/k8s-skaffold/skaffold-<a href="bazel:latest">bazel:latest</a><br /><br /> Tags generated in 55.731µs<br /><br /> Checking cache...</li><br /> <li>gcr.io/k8s-skaffold/skaffold-bazel: Not found. Building<br /><br /> Cache check complete in 61.56631ms<br /><br /> Starting build...<br /><br /> Building [gcr.io/k8s-skaffold/skaffold-bazel]...<br /><br /> INFO: Analyzed target //:skaffold_example.tar (0 packages loaded, 0 targets configured).<br /><br /> INFO: Found 1 target...<br /><br /> Target //:skaffold_example.tar up-to-date:<br /><br /> bazel-bin/skaffold_example.tar<br /><br /> INFO: Elapsed time: 0.079s, Critical Path: 0.00s<br /><br /> INFO: 0 processes.<br /><br /> INFO: Build completed successfully, 1 total action<br /><br /> FATA[0000] build failed: build failed: building [gcr.io/k8s-skaffold/skaffold-bazel]: build artifact: writing image "gcr.io/k8s-skaffold/skaffold-<a href="bazel:latest">bazel:latest</a>": UNAUTHORIZED: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: <a href="https://cloud.google.com/container-registry/docs/advanced-authentication">https://cloud.google.com/container-registry/docs/advanced-authentication</a><br /><br /> <code>md5-1c60785977e2e59ad7b09a33ad2663e0</code>latest.

[dgageot]

Sorry @morozov, I made a type. Instead of {{.IMAGE}}, it should have been {{.IMAGE_NAME}}

Now, the feature you are asking is not currently possible. There's no tagger that gives you the digests of the inputs. Since it was asked many times, I'll write a proposal.

[dgageot]

See https://github.com/GoogleContainerTools/skaffold/pull/2804

[elisiano]

@dgageot I had a look at the #2804 proposal and I think it's still missing what we're looking for.
The functionality you're describing for sha256 should also be somehow available for the envTemplate tagger for the reason specified in https://github.com/GoogleContainerTools/skaffold/issues/2162#issuecomment-504312366.

[dgageot]

@elisiano Thanks a lot for the feedback! I'll update the proposal.

[tejal29]

Waiting for the proposal to get in.

[douglascamata]

I was also surprisingly affected by this issue. Reading the official documentation about the tagging strategies gives the impression that sha256 will put a digest in the image tag and it doesn't. Can the official docs be update to avoid this confusion until the situation is fixed?

[Norbo11]

For those that are trying to consume the tag that skaffold generates on skaffold build, I am running it with skaffold build --file-output /dev/stdout which prints out the full tag name (including image digest) regardless of your tagPolicy.

[morozov]

@Norbo11 so what does the process look like? You build the image once to get the digest and then rebuild it with the tag?

[Norbo11]

@Norbo11 so what does the process look like? You build the image once to get the digest and then rebuild it with the tag?

Not sure what your specific use-case is, but for me, I just wanted to use Skaffold to build me an image and tag it with something that included the digest of the contents. I needed to retrieve that tag somehow to pass it to our deployment mechanism (we are not using Skaffold to deploy). Skaffold refused to output anything that included the digest.

If you pass --file-output some_file.json and then read the contents of that file after running the skaffold build command, it will contain the tag you need. I am not sure what running skaffold run or skaffold deploy would do after that, but it appears to me that Skaffold seems to build an image that is tagged with the contents digest by default without needing to specify the tagPolicy field.

[tstromberg]

Will bring this up for discussion again now that there is a proposal: https://github.com/GoogleContainerTools/skaffold/blob/master/docs/design_proposals/digest-tagger.md