Kustomize: Support changing of image registries in addition to image tags

[schweikert]

In our company we first put docker images in a test registry, then "promote" them to a production registry. We also often have the case that we deploy the same image to a cloud-hosted registry that is specific to a certain project.

For that, it would be nice to be able to customize the location of the image, and not only the tag. Mayb e something like:

imageRegistries:
- name: my/cool-app
  newRegistry: company-registry-prod.example.com

This would find and replace any image like:

company-registry-test.example.com/my/cool-app:latest

with

company-registry-prod.example.com/my/cool-app:latest

Let me know if you think this would be a good idea, and I can try to implement it.

[twz123]

I'd even suggest to also replace the image name, which could be merged with imageTags, like so:

```yaml
imageOverrides:

• name: my/cool-app
  newRegistry: company-registry-prod.example.com
  newName: my-personal/cool-app
  newTag: my-cool-image-tag

[Liujingfang1]

I agree with @twz123 that this should go into the same cusotmization fields as imageTags.
In imageTags, the name field is the full image name, including the registry. Kustomize will search all the containers and find the matched imagename and replace the tag. This searching/replacing can be extended to the registry and image name.

There is another part related to this, kustomize edit set imageTag. It accepts an image associated with the new tag. For registry and name, this won't work since the input and number of input will be different. We need to think about a way to extend it as well. It should be one subcommand with the ability to override different parts of the image.

[schweikert]

I like the suggestion too, thanks! I'll work on a pull request, focusing on the first part, and then implement as a separate pull request the kustomize edit part.

[narg95]

Hi,

is there any update on this? is there help needed? I am interested in this feature since we use different registries depending on the overlay (dev/stg/prod).

Thanks!

[Liujingfang1]

@narg95 We don't have cycle to work on this feature. Would you like to help?

[narg95]

@Liujingfang1 sure I can give it a try if you point me to the right direction :)

[narg95]

I like the idea of @twz123 with some subtle changes: using images instead of imageOverrides just be to consistent with imageTags, and not mentioning the newRegistry but only the name, e.g.:

Proposal

images:
- name: my/cool-app
  newName: my-registry/foo/cool-app
  newTag: my-cool-image-tag
- name: postgres
  newName: my-registry/my-postgres
- name: alpine
  digest: sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3

The following command produces that output :

kustomize edit set image \ 
   my/cool-app,my-registry/foo/cool-app:my-cool-image-tag \
   postgres,my-registry/my-postgres \
   alpine:sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3

Remarks

• Images is a superset of ImageTags
• imageTags will be marked as deprecated and internally imageTags is converted to images for backward compatibility and to keep only one code implementation
• It errors in case of colission between images and imageTags: e.g:

images:
- name: alpine
   newTag: latest
   newName: my-registry/alpine

# collision
imageTags:
- name: alpine
  newTag: 3

• newTag and digest behaves as the current imageTags
• newName overrides the image name that matches the given -name: <given-name>

[narg95]

Hi @Liujingfang1 I would love to help with this feature, but I would like to hear your opinion.

[duboisf]

Hi @narg95! I'm just a guy that's interested in this feature.

I like what you are proposing except for the part where if you don't specify newTag it removes the tag. I just wouldn't touch the tag if newTag isn't present. Otherwise for those that are interested in just modifying the registry it gives them extra work.

What's your reason for wanting to remove the tag if newTag isn't present?

[narg95]

Hi @duboisf,

what I had in mind at that time was to keep the behavior of docker, where an empty tag is valid and resolves to latest, e.g. docker pull postgres. But I agree with you, it is better to be explicit and more when deleting data. Also it is contrary to the behavior when imageName is not giving. Thanks for pointing it out, I will edit it, and if the latest tag is needed, it must be given as latest.

[Liujingfang1]

@narg95 Thank you for working on this and coming up with a concrete proposal. It looks good to me and I'd like add a few comments.

• Is tag allowed in NewName? If allowed, what will happen when NewTag is also present.
• The edit command looks not quite intuitive. I have to look at the arguments very carefully to figure out its meaning. What about some other options:
  kustomize edit set image --old <old_image_name> --new <new_image_name>
kustomize edit set image <old_image_name>=<new_image_name> <old_image_2>=<new_image_2>

[narg95]

Hi @Liujingfang1

Is tag allowed in NewName? If allowed, what will happen when NewTag is also present.

No, the tag can only be changed by the newTag.

The edit command looks not quite intuitive. I have to look at the arguments very carefully to figure out its meaning. What about some other options:

Agree, it could be a bit confussing, so with both options the examples look like:

-option 1

kustomize edit set image \ 
   --old my/cool-app --new my-registry/foo/cool-app:my-cool-image-tag \
   --old postgres --new my-registry/my-postgres \
   --new alpine:sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3

• option 2

kustomize edit set image \ 
   my/cool-app=my-registry/foo/cool-app:my-cool-image-tag \
   postgres=my-registry/my-postgres \
   alpine:sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3

• Option 1 reads clear and looks organized, but it uses two commnd flags for one image change (which is more implementation work) and the only way to correlate old with new image is by the order. Also it differs in how set imageTag works, see example alpine:sha256, so migration can be harder and error prone

• Option 2 it is less verbose, it is not as clear and organized as --old --new but the = sign is good enought to read and to relate old and new images, also it only requires one command argument and behaves the same as imageTags, so migration is trivial, it just requires to change imageTag with image

In my opinion if it were a new command without any backward compatibility I will choose first option, but because we want to replace and old command, keep backward compatibility and make the migration easy, option two seems more viable.

What do you think?

[Liujingfang1]

@narg95 Option 2 looks better than Option 1. I only have one concern about Option 2. In kubectl, there is a command
kubectl set image <container_name>=<container_image>
It we use Option 2, it will have similar format as this kubectl command, but the meaning is different.
kustomize set image <image_name>=<new_image_name>
container_name and image_name are different fields in container. One is name and the other is image. This might lead to some confusions. People might expect set image in kustomize work the same way as in kubectl.

[narg95]

hmm, I was not aware of the similarity, I see your point ... actualy they differ in the edit.
Still my preference is option 2 because it is simple, despite the similarities, also considering that the tools are used in different contexts for different purpuses and that the kubectl set image seems not to be so popular (no yet autocompletion for this command in the zsh plugin).

So, some options for option 2:

2.1. Still use option 2 as it is
2.2 Use double ::

kustomize edit set image \ 
   my/cool-app::my-registry/foo/cool-app:my-cool-image-tag \
   postgres::my-registry/my-postgres \
   alpine:sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3

2.3 Use double ==:

kustomize edit set image \ 
   my/cool-app==my-registry/foo/cool-app:my-cool-image-tag \
   postgres==my-registry/my-postgres \
   alpine:sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3

2.4. Use option 2 with a different command name, which?

My preferences are, from high to low: 2.1, 2.2, 2.3, 1

what do you think?

[Liujingfang1]

Among those options, I prefer 2.1.
Let's have some opinions from others. @schweikert @twz123 @duboisf What do you think?

[schweikert]

Thanks @narg95 for picking up the work on this.

I wonder if being able to modify multiple images with a single "kustomize edit set image" command should really be a requirement. If it isn't, then it might be better to use more explicit arguments (option 1 or similar), so that it is also more extensible. You could also support changing only the tag, for example, without the need to repeat the image name, something like:

kustomize edit set image nginx --new-tag 1.8.0
kustomize edit set image nginx --new-image my-nginx:1.8.1

[narg95]

Hi @schweikert thanks for sharing your opinion,

From your example I really like that the <image-name> is the main argument of the command, and it reads clear without ambiguity of what needs be changed.

Here I try to address your points:

I wonder if being able to modify multiple images with a single "kustomize edit set image" command should really be a requirement

there is not such strict requirement, but because our drivers have been:

1. stay consistent with other similar functionality like imagetag
2. replace imageTag with the new and more complete image feature
3. and make easy and intuitive, therefore attractive, the migration/adoption from imageTag to image

therefore modifing multiple images in a single run has been implicitly always there as a requirement to achieve points 1. and 3.

Also it performs better for multiple image modifications, because it reads and writes only once compared to n read/writes when using single commands. True, this difference is negligible but not for big ns.

You could also support changing only the tag, for example, without the need to repeat the image name

with option 2 you can also do it, this is how:

kustomize edit set image ngnix:1.8.0
kustomize edit set image ngnix=my-ngnix:1.8.1

# or with a single command

kustomize edit set image ngnix:1.8.0 ngnix=my-ngnix:1.8.1

What I like from option 2 is:

1. it is behaves exactly the same command as set imagtag <image-name>:<new-tag> and migrating is just as easy as removing tag from the command.
2. And prefix with <image-name>= adds the possibility to modify the image name

---

So the trade-off by and large IMHO is readability vs backwards compatibility.

I am fine with both options, but still inclined for option 2 :) !

so what is your preference @schweikert @Liujingfang1?

[Liujingfang1]

Option 2 supports editing multiple images by one command. We've already done that for imageTags.
I feel it's better than option 1. For readability, both 1 and 2 are easy to read and understand, but 2 with a compact format.

[schweikert]

Option 2 looks also fine to me

[narg95]

Great, then option 2.1 wins!
The PR #707 implements it.

@Liujingfang1 @schweikert @duboisf @twz123 Thanks for sharing your opinions.

[cmberryau]

I'm currently using this functionality to override container repositories and it is great.

I'm not sure if I am missing something; however it seems if I add a newName statement to a base kustomization.yaml, the expected newName changes are not applied to resources declared at the overlay kustomization.yaml level.

Currently I am getting around this by simply adding the newName statement for the same image in the overlay kustomization.yaml, but this seems superfluous.

Is this intended or is there a way to merge image changes?