Trident: Specify custom images in TridentProvisioner

[haugom]

Describe the solution you'd like

If I try to use trident-operator:20.07.0 and submit a TridentProvisioner to kubernetes,
I would like to be able to specify custom images in the spec for

quay.io/k8scsi/csi-attacher:v2.2.0
quay.io/k8scsi/csi-provisioner:v1.6.0
quay.io/k8scsi/csi-resizer:v0.5.0"

Describe alternatives you've considered

If we want to use the operator as it is today we need to write a mutating webhook and replace the images on the fly before it is stored in kubernetes.

Alternatively we can't use the operator and have to install trident the old way.

Additional context

We have a different registry where the paths are not the same as they are in quay.io

Since it is already possible to specify images for trident and tridentAutosupport I would like to be able to include the three CSI-* images.

[mrlindblom]

As a workaround, we runs two passes with kustomize to do this.

First the normal kustomization.yaml included with trident.

kubectl kustomize ./trident/deploy/ > ~/workdir/trident-bundle.yml

Then we run it again and padded with our prefix for the image. Replace {{ trident.registry }} with you registry prefix

kubectl kustomize ~/workdir/ | kubectl apply -f -

our kustomization.yml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - trident-bundle.yml
  - tridentprovisioner.yml

images:
  - name: netapp/trident-operator
    newName: {{ trident.registry }}/netapp/trident-operator
    newTag: 20.04.0

tridentprovisioner.yml

apiVersion: trident.netapp.io/v1
kind: TridentProvisioner
metadata:
  name: trident
  namespace: trident
spec:
  imageRegistry: {{ trident.registry }}/quay.io
  tridentImage: {{ trident.registry }}/netapp/trident:20.04

[ntap-arorar]

@haugom Just want to make sure I understand your requirement correctly.

If your requirement is to use an alternate registry for the CSI images, then operator imageRegistry option lets you do that. So essentially instead of quay.io/k8scsi/csi-attacher:v2.2.0 CSI image will be pulled from <private-registry-FQDN>/k8scsi/csi-attacher:v2.2.0. This option is the equivalent of tridentctl install --image-registry <private-registry-FQDN>.

If you have a different path for the CSI images within your private registry e.g. <private-registry-FQDN>/<alt-k8scsi-path>/csi-attacher:v2.2.0 then the current operator cannot pull those CSI images and the alternative is to use tridentctl install --generate-custom-yaml and tridentctl install --use-custom-yaml option.

Let us know which one it is?

[haugom]

Hi @ntap-arorar

You are correct. It is the second option and that is why I am asking if this feature could be included in a future release.

We have been using trident and installing it ourselves for several versions but thought we would try the trident-operator, but since our CSI images are hosted, we can't simply change the quay.io registry to our internal registry.

We need to be able to specify the image image repository for all the images used.

for example, our quay.io/k8scsi/csi-attacher:v2.2.0 is <private-registry-FQDN>/<path1>/k8scsi_csi-attacher:v2.2.0

So our workaround is to not use the trident-operator as it is today in 20.07.0, but we would love to give it a try.

[ntap-arorar]

Hi @ntap-arorar

You are correct. It is the second option and that is why I am asking if this feature could be included in a future release.

We have been using trident and installing it ourselves for several versions but thought we would try the trident-operator, but since our CSI images are hosted, we can't simply change the quay.io registry to our internal registry.

We need to be able to specify the image image repository for all the images used.

for example, our quay.io/k8scsi/csi-attacher:v2.2.0 is <private-registry-FQDN>/<path1>/k8scsi_csi-attacher:v2.2.0

So our workaround is to not use the trident-operator as it is today in 20.07.0, but we would love to give it a try.

Thank you for confiming the requirements.

[rpmduarte]

This worked for 20.10.0 deployments as well

As a workaround, we runs two passes with kustomize to do this.

First the normal kustomization.yaml included with trident.

kubectl kustomize ./trident/deploy/ > ~/workdir/trident-bundle.yml

Then we run it again and padded with our prefix for the image. Replace {{ trident.registry }} with you registry prefix

kubectl kustomize ~/workdir/ | kubectl apply -f -

our kustomization.yml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - trident-bundle.yml
  - tridentprovisioner.yml

images:
  - name: netapp/trident-operator
    newName: {{ trident.registry }}/netapp/trident-operator
    newTag: 20.04.0

tridentprovisioner.yml

apiVersion: trident.netapp.io/v1
kind: TridentProvisioner
metadata:
  name: trident
  namespace: trident
spec:
  imageRegistry: {{ trident.registry }}/quay.io
  tridentImage: {{ trident.registry }}/netapp/trident:20.04

[ntap-arorar]

A recent commit (scheduled for 21.01.0) https://github.com/NetApp/trident/commit/892e75d8ce216c9d024ce47ca5876ad89c08d312 is a step in this direction. Using the imageRegistry option users will be able to replace the quay.io/k8scsi portion of the CSI-sidecars.

[roycec]

With 21.01.1 in my case the chaos continues, because it's only possible to specify a path to the trident image and a general private image path. But the images for e.g. csi-* and trident-autosupport are now in different locations/paths, so only one of the images could be found. Either there will be a solution for the operator install, or we still have to stick with the tridentctl installer.

[ntap-arorar]

@roycec

Please provide an absolute path to the autosupportImage as well when you are using the imageRegistry option.

e.g.

imageRegistry: localhost:5000/sig-storage
tridentImage: localhost:5000/netapp/trident:tag
autosupportImage: localhost:5000/netapp/trident-autosupport:tag

[roycec]

@ntap-arorar : You are correct, it's possible to specify a path for the autosupport image, so I withdraw from my previuos comment ;)

[Numblesix]

Hi,

i also would heavily vote for dedicated variables per Image :)

Currently i could specify tridentImage which is already good but sadly im not able to pull the OperatorImage :D

It would be great too have all Images as a Variable including the imageRegistry option to have the most flexibility as possible.

Keep up the great Work guys :) !

[ntap-arorar]

Hello @Numblesix,

Today operator does work with a custom image.

Since Trident operator runs as deployment and needs to be up and running before acting on the tprov/torc CR, it cannot be part of the CR. To specify a custom operator image please modify the operator's deployment section of the bundle.yaml (this section).

Hope it helps!

[Numblesix]

Hi @ntap-arorar ,

Thanks for that I'll try that tomorrow :)

Do you see any chance of publishing the image also on quay.io That would already fix 90% of my issue see also #467 :) ?

[ntap-arorar]

I would let @gnarl answer your last question.