Charts: [stable/velero] Plugins for Velero missing after Upgrade to 1.2.0

[ZeroVEVO]

Describe the bug
After upgrading to Velero 1.2.0 the installer fails, asking for the Velero Plugins. The HELM Chart does not install the Veleo Plugin, because there is no value for it in the values file.

Version of Helm and Kubernetes:
Helm Version: 2.16.0
Kubernetes: v1.13.10

Which chart:
stable/velero

What happened:
After upgrading to Velero Version 1.2.0 the deployment of the pod fails with error:
"An error occurred: some backup storage locations are invalid: error getting backup store for location "default": unable to locate ObjectStore plugin named velero.io/azure"

What you expected to happen:
I would expect the Helm Chart to also install the new Velero Plugins by itself, according to which provider you set in the values file.

How to reproduce it (as minimally and precisely as possible):
I had Velero 1.1.0 installed and running and followed the instructions to upgrade the chart and the velero version in the Charts.yaml. I also checked the values.yaml file for differences.

Anything else we need to know:
Is it possible to tell the chart to add the plugins?

[ZeroVEVO]

I would expect the Helm Chart to install Velero like the CLI does:

velero install \
--provider azure \
*--plugins velero/velero-plugin-for-microsoft-azure:v1.0.0 *
[...]

[josefschabasser]

I think modifying the command to install the plugin and then run the server is the way to go.
Please keep in mind to make the plugin repository and tag configurable.

values:

  repository: velero/velero-plugin-for-microsoft-azure
  tag: v1.0.0

deployment:

command:
- /bin/sh
- -c
- /velero plugin add {{ .Values.plugin.repository }}:{{ .Values.plugin.tag }} && /velero

I think something like this should work.

PS: Of course a template to detect the provider and put everything together would be useful, too.

[ZeroVEVO]

Thanks for your answer,
I think, this could help, but i dont want to edit the deployment because im installing velero via the helm chart directly ("helm install velero -f values.yaml stable/velero").
First I thought of using the init Containers, but im afraid this is the wrong sequence of doing.
I also can´t send the "velero plugin add..." command to the Container with another Container, because the install of velero fails and wont listen for "velero commands".

It would be nice to know where the providers must be addressed inside the helm chart or where to definde the plugin if there already is the option to do so.

[josefschabasser]

Oh, I'm sorry, I should have added something like @maintainers.

[josefschabasser]

After looking at the deployment again, I think this would be more appropriate:

          command:
            - /bin/sh
            - -ec
            - |
              /velero plugin add {{ .Values.plugin.repository }}:{{ .Values.plugin.tag }}

              /velero server
              {{- with .Values.configuration }}
                {{- with .backupSyncPeriod }}
                  --backup-sync-period={{ . }}
                {{- end }}
                {{- with .resticTimeout }}
                  --restic-timeout={{ . }}
                {{- end }}
                {{- if .restoreOnlyMode }}
                  --restore-only
                {{- end }}
                {{- with .restoreResourcePriorities }}
                  --restore-resource-priorities={{ . }}
                {{- end }}
                {{- with .logLevel }}
                  --log-level={{ . }}
                {{- end }}
                {{- with .logFormat }}
                  --log-format={{ . }}
                {{- end }}
              {{- end }}

[ZeroVEVO]

I think i found what I was looking for, but have to verify that:
https://github.com/helm/charts/pull/18492

[ZeroVEVO]

After looking at the deployment again, I think this would be more appropriate:

          command:
            - /bin/sh
            - -ec
            - |
              /velero plugin add {{ .Values.plugin.repository }}:{{ .Values.plugin.tag }}

              /velero server
              {{- with .Values.configuration }}
                {{- with .backupSyncPeriod }}
                  --backup-sync-period={{ . }}
                {{- end }}
                {{- with .resticTimeout }}
                  --restic-timeout={{ . }}
                {{- end }}
                {{- if .restoreOnlyMode }}
                  --restore-only
                {{- end }}
                {{- with .restoreResourcePriorities }}
                  --restore-resource-priorities={{ . }}
                {{- end }}
                {{- with .logLevel }}
                  --log-level={{ . }}
                {{- end }}
                {{- with .logFormat }}
                  --log-format={{ . }}
                {{- end }}
              {{- end }}

Thanks again, im trying to get that running. MAybe this will be integrated soon?

[josefschabasser]

Oh it already is done through init containers: https://github.com/helm/charts/pull/18492/files/69570951594517b03ee62a9fc16e5e89e436d00d#r343845529 (scroll up a little bit)

[ZeroVEVO]

That was what I was looking for. I try to implement the Plugin as an Init-Container. So my first thought might have been right..

[josefschabasser]

I just did this:

helm upgrade velero stable/velero \
    --reuse-values \
    --set initContainers[0].name=velero-plugin-for-microsoft-azure \
    --set initContainers[0].image=velero/velero-plugin-for-microsoft-azure:v1.0.0 \
    --set initContainers[0].volumeMounts[0].mountPath=/target \
    --set initContainers[0].volumeMounts[0].name=plugins

It should work, but the SP secret expired and I can't renew it...

[ZeroVEVO]

This should work, too.
But I read all the PRs and the deployment.yaml and I cloud solve my issue by adding the following to the values.yaml:

initContainers:
  - name: velero-plugin-for-microsoft-azure
    image: velero/velero-plugin-for-microsoft-azure:v1.0.0
    volumeMounts:
      - name: plugins
        mountPath: /target

The values.yaml is already prepared for that, but its set to optional. In the PR https://github.com/helm/charts/pull/18492 the readme is requested to be changed. That would help users finding the spot where to add the plugins.

Thanks for your help!

[josefschabasser]

I had to remove and re-deploy velero, because it was having some weird issues. It started always with image tag v1.1.0 which already contained the plugins. The init container added a second plugin with the same name and the main container crashed because it had 2 azure plugins...

Renewed secret: ✔️
Upgrade: ❌
Deploy from scratch: ✔️