Velero: OpenShift/Velero/Restic - open /host_pods/: permission denied

[mircster]

Hello,

i have installed and tested Velero v1.2.0 on OpenShift on Azure with Azure Provider successful.
Now I uninstalled Velero and want to install Velero v1.2.0 with Restic.

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)

OpenShift Version:
oc v3.11.153
kubernetes v1.11.0+d4cacc0

./velero install \
    --provider azure \
    --plugins velero/velero-plugin-for-microsoft-azure:v1.0.0 \
    --bucket $BLOB_CONTAINER \
    --secret-file ./credentials-velero \
    --backup-location-config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID \
    --snapshot-location-config apiTimeout=5m \
    --use-restic 

I edited the image in the deployment and deamon set, as we are using a private repository.
Also i had to add proxy configuration to the environment variables of the deployment.

The installation works fine and the velero pod runs fine, but the restic pods run into CrashLoopBackOff with following error:

An error occurred: could not read pod volumes host path: open /host_pods/: permission denied

I followed the instructions on https://velero.io/docs/v1.2.0/restic/

oc adm policy add-scc-to-user privileged -z velero -n velero

Modify the DaemonSet yaml to request a privileged mode and mount the correct hostpath to pods volumes.

@@ -35,7 +35,7 @@ spec:
             secretName: cloud-credentials
         - name: host-pods
           hostPath:
-            path: /var/lib/kubelet/pods
+            path: /var/lib/origin/openshift.local.volumes/pods
         - name: scratch
           emptyDir: {}
       containers:
@@ -67,3 +67,5 @@ spec:
               value: /credentials/cloud
             - name: VELERO_SCRATCH_DIR
               value: /scratch
+          securityContext:
+            privileged: true

The first edit (path: /var/lib/origin/openshift.local.volumes/pods) in the deamon set yaml is no problem, but the second one (privileged: true) doesen´t work!
The + securityContext: is not correct for us as securityContext is already there plus runAsUser: 0

        securityContext:
          runAsUser: 0

But now the Problem starts!
If i no want to add privileged: true and reload the deamon set it is gone.
When i replace runAsUser: 0 with privileged: true it is gone.
It seems like it gets removed all the time.

What did you expect to happen:

I expect that i can set privileged: true in the deamon set and after this the pods are able to mount host_path.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero Attached
  velero_logs_gh-issue.txt

• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml N.A.

• velero backup logs <backupname> N.A.
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml N.A.
• velero restore logs <restorename> N.A.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

No, not yet :-) Thanks in advance!

Environment:

• Velero version (use velero version):
  Version: v1.2.0
  Git commit: 5d008491bbf681658d3e372da1a9d3a21ca4c03c

• Velero features (use velero client config get features):
  features:

• Kubernetes version (use kubectl version):
  OpenShift Version:
  oc v3.11.153
  kubernetes v1.11.0+d4cacc0

• Kubernetes installer & version:

• Cloud provider or hardware configuration:
  Azure

• OS (e.g. from /etc/os-release):
  NAME="Red Hat Enterprise Linux Server"
  VERSION="7.7 (Maipo)"

[mircster]

Hello,

The solution is "very easy" and I think the problem is not a problem for somebody who works with DIFF, Kubernetes/OpenShift and YAML files daily.
I didn´t understood that the securityContext needs to be added to the container configuration additional.
This is the reason why i always modified the already existing securityContext.
After adding it at the right place the pods are running fine :-)

So under spec: you need:

        securityContext:
          runAsUser: 0

And under env: you need:

        securityContext:
          privileged: true

I would recommend to discribe this in the procedure for OpenShift a little bit more detailed so people like me also understand it ;-)

A colleague also wants to add oc patch command to make it more easy in the future.
Comment will hopefully come within the next days.

[git001]

Here are the oc commands which should make the setup on oc easier.
We have setuped velero in the velero project.

securityContext

check if the securityContext is set.

oc -n velero get ds/restic -o jsonpath='{.spec.template.spec.containers[?(@.name == "restic")].securityContext.privileged}'

set the securityContext

The handicap is that we can't use the json selector from the command above therefore is the '0' hard coded here.

oc -n velero patch ds/restic --type json -p '[{"op":"add","path":"/spec/template/spec/containers/0/securityContext","value": { "privileged": true}}]'

hostpath

check which value the hostpath have

oc -n velero get ds/restic -o jsonpath='{.spec.template.spec.volumes[?(@.name == "host-pods")].hostPath.path}'

replace the hostpath

oc -n velero patch ds/restic --type json -p '[{"op":"replace","path":"/spec/template/spec/volumes/0/hostPath","value": { "path": "/var/lib/origin/openshift.local.volumes/pods"}}]'