Velero: 2nd velero installation in non-default namespace does not work

[ghost74-tg]

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)
I have tried to install velero twice in my kubernetes cluster. The first velero installation was installed in the default namespace velero and the second in the namespace new-velero.
The default installation works fine. The velero pod is in running state and I could backup namespaces.
The velero pod of the second installation is in error state.

root@kubmaster1 ~]# kubectl get pods -n new-velero -o wide
NAME                      READY   STATUS   RESTARTS   AGE    IP               NODE         NOMINATED NODE   READINESS GATES
velero-7c6d9bb58d-h7brr   0/1     Error    4          114s   192.168.77.217   kubworker2   <none>           <none>

When checking for the logs I get the following:

[root@kubmaster1 ~]# kubectl logs deployment/velero -n new-velero
time="2020-08-18T13:02:12Z" level=info msg="setting log-level to INFO" logSource="pkg/cmd/server/server.go:177"
time="2020-08-18T13:02:12Z" level=info msg="Starting Velero server v1.4.2 (56a08a4d695d893f0863f697c2f926e27d70c0c5)" logSource="pkg/cmd/server/server.go:179"
time="2020-08-18T13:02:12Z" level=info msg="1 feature flags enabled []" logSource="pkg/cmd/server/server.go:181"
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-remap-version
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pv
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pv-from-pvc
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pvc-from-pod
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-pvc-node-selector
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-storage-class
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/cluster-role-bindings
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-preserve-fields
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/job
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/restic
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/role-bindings
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
time="2020-08-18T13:02:12Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
time="2020-08-18T13:02:12Z" level=info msg="Checking existence of namespace" logSource="pkg/cmd/server/server.go:361" namespace=new-velero
An error occurred: namespaces "new-velero" is forbidden: User "system:serviceaccount:new-velero:velero" cannot get resource "namespaces" in API group "" in the namespace "new-velero"

I assume the problem is that the clusterrolebinding for the second velero namespace is not added during install of the 2nd velero.

[root@kubmaster1 ~]# kubectl describe clusterrolebindings velero
Name:         velero
Labels:       component=velero
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind            Name    Namespace
  ----            ----    ---------
  ServiceAccount  velero  velero

After editing the clusterrolebinding for velero and adding the serviceaccoudn for namespace new-velero everything worked fine for both velero installations.

[root@kubmaster1 ~]# kubectl describe clusterrolebindings velero
Name:         velero
Labels:       component=velero
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind            Name    Namespace
  ----            ----    ---------
  ServiceAccount  velero  spp-velero
  ServiceAccount  velero  velero

What did you expect to happen:
The expectation is that multiple velero installations could be installed with the velero install command by specifying a dedicated workspace for the velero installation and that all velero installations could be used to backup,restore cluster,namespace resources,etc.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)
The logs for the deployment is already included in my description above.

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Velero version (use velero version):

[root@kubmaster1 ~]# velero version
Client:
    Version: v1.4.2
    Git commit: 56a08a4d695d893f0863f697c2f926e27d70c0c5
Server:
    Version: v1.4.2

• Velero features (use velero client config get features): features: <NOT SET>
• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.4", GitCommit:"224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba", GitTreeState:"clean", BuildDate:"2019-12-11T12:47:40Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.7", GitCommit:"be3d344ed06bff7a4fc60656200a93c74f31f9a4", GitTreeState:"clean", BuildDate:"2020-02-11T19:24:46Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
  Red Hat Enterprise Linux Server 7.8

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "I would like to see this bug fixed as soon as possible"
• :-1: for "There are more important bugs to focus on right now"

[dymurray]

I believe the issue here is that the first Velero installation installed a ClusterRoleBinding with a single name which maps to the velero Service Account in the original default namespace. When you installed Velero the second time the CRB already exists so it doesn't create a new one. You would need to create a new CRB that points to the SA in the new namespace.

[dymurray]

From velero install perspective the name of the CRB should likely take on the installation namespace as well.

[carlisia]

Thanks @dymurray, this sounds like the fix for this problem.

@ghost74-tg please reopen this issue if you still have problems with the install.

[ghost74-tg]

@carlisia yes making the installation namespace a part of the CRB could fix the problem. Nevertheless this needs to be implement in the velero install. Or the user must manually create an additional CRB witch points to the new SA. My work-around was to add the new SA to the existing CRB. Anyway I think the velero install should create the CRB correctly and it should not be a manual work-around, to get multiple velero installations working.

[carlisia]

@vmware-tanzu/velero-maintainers ^ - should we reopen this issue?

[nrb]

@carlisia Yes, we should. I don't believe @dymurray's comment was an actual fix, but rather that he was saying we should enhance velero install/velero config.

[alaypatel07]

Hey folks, I can take this issue, feel free to assign this to me!

[ssethuma1155]

what build will this fix be available in?