Velero: Azure AKS velero backups are not showing in GKE Cluster

[ashoks27]

What steps did you take and what happened:

I am trying to migrate a Namespace with PV from Azure to GKE cluster. I have taken a backup from Azure and its completed. In GKE , I have added the Azure backup-location and update the configmap with the new storage class. But still the Azure backup is showing in GKE cluster.

$ velero backup-location create aks-ashok --provider azure --bucket velero-ashok --access-mode ReadOnly

ashok@Azure:~/velero-v1.1.0-linux-amd64$ ./velero get backup
NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR
aks-restic Completed 2019-10-22 12:52:01 +0000 UTC 29d default
backup1 Completed 2019-10-21 14:50:36 +0000 UTC 29d default
ashok@Azure:~/velero-v1.1.0-linux-amd64$

What did you expect to happen:

All backups taken from Azure AKS should be visible from GKE cluster.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

ashok@Azure:~/velero-v1.1.0-linux-amd64$ ./velero backup describe aks-restic --details
Name: aks-restic
Namespace: velero
Labels: velero.io/storage-location=default
Annotations:

Phase: Completed

Namespaces:
Included: vel-test-aks
Excluded:

Resources:
Included: *
Excluded:
Cluster-scoped: auto

Label selector:

Storage Location: default

Snapshot PVs: auto

TTL: 720h0m0s

Hooks:

Backup Format Version: 1

Started: 2019-10-22 12:52:01 +0000 UTC
Completed: 2019-10-22 12:52:10 +0000 UTC

Expiration: 2019-11-21 12:52:01 +0000 UTC

Resource List:
apps/v1/Deployment:
- vel-test-aks/mysql
- vel-test-aks/wordpress
apps/v1/ReplicaSet:
- vel-test-aks/mysql-5b648d7887
- vel-test-aks/wordpress-77bc8b5f57
v1/Endpoints:
- vel-test-aks/mysql
- vel-test-aks/wordpress
v1/Namespace:
- vel-test-aks
v1/PersistentVolume:
- pvc-b0762ee1-f414-11e9-9ae5-1e7a4a6457e9
- pvc-b772f764-f414-11e9-9ae5-1e7a4a6457e9
v1/PersistentVolumeClaim:
- vel-test-aks/mysqlashok-volumeclaim
- vel-test-aks/wordpress-volumeclaim
v1/Pod:
- vel-test-aks/mysql-5b648d7887-9mdrt
- vel-test-aks/wordpress-77bc8b5f57-44tcn
v1/Secret:
- vel-test-aks/default-token-r5n4b
- vel-test-aks/mysql
v1/Service:
- vel-test-aks/mysql
- vel-test-aks/wordpress
v1/ServiceAccount:
- vel-test-aks/default

Persistent Volumes:

Restic Backups:
Completed:
vel-test-aks/mysql-5b648d7887-9mdrt: mysql-persistent-storage
vel-test-aks/wordpress-77bc8b5f57-44tcn: wordpress-persistent-storage

---

FROM GKE cluster

./velero get backup-location
NAME PROVIDER BUCKET/PREFIX ACCESS MODE
aks-ashok azure velero-ashok ReadOnly
default gcp mybucket-sep ReadWrite

:~/velero-v1.1.0-linux-amd64 $ ./velero get backup-location aks-ashok -o yaml
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"velero.io/v1","kind":"BackupStorageLocation","metadata":{"annotations":{},"creationTimestamp":"2019-10-22T12:36:29Z","generation":1,"name":"aks-ashok","namespace":"velero","resourceVersion":"9255302","selfLink":"/apis/velero.io/v1/namespaces/velero/backupstoragelocations/aks-ashok","uid":"9244de12-f4c8-11e9-a480-42010a800125"},"spec":{"accessMode":"ReadOnly","config":{"resourceGroup":"Velero_Backups","storageAccount":"velerob65a9f6d0b0c"},"objectStorage":{"bucket":"velero-ashok","prefix":""},"provider":"azure"},"status":{"lastSyncedTime":null}}
creationTimestamp: 2019-10-22T12:36:29Z
generation: 2
name: aks-ashok
namespace: velero
resourceVersion: "9256155"
selfLink: /apis/velero.io/v1/namespaces/velero/backupstoragelocations/aks-ashok
uid: 9244de12-f4c8-11e9-a480-42010a800125
spec:
accessMode: ReadOnly
config:
resourceGroup: Velero_Backups
storageAccount: velerob65a9f6d0b0c
objectStorage:
bucket: velero-ashok
prefix: ""
provider: azure
status:
lastSyncedTime: null

---

cloudshell:~/velero-v1.1.0-linux-amd64 $ kubectl get cm -n velero -o yaml
apiVersion: v1
items:

• apiVersion: v1
  data:
  default: standard
  kind: ConfigMap
  metadata:
  annotations:
  kubectl.kubernetes.io/last-applied-configuration: |
  {"apiVersion":"v1","data":{"default":"standard"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"velero.io/change-storage-class":"RestoreItemAction","velero.io/plugin-config":""},"name":"change-storage-class-config","namespace":"velero"}}
  creationTimestamp: "2019-10-22T12:58:20Z"
  labels:
  velero.io/change-storage-class: RestoreItemAction
  velero.io/plugin-config: ""
  name: change-storage-class-config
  namespace: velero
  resourceVersion: "9260604"
  selfLink: /api/v1/namespaces/velero/configmaps/change-storage-class-config
  uid: 9fd4ed9f-f4cb-11e9-a480-42010a800125
  kind: List
  metadata:
  resourceVersion: ""
  selfLink: ""

@cloudshell:~/velero-v1.1.0-linux-amd64$ kubectl get sc
NAME PROVISIONER AGE
standard (default) kubernetes.io/gce-pd 29d

/velero get backup
NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Velero version (use velero version): 1.1
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration: AKS to GKE
• OS (e.g. from /etc/os-release):

[skriss]

Do you have a secret in your GKE cluster that contains your Azure credentials, and that's mounted into the Velero pods and has the $AZURE_CREDENTIALS_FILE variable holding the path of the secret file?

[ashoks27]

Hi Steve ,

No , I have just used the backup-location create option to configure the Azure backup-location. Could you please let me share me with any documentation on how to add to path of the secret file to the $AZURE_CREDENTIALS_FILE variable .

[skriss]

the steps are basically:

1. Create an Azure credentials file locally per https://velero.io/docs/v1.1.0/azure-config/#create-service-principal (you could also download this from your AKS cluster's secret)
2. Create a Kubernetes secret from it in the velero namespace -- something like kubectl -n velero create secret generic azure-cloud-credentials --from-file cloud=credentials-velero
3. Edit your velero deployment & daemonset to mount this secret as a volume - similarly to how the GKE secret is mounted, but give it a different mount path, e.g. /credentials-azure
4. Edit the velero deployment & daemonset YAML to add the AZURE_CREDENTIALS_FILE env var, similarly to how the GOOGLE_APPLICATION_CREDENTIALS env var is set up, except set the value to e.g. /credentials-azure/cloud if you use my sample commands from the previous steps.

[ashoks27]

Hi Steve,

Thanks for your input .. Now I am able to see the backups from Azure and able to restore the PVs to the GKE.

[harikabeeram]

Hi Ashok,

i am planning do POC on AKS. so i have my architecture. could yoou please help on how to take backup using velero and restore. please provide if u have any documents. and also cloud u please suggest which ingress controller is best weather application gateway or ngnix.