Velero: unable to get all required environment variables for azure cluster

[ahmadhajali]

What steps did you take and what happened:

I'm still getting the following error:
time="2019-10-30T12:30:59Z" level=info msg="Checking that all backup storage locations are valid" logSource="pkg/cmd/server/server.go:412"
An error occurred: some backup storage locations are invalid: error getting backup store for location "default": rpc error: code = Unknown desc = unable to get all required environment variables: the following keys do not have values: AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_SUBSCRIPTION_ID

I'm using helm command to install velero on azure cluster:
helm install --name velero --namespace velero --set-file credentials.secretContents.cloud=./credentials-velero -f ${VALUES_YAML_FILE} ${VELERO_HELM_CHART};

The values yaml secret section looks like:
secretContents:
cloud: |
[default]
AZURE_SUBSCRIPTION_ID={{ subscriptionID }}
AZURE_TENANT_ID={{ tenantID }}
AZURE_CLIENT_ID={{ clientID }}
AZURE_CLIENT_SECRET={{ clientSecret }}

velero version is 1.1.0

the velero secret value is equal to:
cloud:
[default]
AZURE_SUBSCRIPTION_ID={{ subscriptionID }}
AZURE_TENANT_ID={{ tenantID }}
AZURE_CLIENT_ID={{ clientID }}
AZURE_CLIENT_SECRET={{ clientSecret }}

No AZURE_CREDENTIALS_FILE env found like AWS.

What did you expect to happen:
I'm expecting to install velero on azure cluster and it should be up & running

Environment:*

• Velero version (use velero version): 1.1.0
• Kubernetes version (use kubectl version): 1.15.4

[skriss]

@ahmadhajali couple things:

• you don't need the [default] entry on the first line - that's an AWS-specific thing. Please remove it.
• can you provide the YAML for the velero deployment? it's unclear if it's being mounted in properly or not

[ahmadhajali]

• [default] value - removed as per your request but the same error.
• I'm using helm to deploy velero so the YAML file is created by the helm and not provided by us. Here is the YAML values from velero deployment:
  {
  "kind": "Deployment",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
  "name": "velero",
  "namespace": "velero",
  "selfLink": "/apis/extensions/v1beta1/namespaces/velero/deployments/velero",
  "uid": "89b06ebc-19da-4ce1-b432-205c5ce4b98c",
  "resourceVersion": "88732",
  "generation": 3,
  "creationTimestamp": "2019-10-30T14:52:49Z",
  "labels": {
  "app.kubernetes.io/instance": "velero",
  "app.kubernetes.io/managed-by": "Tiller",
  "app.kubernetes.io/name": "velero",
  "helm.sh/chart": "velero-2.3.0"
  },
  "annotations": {
  "deployment.kubernetes.io/revision": "1"
  }
  },
  "spec": {
  "replicas": 1,
  "selector": {
  "matchLabels": {
  "app.kubernetes.io/instance": "velero",
  "app.kubernetes.io/name": "velero"
  }
  },
  "template": {
  "metadata": {
  "creationTimestamp": null,
  "labels": {
  "app.kubernetes.io/instance": "velero",
  "app.kubernetes.io/managed-by": "Tiller",
  "app.kubernetes.io/name": "velero",
  "helm.sh/chart": "velero-2.3.0"
  }
  },
  "spec": {
  "volumes": [
  {
  "name": "plugins",
  "emptyDir": {}
  },
  {
  "name": "scratch",
  "emptyDir": {}
  }
  ],
  "containers": [
  {
  "name": "velero",
  "image": "gcr.io/heptio-images/velero:v1.1.0",
  "command": [
  "/velero"
  ],
  "args": [
  "server"
  ],
  "envFrom": [
  {
  "secretRef": {
  "name": "velero"
  }
  }
  ],
  "env": [
  {
  "name": "VELERO_SCRATCH_DIR",
  "value": "/scratch"
  },
  {
  "name": "VELERO_NAMESPACE",
  "valueFrom": {
  "fieldRef": {
  "apiVersion": "v1",
  "fieldPath": "metadata.namespace"
  }
  }
  }
  ],
  "resources": {},
  "volumeMounts": [
  {
  "name": "plugins",
  "mountPath": "/plugins"
  }
  ],
  "terminationMessagePath": "/dev/termination-log",
  "terminationMessagePolicy": "File",
  "imagePullPolicy": "IfNotPresent"
  }
  ],
  "restartPolicy": "Always",
  "terminationGracePeriodSeconds": 30,
  "dnsPolicy": "ClusterFirst",
  "serviceAccountName": "velero-server",
  "serviceAccount": "velero-server",
  "securityContext": {},
  "schedulerName": "default-scheduler"
  }
  },
  "strategy": {
  "type": "RollingUpdate",
  "rollingUpdate": {
  "maxUnavailable": "25%",
  "maxSurge": "25%"
  }
  },
  "revisionHistoryLimit": 10,
  "progressDeadlineSeconds": 600
  },
  "status": {
  "observedGeneration": 3,
  "replicas": 1,
  "updatedReplicas": 1,
  "unavailableReplicas": 1,
  "conditions": [
  {
  "type": "Progressing",
  "status": "True",
  "lastUpdateTime": "2019-10-30T14:52:53Z",
  "lastTransitionTime": "2019-10-30T14:52:49Z",
  "reason": "NewReplicaSetAvailable",
  "message": "ReplicaSet \"velero-695f8d745b\" has successfully progressed."
  },
  {
  "type": "Available",
  "status": "False",
  "lastUpdateTime": "2019-10-30T15:59:39Z",
  "lastTransitionTime": "2019-10-30T15:59:39Z",
  "reason": "MinimumReplicasUnavailable",
  "message": "Deployment does not have minimum availability."
  }
  ]
  }
  }
  according to this documentation https://velero.io/docs/v1.1.0/debugging-install/ cloud-credentials secret should be exist but it is not installed by helm.
  I created manually, but did not manage to create a volume.
  In addition, I ran the following kubectl command before install velero via helm:
  kubectl create secret generic cloud-credentials --namespace velero --from-file cloud=credentials-velero

but with no success.

Any idea?

Best regards,
Ahmad

[skriss]

I see this:

"envFrom": [
{
"secretRef": {
"name": "velero"
}
}
],

which doesn't look right - the name of the secret should be cloud-credentials. I would try updating that and see if it fixes the issue.

If so - please file an issue in the Helm repo around this.

[ahmadhajali]

same issue even after replacing velero to cloud-credentials in secretRef. Since velero is exist as well in secerts (both velero and cloud-credentials are exist in secret section).

Here is the velero secret:

Details
Name:
velero
Namespace:
velero
Labels:
app.kubernetes.io/instance: velero
app.kubernetes.io/managed-by: Tiller
app.kubernetes.io/name: velero
helm.sh/chart: velero-2.3.0
Creation Time:
2019-10-30T14:52 UTC
Type:
Opaque
Data
cloud:
AZURE_SUBSCRIPTION_ID=*
AZURE_TENANT_ID=
AZURE_CLIENT_ID=
AZURE_CLIENT_SECRET=
AZURE_RESOURCE_GROUP=**

Here is cloud-credentials secret values:
Details
Name:
cloud-credentials
Namespace:
velero
Creation Time:
2019-10-30T15:46 UTC
Type:
Opaque
Data
cloud:
AZURE_SUBSCRIPTION_ID=*
AZURE_TENANT_ID=
AZURE_CLIENT_ID=
AZURE_CLIENT_SECRET=
AZURE_RESOURCE_GROUP=**

even both of them are exist, the pod failed to initialize due to missing env values for azure as mentioned above.

The pod env looks like:
Containers
velero
Image:
gcr.io/heptio-images/velero:v1.1.0
Environment variables:
VELERO_SCRATCH_DIR: /scratch
VELERO_NAMESPACE: (v1:metadata.namespace)
cloud (cloud-credentials): AZURE_SUBSCRIPTION_ID=*
AZURE_TENANT_ID=
AZURE_CLIENT_ID=
AZURE_CLIENT_SECRET=
AZURE_RESOURCE_GROUP=**
Commands:
/velero
Args:
server

[skriss]

ah, I think I see. There are two ways of specifying Azure credentials:

1. the old way - where the cloud-credentials secret contains multiple key-value pairs, each one being the name & value of an environment variable, and the secret is used in the velero deployment to set the envFrom field
2. the new wway - where the cloud-credentials secret contains a single key, cloud, whose value is a multi-line list of environment variables, and the secret is mounted into the velero deployment as a volume at /credentials, and the velero deployment has the AZURE_CREDENTIALS_FILE environment variable set to /credentials/cloud.

It looks like you're somewhere in between the two approaches right now, so I'd pick one (preference for #2 since it's the "new" way) and go with it. To get to #2, mount the secret into the velero deployment as a volume at /credentials, and add the AZURE_CREDENTIALS_FILE to the deployment spec as well, with a value of /credentials/cloud. It looks like the secret is already formatted correctly to support this approach.

[ahmadhajali]

As per your #2 explanation, it is supposed to be deployed correctly running helm install command?
our AWS installation goes throw the same process (using helm install) and the results are as per #2.

[skriss]

it looks like the Helm chart is still set up to use the "old" way which requires creating the secret differently. We should update the Helm chart -- please file an issue in that repo! Also, if you're interested in contributing, we'd love to have a patch for this

[ahmadhajali]

A great opportunity to contribute to helm repo. will give it a try.

[skriss]

@ahmadhajali I'm closing this out here since you have a PR open in the Helm repo -- thanks for submitting that change!

[vukasin-nikodijevic]

As documentation for 1.2.0 doesn't help, here is working example of secret yaml:

1. Format of file should be preserved as described on the azure plugin page:
   > https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure#create-service-principal

$ cat << EOF  > ./credentials-velero
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
AZURE_CLOUD_NAME=AzurePublicCloud
EOF

Note here:

Values of the AZURE_* should be plain text but not base64 encoded

1. Get the whole file base64 encoded:

$ echo ./credentials-velero | base64 | tr -d '\n'

1. secret.yaml data block:

apiVersion: v1
kind: Secret
data:
  cloud: <base64 string from the above>

I hope that this will help someone as I spent some time to figure out this.