Velero: Velero pod not running in GCP-some backup storage locations are invalid

[MeghanaSrinath]

What steps did you take and what happened:
I have exported GOOGLE_APPLICATION_CREDENTIALS as path to my service account JSON file and have created the credentials-velero file using this key.
export GOOGLE_APPLICATION_CREDENTIALS="/home/ubuntu/creds.json"
$GOOGLE_APPLICATION_CREDENTIALS >> /home/ubuntu/credentials-velero

I installed velero using the below command:
velero install --provider gcp --bucket velero_bucket1 --secret-file /home/ubuntu/credentials-velero
But velero pod and the deployment is not running and has error status.

What did you expect to happen:
Velero deployment and pod should have been ready without error

The output of the following commands will help us better understand what's going on:

• kubectl logs deployment/velero -n velero
  File attached
  Velero_GCP.txt

Anything else you would like to add:
I also tried creating credentials-velero by the following command:
gcloud iam service-accounts keys create credentials-velero --iam-account $SERVICE_ACCOUNT_EMAIL
But I received the same error.

Environment:

• Velero version (use velero version): v1.1.0
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.7-gke.8", GitCommit:"7d3d6f113e933ed1b44b78dff4baf649258415e5", GitTreeState:"clean", BuildDate:"2019-06-19T16:37:16Z", GoVersion:"go1.11.5b4", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: GCP
• OS (e.g. from /etc/os-release): Ubuntu

[skriss]

Can you confirm that whichever file you passed as the value to --secret-file has a valid JSON service account key?

Can you also provide the output of velero backup-location get -o yaml?

[MeghanaSrinath]

Can you confirm that whichever file you passed as the value to --secret-file has a valid JSON service account key?

Result of kubectl get secret cloud-credentials -n velero -o jsonpath={.data.cloud} | base64 --decode (key values are masked):

[default]
gcp_credentials={
"type": "service_account",
"project_id": "acn-devopsgcp",
"private_key_id": "XXXX",
"private_key": "-----BEGIN PRIVATE KEY-----
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
-----END PRIVATE KEY-----
",
"client_email": "[email protected]",
"client_id": "XXXX",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "XXXX",
"client_x509_cert_url": "XXXX"

Can you also provide the output of velero backup-location get -o yaml?

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
creationTimestamp: 2019-09-24T06:34:11Z
generation: 1
labels:
component: velero
name: default
namespace: velero
resourceVersion: "20306"
selfLink: /apis/velero.io/v1/namespaces/velero/backupstoragelocations/default
uid: 518a99c8-de95-11e9-b514-42010a800275
spec:
config: {}
objectStorage:
bucket: velero_bucket1
prefix: ""
provider: gcp
status:
lastSyncedTime: null

[skriss]

The credentials file doesn't look like valid JSON, e.g. [default] right at the very top. You need a valid JSON credentials file per the process documented here: https://cloud.google.com/docs/authentication/getting-started

[skriss]

Closing this out as the root cause was identified, but feel free to reach out again as needed.