Velero: Not able to install and run the velero (Crashloop when running velero pod)

[Sam123ben]

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)
I went through the steps as maintained in the below url as I am using Azure resource:
https://velero.io/docs/v1.0.0/azure-config/

What did you expect to happen:
What I expected is that the pod under the velero namespace but its failing

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

kc get po -n velero                           
NAME                      READY   STATUS    RESTARTS   AGE
velero-5dd4bbdd9c-m4q2r   1/1     Running   1          16s

kc logs -f velero-5dd4bbdd9c-m4q2r -n velero
time="2019-07-13T10:41:51Z" level=info msg="setting log-level to INFO"
time="2019-07-13T10:41:51Z" level=info msg="Starting Velero server v1.0.0 (72f5cadc3a865019ab9dc043d4952c9bfd5f2ecb)" logSource="pkg/cmd/server/server.go:165"
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pv
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/serviceaccount
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/azure
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/gcp
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/azure
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/gcp
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/addPVCFromPod
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/addPVFromPVC
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/job
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/restic
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service
time="2019-07-13T10:41:51Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/serviceaccount
time="2019-07-13T10:41:51Z" level=info msg="Checking existence of namespace" logSource="pkg/cmd/server/server.go:355" namespace=velero
time="2019-07-13T10:41:51Z" level=info msg="Namespace exists" logSource="pkg/cmd/server/server.go:361" namespace=velero
time="2019-07-13T10:41:53Z" level=info msg="Checking existence of Velero custom resource definitions" logSource="pkg/cmd/server/server.go:390"
time="2019-07-13T10:41:53Z" level=info msg="All Velero custom resource definitions exist" logSource="pkg/cmd/server/server.go:424"
time="2019-07-13T10:41:53Z" level=info msg="Checking that all backup storage locations are valid" logSource="pkg/cmd/server/server.go:431"
An error occurred: some backup storage locations are invalid: error getting backup store for location "default": rpc error: code = Unknown desc = azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/d0227861-50d5-40f8-8ddb-187234167a26/resourceGroups/civ-authority-nonprod-rg-v1/providers/Microsoft.Storage/storageAccounts/backupcivnonprodv1/listKeys?api-version=2018-02-01: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret is provided.\r\nTrace ID: 789f6686-a8fd-4fbb-b78d-987cbb260e00\r\nCorrelation ID: f7627f8e-84e8-4ece-b36f-f46eafc1dd6c\r\nTimestamp: 2019-07-13 10:41:54Z","error_codes":[7000215],"timestamp":"2019-07-13 10:41:54Z","trace_id":"789f6686-a8fd-4fbb-b78d-987cbb260e00","correlation_id":"f7627f8e-84e8-4ece-b36f-f46eafc1dd6c"}
time="2019-07-13T10:41:54Z" level=error msg="reading plugin stderr" cmd=/velero error="read |0: file already closed" logSource="pkg/plugin/clientmgmt/logrus_adapter.go:89" pluginName=velero

• kubectl logs deployment/velero -n velero
time="2019-07-13T10:44:53Z" level=info msg="setting log-level to INFO" time="2019-07-13T10:44:53Z" level=info msg="Starting Velero server v1.0.0 (72f5cadc3a865019ab9dc043d4952c9bfd5f2ecb)" logSource="pkg/cmd/server/server.go:165" time="2019-07-13T10:44:53Z" level=error msg="reading plugin stderr" cmd=/velero error="read |0: file already closed" logSource="pkg/plugin/clientmgmt/logrus_adapter.go:89" pluginName=velero time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pv time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/serviceaccount time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/azure time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/gcp time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/azure time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/gcp time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/addPVCFromPod time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/addPVFromPVC time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/job time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/restic time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service time="2019-07-13T10:44:53Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/serviceaccount time="2019-07-13T10:44:53Z" level=info msg="Checking existence of namespace" logSource="pkg/cmd/server/server.go:355" namespace=velero time="2019-07-13T10:44:53Z" level=info msg="Namespace exists" logSource="pkg/cmd/server/server.go:361" namespace=velero time="2019-07-13T10:44:55Z" level=info msg="Checking existence of Velero custom resource definitions" logSource="pkg/cmd/server/server.go:390" time="2019-07-13T10:44:59Z" level=info msg="All Velero custom resource definitions exist" logSource="pkg/cmd/server/server.go:424" time="2019-07-13T10:44:59Z" level=info msg="Checking that all backup storage locations are valid" logSource="pkg/cmd/server/server.go:431" An error occurred: some backup storage locations are invalid: error getting backup store for location "default": rpc error: code = Unknown desc = azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/d0227861-50d5-40f8-8ddb-187234167a26/resourceGroups/civ-authority-nonprod-rg-v1/providers/Microsoft.Storage/storageAccounts/backupcivnonprodv1/listKeys?api-version=2018-02-01: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret is provided.\r\nTrace ID: 732175eb-36b5-4ab3-a0e5-54f27a511100\r\nCorrelation ID: 46b62921-0a81-494f-9c0d-a8bf84c14401\r\nTimestamp: 2019-07-13 10:44:59Z","error_codes":[7000215],"timestamp":"2019-07-13 10:44:59Z","trace_id":"732175eb-36b5-4ab3-a0e5-54f27a511100","correlation_id":"46b62921-0a81-494f-9c0d-a8bf84c14401"}

• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml

• velero backup logs <backupname>

• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml

• velero restore logs <restorename>

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Velero version (use velero version):
  v1.0.0

• Kubernetes version (use kubectl version):
  Client Version: v1.14.2
  Server Version: v1.13.5

• Kubernetes installer & version:

• Cloud provider or hardware configuration:
  Azure

• OS (e.g. from /etc/os-release):

[Sam123ben]

What i found is the below command:
as per my az command:

az ad sp create-for-rbac --name "velero-prd" --role "Contributor" --password $AZURE_CLIENT_SECRET
az: error: unrecognized arguments: --password c3j94ti6RHskKZpf39/bK9SgS6VM/h2huaWMmdKQQDE=

az --version
azure-cli 2.0.68

command-modules-nspkg 2.0.3
core 2.0.68
nspkg 3.0.4
telemetry 1.0.3

Extensions:
aks-preview 0.4.5

Python location '/usr/local/Cellar/azure-cli/2.0.68/libexec/bin/python'
Extensions directory '/Users/samyakrout/.azure/cliextensions'

Python (Darwin) 3.7.4 (default, Jul 9 2019, 18:13:23)
[Clang 10.0.1 (clang-1001.0.46.4)]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-dat

#

As you can see I dont have the --password option

az ad sp create-for-rbac -h                                                                      

Command
    az ad sp create-for-rbac : Create a service principal and configure its access to Azure
    resources.

Arguments
    --name -n         : A URI to use as the logic name. It doesn't need to exist. If not present,
                        CLI will generate one.
    --role            : Role of the service principal.  Default: Contributor.
    --scopes          : Space-separated list of scopes the service principal's role assignment
                        applies to. Defaults to the root of the current subscription.
    --sdk-auth        : Output result in compatible with Azure SDK auth file.  Allowed values:
                        false, true.
    --skip-assignment : Skip creating the default assignment, which allows the service principal to
                        access resources under the current subscription.  Allowed values: false,
                        true.

Credential Arguments
    --cert            : Certificate to use for credentials.
        When used with `--keyvault,` indicates the name of the cert to use or create. Otherwise,
        supply a PEM or DER formatted public certificate string. Use `@{path}` to load from a file.
        Do not include private key info.
    --create-cert     : Create a self-signed certificate to use for the credential.
        Use with `--keyvault` to create the certificate in Key Vault. Otherwise, a certificate will
        be created locally.
    --keyvault        : Name or ID of a KeyVault to use for creating or retrieving certificates.
    --years           : Number of years for which the credentials will be valid. Default: 1 year.

Global Arguments
    --debug           : Increase logging verbosity to show all debug logs.
    --help -h         : Show this help message and exit.
    --output -o       : Output format.  Allowed values: json, jsonc, none, table, tsv, yaml.
                        Default: json.
    --query           : JMESPath query string. See http://jmespath.org/ for more information and
                        examples.
    --subscription    : Name or ID of subscription. You can configure the default subscription using
                        `az account set -s NAME_OR_ID`.
    --verbose         : Increase logging verbosity. Use --debug for full debug logs.

Examples
    Create with a default role assignment.
        az ad sp create-for-rbac


    Create using a custom name, and with a default assignment.
        az ad sp create-for-rbac -n "MyApp"


    Create without a default assignment.
        az ad sp create-for-rbac --skip-assignment


    Create with customized contributor assignments.
        az ad sp create-for-rbac -n "MyApp" --role contributor \
            --scopes /subscriptions/{SubID}/resourceGroups/{ResourceGroup1} \
            /subscriptions/{SubID}/resourceGroups/{ResourceGroup2}


    Create using a self-signed certificte.
        az ad sp create-for-rbac --create-cert

    Create using a self-signed certificate, and store it within KeyVault.
        az ad sp create-for-rbac --keyvault MyVault --cert CertName --create-cert

    Create using existing certificate in KeyVault.
        az ad sp create-for-rbac --keyvault MyVault --cert CertName

[Sam123ben]

Also what I did is that I used the cert generation:

az ad sp create-for-rbac --name "velero-prd" --role "Contributor" --keyvault civ-nonprod-vault --cert CertName --create-cert

But now after this how should I proceed with?

As I cant use the password, is there a way how I can use the certificate in the secret.

cat << EOF  > ./credentials-velero
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
EOF

[Sam123ben]

@nrb Can anyone please look into this

[prydonius]

According to the azure-cli docs, the --password option was removed in 2.0.68 (https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli?view=azure-cli-latest), we should probably update the docs to mention this.

You should be able to run

AZURE_CLIENT_SECRET=`az ad sp create-for-rbac --name "velero" --role "Contributor" --query 'password' -o tsv`

which will have the CLI generate a password for you.

As far as I can tell, Velero doesn't currently support certificate-based authentication, are you able to use password authentication?

[Sam123ben]

Cool, thanks that worked, let me check again from the scratch and in that case if anyone please update the wiki/doc for the same.

[Sam123ben]

I still strongly feel if possible please ensure the certificate auth is enabled in the future, this will be right way to proceed further

[Sam123ben]

Still getting the same issue:

An error occurred: some backup storage locations are invalid: backup store for location "default" is invalid: rpc error: code = Unknown desc = storage: service returned error: StatusCode=404, ErrorCode=ContainerNotFound, ErrorMessage=The specified container does not exist. RequestId:2b0e785e-501e-0018-3d6e-3bd0f8000000 Time:2019-07-16T00:35:55.7806347Z, RequestInitiated=Tue, 16 Jul 2019 00:35:55 GMT, RequestId=2b0e785e-501e-0018-3d6e-3bd0f8000000, API Version=2016-05-31, QueryParameterName=, QueryParameterValue=

[Sam123ben]

Sorry my bad the wrong container name was maintained

[prydonius]

Reopening to track updating the documentation