Velero: facing issues while trying to install the ark helm chart

[Sam123ben]

What steps did you take and what happened:
I first used the ark-prereq.yaml file to create the k8s api resources on my cluster and then used the helm chart to create the ark-server. I am facing issues while trying to install the ark-server

I am getting the error while executing the helm chart at the beginning itself:

$ kubectl apply -f ansible/files/ark-prereq.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
customresourcedefinition "backups.ark.heptio.com" configured
customresourcedefinition "schedules.ark.heptio.com" configured
customresourcedefinition "restores.ark.heptio.com" configured
customresourcedefinition "downloadrequests.ark.heptio.com" configured
customresourcedefinition "deletebackuprequests.ark.heptio.com" configured
customresourcedefinition "podvolumebackups.ark.heptio.com" configured
customresourcedefinition "podvolumerestores.ark.heptio.com" configured
customresourcedefinition "resticrepositories.ark.heptio.com" configured
customresourcedefinition "backupstoragelocations.ark.heptio.com" created
customresourcedefinition "volumesnapshotlocations.ark.heptio.com" configured
namespace "backup" configured
serviceaccount "ark" unchanged
clusterrolebinding "ark" configured

$ helm upgrade --install ark-server --namespace backup ./helm-charts/ark-server -f $ENVIRONMENT_FOLDER/config/config.yml -f $ENVIRONMENT_FOLDER/config/secrets/secrets.yml
Release "ark-server" does not exist. Installing it now.
Error: object is being deleted: customresourcedefinitions.apiextensions.k8s.io "backups.ark.heptio.com" already exists

What did you expect to happen:
$✗ kubectl apply -f ansible/files/ark-prereq.yaml
customresourcedefinition "backups.ark.heptio.com" created
customresourcedefinition "schedules.ark.heptio.com" created
customresourcedefinition "restores.ark.heptio.com" created
customresourcedefinition "downloadrequests.ark.heptio.com" created
customresourcedefinition "deletebackuprequests.ark.heptio.com" created
customresourcedefinition "podvolumebackups.ark.heptio.com" created
customresourcedefinition "podvolumerestores.ark.heptio.com" created
customresourcedefinition "resticrepositories.ark.heptio.com" created
customresourcedefinition "backupstoragelocations.ark.heptio.com" created
customresourcedefinition "volumesnapshotlocations.ark.heptio.com" created
namespace "backup" created
serviceaccount "ark" created
clusterrolebinding "ark" created

$ helm upgrade --install ark-server --namespace backup ./helm-charts/ark-server -f $ENVIRONMENT_FOLDER/config/config.yml -f $ENVIRONMENT_FOLDER/config/secrets/secrets.yml
Release "ark-server" does not exist. Installing it now.
NAME: ark-server
LAST DEPLOYED: Thu Feb 7 10:27:53 2019
NAMESPACE: backup
STATUS: DEPLOYED

RESOURCES:
==> v1/Schedule
NAME AGE
ark-server-cluster-full-backup 2s

==> v1/VolumeSnapshotLocation
default 2s

==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
ark-server-65f8dcb49b-5z742 0/1 ContainerCreating 0 2s

==> v1/ServiceAccount
NAME SECRETS AGE
ark-server-server 1 2s

==> v1beta1/ClusterRoleBinding
NAME AGE
ark-server-server 2s

==> v1beta2/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
ark-server 1 1 1 0 2s

==> v1/BackupStorageLocation
NAME AGE
default 2s

NOTES:
Check that the ark is up and running:

Check that the secret has been created:

Once ark server is up and running you need the client before you can use it

1. wget https://github.com/heptio/ark/releases/download/v0.10.1/ark-v0.10.1-darwin-amd64.tar.gz
2. tar -xvf ark-v0.10.1-darwin-amd64.tar.gz -C ark-client

More info on the official site: https://github.com/heptio/ark#install-client

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/ark -n heptio-ark
• ark backup describe <backupname> or kubectl get backup/<backupname> -n heptio-ark -o yaml
• ark backup logs <backupname>
• ark restore describe <restorename> or kubectl get restore/<restorename> -n heptio-ark -o yaml
• ark restore logs <restorename>

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Velero version (use ark version): 0.10.1
• Kubernetes version (use kubectl version): v1.8.0+coreos.0 and on v1.10.11 version as well
• Kubernetes installer & version: v1.8.0+coreos.0
• Cloud provider or hardware configuration: aws
• OS (e.g. from /etc/os-release): coreos

[skriss]

@Sam123ben I believe the helm chart will install the CRDs for you - is there a reason you're manually applying the prereqs file before trying to install from helm?

[kivagant-ba]

I got the same issue when I was trying to reinstall the chart. I deleted all custom resource definitions created by Ark and it works after. I'm not sure why you are trying to apply ark-prereq.yaml before the Helm chart, so try to remove all created resources except namespace and cloud secret and install the chart again.

kubectl delete customresourcedefinitions backups.ark.heptio.com deletebackuprequests.ark.heptio.com downloadrequests.ark.heptio.com podvolumebackups.ark.heptio.com podvolumerestores.ark.heptio.com resticrepositories.ark.heptio.com restores.ark.heptio.com schedules.ark.heptio.com volumesnapshotlocations.ark.heptio.com backupstoragelocations.ark.heptio.com

[uschtwill]

I think there is sort of a timing problem with the helm chart, where the CRD backupstoragelocations.ark.heptio.com is not "ready" before helm tries to create an "instance" of it. Trying to run it usually errors out with:

Error: validation failed: unable to recognize "": no matches for kind "BackupStorageLocation" in version "ark.heptio.com/v1"

It only works very rarely and randomly. Why we think this is a timing issue is because we get the same behaviour, if we try to deploy ark without helm by simply running the following from a script:

kubectl apply -f config/common/00-prereqs.yaml
kubectl apply -f config/aws/05-ark-backupstoragelocation.yaml
kubectl apply -f config/aws/06-ark-volumesnapshotlocation.yaml
kubectl apply -f config/aws/10-deployment.yaml
kubectl apply -f config/aws/20-restic-daemonset.yaml

But if we run the prereqs first/separately and the other stuff with a couple of seconds of delay, it works. Running the prereqs manually and then using the helm charts doesn't work because of the behaviour @Sam123ben describes.

At least this is our observation and theory. If this can be reproduced, I can also open a separate issue here or in the helm chart's repository. Please advise.

[skriss]

@uschtwill this makes sense - 00-prereqs.yaml creates the CRD definitions, but it takes a little bit of time for the Kubernetes controllers to process them and make them available for use/creation.

I'd file an issue in the Helm chart repo.

[uschtwill]

Thanks for confirming, @skriss. I opened an issue over there.

[nrb]

An update - https://github.com/helm/charts/pull/12031 uses the crd-install Helm hook on the CRDs, which will prevent these timing issues.

[nrb]

https://github.com/helm/charts/pull/12031 is merged, which contains the fix for this. Closing out this issue.