Trident: Trident cannot deploy with "err: could not create the Trident service account”

Created on 14 Dec 2020 · 11Comments · Source: NetApp/trident

Describe the bug
When I try to install Trident Operator,
Trident cannot deploy with "err: could not create the Trident service account"

$ kubectl describe tprov trident -n trident
Name:         trident
Namespace:    trident
Labels:       <none>
Annotations:  <none>
API Version:  trident.netapp.io/v1
Kind:         TridentProvisioner
Metadata:
  Creation Timestamp:  2020-12-14T07:57:12Z
  Generation:          1
  Managed Fields:
    API Version:  trident.netapp.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:debug:
    Manager:      kubectl-create
    Operation:    Update
    Time:         2020-12-14T07:57:12Z
    API Version:  trident.netapp.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:currentInstallationParams:
          .:
          f:IPv6:
          f:autosupportHostname:
          f:autosupportImage:
          f:autosupportProxy:
          f:autosupportSerialNumber:
          f:debug:
          f:enableNodePrep:
          f:imagePullSecrets:
          f:imageRegistry:
          f:k8sTimeout:
          f:kubeletDir:
          f:logFormat:
          f:silenceAutosupport:
          f:tridentImage:
        f:message:
        f:status:
        f:version:
    Manager:         trident-operator
    Operation:       Update
    Time:            2020-12-14T09:07:35Z
  Resource Version:  10150201
  Self Link:         /apis/trident.netapp.io/v1/namespaces/trident/tridentprovisioners/trident
  UID:               31c70948-d7db-4eba-9718-5b1d8b5c6723
Spec:
  Debug:  true
Status:
  Current Installation Params:
    IPv6:
    Autosupport Hostname:
    Autosupport Image:
    Autosupport Proxy:
    Autosupport Serial Number:
    Debug:
    Enable Node Prep:
    Image Pull Secrets:         <nil>
    Image Registry:
    k8sTimeout:
    Kubelet Dir:
    Log Format:
    Silence Autosupport:
    Trident Image:
  Message:                      Failed to install Trident; err: could not create the Trident service account; could not create service account; yamlDocument ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: trident-csi
  labels:
    app: controller.csi.trident.netapp.io
    k8s_version: v1.18.5
    trident_version: v20.10.0
  ownerReferences:
  - apiVersion: trident.netapp.io/v1
    controller: true
    kind: TridentProvisioner
    name: trident
    uid: 31c70948-d7db-4eba-9718-5b1d8b5c6723
 was not created after 30.00 seconds
  Status:   Failed
  Version:
Events:
  Type     Reason  Age                From                        Message
  ----     ------  ----               ----                        -------
  Warning  Failed  60m (x2 over 71m)  trident-operator.netapp.io  Failed to install Trident; err: could not create the Trident service account; could not create service account; yamlDocument ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: trident-csi
  labels:
    k8s_version: v1.18.5
    trident_version: v20.10.0
    app: controller.csi.trident.netapp.io
  ownerReferences:
  - uid: 31c70948-d7db-4eba-9718-5b1d8b5c6723
    apiVersion: trident.netapp.io/v1
    controller: true
    kind: TridentProvisioner
    name: trident
 was not created after 30.00 seconds
  Warning  Failed  57m  trident-operator.netapp.io  (combined from similar events): Failed to install Trident; err: could not create the Trident service account; could not create service account; yamlDocument ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: trident-csi
  labels:
    k8s_version: v1.18.5
    trident_version: v20.10.0
    app: controller.csi.trident.netapp.io
  ownerReferences:
  - name: trident
    uid: 31c70948-d7db-4eba-9718-5b1d8b5c6723
    apiVersion: trident.netapp.io/v1
    controller: true
    kind: TridentProvisioner
 was not created after 30.00 seconds
  Warning  Failed  40m (x7 over 75m)  trident-operator.netapp.io  Failed to install Trident; err: could not create the Trident service account; could not create service account; yamlDocument ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: trident-csi
  labels:
    app: controller.csi.trident.netapp.io
    k8s_version: v1.18.5
    trident_version: v20.10.0
  ownerReferences:
  - name: trident
    uid: 31c70948-d7db-4eba-9718-5b1d8b5c6723
    apiVersion: trident.netapp.io/v1
    controller: true
    kind: TridentProvisioner
 was not created after 30.00 seconds
  Warning  Failed  35m (x4 over 63m)  trident-operator.netapp.io  Failed to install Trident; err: could not create the Trident service account; could not create service account; yamlDocument ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: trident-csi
  labels:
    trident_version: v20.10.0
    app: controller.csi.trident.netapp.io
    k8s_version: v1.18.5
  ownerReferences:
  - apiVersion: trident.netapp.io/v1
    controller: true
    kind: TridentProvisioner
    name: trident
    uid: 31c70948-d7db-4eba-9718-5b1d8b5c6723
 was not created after 30.00 seconds
  Warning  Failed  15m (x6 over 69m)  trident-operator.netapp.io  Failed to install Trident; err: could not create the Trident service account; could not create service account; yamlDocument ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: trident-csi
  labels:
    app: controller.csi.trident.netapp.io
    k8s_version: v1.18.5
    trident_version: v20.10.0
  ownerReferences:
  - kind: TridentProvisioner
    name: trident
    uid: 31c70948-d7db-4eba-9718-5b1d8b5c6723
    apiVersion: trident.netapp.io/v1
    controller: true
 was not created after 30.00 seconds
  Warning  Failed  7m21s (x9 over 70m)  trident-operator.netapp.io  Failed to install Trident; err: could not create the Trident service account; could not create service account; yamlDocument ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: trident-csi
  labels:
    app: controller.csi.trident.netapp.io
    k8s_version: v1.18.5
    trident_version: v20.10.0
  ownerReferences:
  - uid: 31c70948-d7db-4eba-9718-5b1d8b5c6723
    apiVersion: trident.netapp.io/v1
    controller: true
    kind: TridentProvisioner
    name: trident
 was not created after 30.00 seconds
  Warning  Failed  5m52s (x27 over 73m)  trident-operator.netapp.io  Failed to install Trident; err: could not create the Trident service account; could not create service account; yamlDocument ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: trident-csi
  labels:
    app: controller.csi.trident.netapp.io
    k8s_version: v1.18.5
    trident_version: v20.10.0
  ownerReferences:
  - apiVersion: trident.netapp.io/v1
    controller: true
    kind: TridentProvisioner
    name: trident
    uid: 31c70948-d7db-4eba-9718-5b1d8b5c6723
 was not created after 30.00 seconds

Environment
Provide accurate information about the environment to help us reproduce the issue.

Trident version: 20.10
Trident installation flags used: I just use default installation
Container runtime:
Kubernetes version: 1.18.5
Kubernetes orchestrator: OSS(kubeadm)
Kubernetes enabled feature gates:
OS: Ubuntu 18.04.05
NetApp backend types: not connected at this moment (before adding NetApp backend)
Other:

To Reproduce
Just do following steps.

3: Creating a TridentProvisioner CR and installing Trident

Expected behavior
Deployment ｗorks Sucessfully

Additional context

bug

Source

sakaia

All 11 comments

Hello @sakaia,

Can you try creating a service account in general within trident namespace, does it work?
It does look like your service account YAML is correct, yet it didn't get created after 30 seconds.

Can you try both these samples to see if a service account can be created:

Save below sample in service_account_1.yaml and then try kubectl create -f service_account_1.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: service1
  labels:
    k8s_version: v1.18.5
    trident_version: v20.10.0
    app: controller.csi.trident.netapp.io

Save below sample in service_account_2.yaml and then try kubectl create -f service_account_2.yaml. Here ensure the tridentprovisioner CR exists and replace uid below with the uid of the tridentprovisioner CR.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: service2
  labels:
    k8s_version: v1.18.5
    trident_version: v20.10.0
    app: controller.csi.trident.netapp.io
  ownerReferences:
  - uid: 31c70948-d7db-4eba-9718-5b1d8b5c6723
    apiVersion: trident.netapp.io/v1
    controller: true
    kind: TridentProvisioner
    name: trident

ntap-arorar on 14 Dec 2020

👍1

Thanks

Both kubectl command works fine. (but I am checking, I found some problem)

Can you try creating a service account in general within trident namespace, does it work?
It does look like your service account YAML is correct, yet it didn't get created after 30 seconds.

Can you try both these samples to see if a service account can be created:

Save below sample in service_account_1.yaml and then try kubectl create -f service_account_1.yaml

snip

Save below sample in service_account_2.yaml and then try kubectl create -f service_account_2.yaml. Here ensure the tridentprovisioner CR exists and replace uid below with the uid of the tridentprovisioner CR.

kubectl get sa -A does not show service2. and service1 seems fine.
kubectl create -f service_account_2.yaml returns created many times. Usual command returns Error from server (AlreadyExists): error when creating "service_account_1.yaml": serviceaccounts "service1" already exists

$ kubectl create -f service_account_2.yaml -n trident
serviceaccount/service2 created
$ kubectl create -f service_account_2.yaml -n trident
serviceaccount/service2 created
$ kubectl create -f service_account_2.yaml -n trident
serviceaccount/service2 created
$ kubectl create -f service_account_2.yaml -n trident
serviceaccount/service2 created

sakaia on 16 Dec 2020

Hi @sakaia,

I have noticed this issue when Tprov (TridentProvisioner) CR does not exist, i.e. uid: 31c70948-d7db-4eba-9718-5b1d8b5c6723 under ownerRef does not belong to any Tprov CR.

$ kubectl create -f service_account_2.yaml -n trident
serviceaccount/service2 created
$ kubectl create -f service_account_2.yaml -n trident
serviceaccount/service2 created
$ kubectl create -f service_account_2.yaml -n trident
serviceaccount/service2 created
$ kubectl create -f service_account_2.yaml -n trident
serviceaccount/service2 created

The solution is to first create a TridentProvisioner CR, take a note of its UUID and put it in the service_account_2.yaml's, then create the serviceaccount.

If you are able to create ServiceAccount, then I am not sure why service account creation fails when Trident Operator tries to create one. May be operator logs can provide us with more insights, to capture those you can run kubectl -n trident logs <trident-operator-pod>.

And could you please also confirm if clusterrole definition in bundle.yaml was modified before installing the operator?

ntap-arorar on 16 Dec 2020

👍1

Thanks but I still have a problem.
With updating uuid for trident-operator, service_account_2.yaml works successfully.
But Still have a problem, Typical log is follows

time="2020-12-16T09:35:22Z" level=info msg="'TridentProvisioner' CR updated." name=trident namespace=trident
time="2020-12-16T09:35:22Z" level=info msg="Trident deployment not found."
time="2020-12-16T09:35:22Z" level=debug msg="No operator based CSI Trident deployment found."
time="2020-12-16T09:35:22Z" level=info msg="Reconciler found Trident installation." callingCRName=trident callingCRNamespace=trident callingResourceType=resourceTridentProvisionerCR controllingCRBasedOnStatus=trident controllingCRBasedOnStatusNamespace=trident isCSI=true namespace= operatorBasedCSIDeployments="[]"
time="2020-12-16T09:35:22Z" level=debug msg="Controlling CR: 'trident', Controlling CR Namespace: 'trident'"
time="2020-12-16T09:35:22Z" level=debug msg="'trident' in namespace 'trident' is a controlling CR based on status."
time="2020-12-16T09:35:22Z" level=warning msg="Could not get server resources, defaulting to Kubernetes flavor." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
time="2020-12-16T09:35:22Z" level=debug msg="Initialized Kubernetes API client." cli=kubectl flavor=k8s namespace=trident timeout=30s version=v1.18.5
time="2020-12-16T09:35:22Z" level=debug msg="Initialized installer." namespace=trident
time="2020-12-16T09:35:22Z" level=info msg="Trident deployment not found."
time="2020-12-16T09:35:22Z" level=info msg="Trident daemonset not found."
time="2020-12-16T09:35:22Z" level=debug msg="Alpha snapshot CRD not present." CRD=volumesnapshots.snapshot.storage.k8s.io
time="2020-12-16T09:35:22Z" level=debug msg="Alpha snapshot CRD not present." CRD=volumesnapshotclasses.snapshot.storage.k8s.io
time="2020-12-16T09:35:22Z" level=debug msg="Alpha snapshot CRD not present." CRD=volumesnapshotcontents.snapshot.storage.k8s.io
time="2020-12-16T09:35:22Z" level=debug msg="Kubernetes version 'v1.18.5' is supported."
time="2020-12-16T09:35:22Z" level=warning msg="Could not get server resources, defaulting to Kubernetes flavor." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
time="2020-12-16T09:35:22Z" level=debug msg="Initialized Kubernetes API client." cli=kubectl flavor=k8s namespace=trident timeout=30s version=v1.18.5
time="2020-12-16T09:35:22Z" level=debug msg="Initialized installer." namespace=trident
time="2020-12-16T09:35:22Z" level=debug msg="Using default Trident image 'netapp/trident:20.10.0', version 'v20.10.0'"
time="2020-12-16T09:35:22Z" level=info msg="Current deployment version '' is not same as the Trident image version 'v20.10.0'; need to update the installation"
time="2020-12-16T09:35:22Z" level=info msg="Starting Trident installation." namespace=trident
time="2020-12-16T09:35:22Z" level=error msg="Could not delete Trident service account." error="serviceaccounts \"service1\" is forbidden: User \"system:serviceaccount:trident:trident-operator\" cannot delete resource \"serviceaccounts\" in API group \"\" in the namespace \"trident\"" label="app=controller.csi.trident.netapp.io" namespace=trident serviceAccount=service1
time="2020-12-16T09:35:22Z" level=debug msg="Updating Trident Provisioner CR after failed installation." name=trident namespace=trident
time="2020-12-16T09:35:22Z" level=info msg="New status is same as the old status, no update needed." name=trident namespace=trident
time="2020-12-16T09:35:22Z" level=error msg="error syncing 'trident/trident': reconcile failed; error re-installing Trident 'trident' in namespace 'trident'; err: reconcile failed; could not create the Trident service account; unable to delete Trident service account(s): [trident/service1], requeuing"
time="2020-12-16T09:35:22Z" level=info msg=-------------------------------------------------
time="2020-12-16T09:35:22Z" level=info msg=-------------------------------------------------
time="2020-12-16T09:35:22Z" level=error msg="error syncing 'trident/trident': reconcile failed; error re-installing Trident 'trident' in namespace 'trident'; err: reconcile failed; could not create the Trident service account; unable to delete Trident service account(s): [trident/service1], requeuing"

p.s.
deployment trident-operator itself seems work

$ kubectl describe deployment trident-operator -n trident
Name:                   trident-operator
Namespace:              trident
CreationTimestamp:      Mon, 14 Dec 2020 07:50:15 +0000
Labels:                 app=operator.trident.netapp.io
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=operator.trident.netapp.io,name=trident-operator
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=operator.trident.netapp.io
                    name=trident-operator
  Service Account:  trident-operator
  Containers:
   trident-operator:
    Image:      netapp/trident-operator:20.10.0
    Port:       <none>
    Host Port:  <none>
    Command:
      /trident-operator
      --debug
    Environment:
      POD_NAME:        (v1:metadata.name)
      OPERATOR_NAME:  trident-operator
    Mounts:           <none>
  Volumes:            <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   trident-operator-67448748f7 (1/1 replicas created)
Events:          <none>

sakaia on 16 Dec 2020

Trident Operator says following errors. Is there any method to solve these problem?

time="2020-12-16T09:35:22Z" level=info msg="Trident deployment not found."
time="2020-12-16T09:35:22Z" level=debug msg="No operator based CSI Trident deployment found."

sakaia on 17 Dec 2020

Trident Operator says the following errors. Is there any method to solve these problem?
time="2020-12-16T09:35:22Z" level=info msg="Trident deployment not found."
time="2020-12-16T09:35:22Z" level=debug msg="No operator based CSI Trident deployment found."

These are normal logs, just identifying that Trident is not installed yet.

The purpose of creating service1 and service2 was to ensure serviceaccount can be created in your setup.

Could you please retry after deleting those serviceaccounts kubectl -n trident delete serviceaccount service1 (similarly for service2) and retry installing Trident by creating CR and post the logs.
Can you please confirm if you installed bundle.yaml and not operator.yaml instead, otherwise operator's cluster-role/cluster-role bindings will not be created.
Also would appreciate if you could confirm if you are running vanilla Kubernetes, is that correct?

Thanks!

ntap-arorar on 17 Dec 2020

Just want to add that I have installed Trident 20.10.0 using operator on K8s 1.18 on Ubuntu 18.04, and haven't noticed any issue, I am curious why you are running into this issue.

ntap-arorar on 17 Dec 2020

Could you please retry after deleting those serviceaccounts kubectl -n trident delete serviceaccount service1 (similarly for service2) and retry installing Trident by creating CR and post the logs.

I attach the log below.

Just want to add that I have installed Trident 20.10.0 using operator on K8s 1.18 on Ubuntu 18.04, and haven't noticed any issue, I am curious why you are running into this issue.

Yes, I am using Ubuntu 18.04.5, Trident 20.10.0 and Kubernetes 1.18.5

time="2020-12-17T07:25:07Z" level=info msg="TridentProvisioner CR added." name=trident namespace=trident
time="2020-12-17T07:25:07Z" level=info msg="Trident deployment not found."
time="2020-12-17T07:25:07Z" level=debug msg="No operator based CSI Trident deployment found."
time="2020-12-17T07:25:07Z" level=debug msg="Kubernetes version 'v1.18.5' is supported."
time="2020-12-17T07:25:07Z" level=debug msg="Alpha snapshot CRD not present." CRD=volumesnapshots.snapshot.storage.k8s.io
time="2020-12-17T07:25:07Z" level=debug msg="Alpha snapshot CRD not present." CRD=volumesnapshotclasses.snapshot.storage.k8s.io
time="2020-12-17T07:25:07Z" level=debug msg="Alpha snapshot CRD not present." CRD=volumesnapshotcontents.snapshot.storage.k8s.io
time="2020-12-17T07:25:07Z" level=info msg="Reconciler found no Operator-based Trident installation."
time="2020-12-17T07:25:07Z" level=debug msg="Updating TridentProvisioner CR before new installation" name=trident namespace=trident
time="2020-12-17T07:25:07Z" level=info msg="'TridentProvisioner' CR updated." name=trident namespace=trident
time="2020-12-17T07:25:07Z" level=warning msg="Could not get server resources, defaulting to Kubernetes flavor." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
time="2020-12-17T07:25:07Z" level=debug msg="Initialized Kubernetes API client." cli=kubectl flavor=k8s namespace=trident timeout=30s version=v1.18.5
time="2020-12-17T07:25:07Z" level=debug msg="Initialized installer." namespace=trident
time="2020-12-17T07:25:07Z" level=debug msg="Using default Trident image 'netapp/trident:20.10.0', version 'v20.10.0'"
time="2020-12-17T07:25:07Z" level=info msg="Current deployment version '' is not same as the Trident image version 'v20.10.0'; need to update the installation"
time="2020-12-17T07:25:07Z" level=info msg="Starting Trident installation." namespace=trident
time="2020-12-17T07:25:07Z" level=info msg="Trident service account not found."
time="2020-12-17T07:25:07Z" level=debug msg="Deleting unlabeled Trident service account by name as it can cause issues during installation."
time="2020-12-17T07:25:07Z" level=debug msg="Parsed YAML into unstructured object." group= kind=ServiceAccount version=v1
time="2020-12-17T07:25:07Z" level=error msg="API resource not found." group= kind=ServiceAccount version=v1
I1217 07:25:08.429963       1 request.go:645] Throttling request took 1.172071709s, request: GET:https://10.96.0.1:443/apis/trident.netapp.io/v1?timeout=32s
I1217 07:25:18.625642       1 request.go:645] Throttling request took 6.797914766s, request: GET:https://10.96.0.1:443/apis/autoscaling/v2beta1?timeout=32s
time="2020-12-17T07:25:19Z" level=error msg="Could not get dynamic resource, failed to list server resources." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request" group= kind=ServiceAccount version=v1
time="2020-12-17T07:25:19Z" level=error msg="Object creation failed." err="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request" yamlDocument="---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: trident-csi\n  labels:\n    app: controller.csi.trident.netapp.io\n    k8s_version: v1.18.5\n    trident_version: v20.10.0\n  ownerReferences:\n  - name: trident\n    uid: d255c7b2-6b94-4819-aa88-7a85ad506015\n    apiVersion: trident.netapp.io/v1\n    controller: true\n    kind: TridentProvisioner\n"
time="2020-12-17T07:25:19Z" level=debug msg="Object not created, waiting." err="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request" increment=633.905063ms yamlDocument="---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: trident-csi\n  labels:\n    app: controller.csi.trident.netapp.io\n    k8s_version: v1.18.5\n    trident_version: v20.10.0\n  ownerReferences:\n  - name: trident\n    uid: d255c7b2-6b94-4819-aa88-7a85ad506015\n    apiVersion: trident.netapp.io/v1\n    controller: true\n    kind: TridentProvisioner\n"
time="2020-12-17T07:25:20Z" level=debug msg="Parsed YAML into unstructured object." group= kind=ServiceAccount version=v1
time="2020-12-17T07:25:20Z" level=error msg="API resource not found." group= kind=ServiceAccount version=v1
time="2020-12-17T07:25:27Z" level=info msg="'TridentProvisioner' CR updated." name=trident namespace=trident
I1217 07:25:29.024244       1 request.go:645] Throttling request took 1.198073859s, request: GET:https://10.96.0.1:443/apis/coordination.k8s.io/v1beta1?timeout=32s
time="2020-12-17T07:25:35Z" level=error msg="Could not get dynamic resource, failed to list server resources." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request" group= kind=ServiceAccount version=v1
time="2020-12-17T07:25:35Z" level=error msg="Object creation failed." err="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request" yamlDocument="---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: trident-csi\n  labels:\n    app: controller.csi.trident.netapp.io\n    k8s_version: v1.18.5\n    trident_version: v20.10.0\n  ownerReferences:\n  - name: trident\n    uid: d255c7b2-6b94-4819-aa88-7a85ad506015\n    apiVersion: trident.netapp.io/v1\n    controller: true\n    kind: TridentProvisioner\n"
time="2020-12-17T07:25:35Z" level=debug msg="Object not created, waiting." err="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request" increment=405.703671ms yamlDocument="---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: trident-csi\n  labels:\n    app: controller.csi.trident.netapp.io\n    k8s_version: v1.18.5\n    trident_version: v20.10.0\n  ownerReferences:\n  - name: trident\n    uid: d255c7b2-6b94-4819-aa88-7a85ad506015\n    apiVersion: trident.netapp.io/v1\n    controller: true\n    kind: TridentProvisioner\n"
time="2020-12-17T07:25:35Z" level=debug msg="Parsed YAML into unstructured object." group= kind=ServiceAccount version=v1
time="2020-12-17T07:25:35Z" level=error msg="API resource not found." group= kind=ServiceAccount version=v1
I1217 07:25:39.223601       1 request.go:645] Throttling request took 3.389005291s, request: GET:https://10.96.0.1:443/apis/batch/v1beta1?timeout=32s
I1217 07:25:49.422272       1 request.go:645] Throttling request took 5.597395196s, request: GET:https://10.96.0.1:443/apis/trident.netapp.io/v1?timeout=32s
time="2020-12-17T07:25:51Z" level=error msg="Could not get dynamic resource, failed to list server resources." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request" group= kind=ServiceAccount version=v1
time="2020-12-17T07:25:51Z" level=error msg="Object creation failed." err="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request" yamlDocument="---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: trident-csi\n  labels:\n    app: controller.csi.trident.netapp.io\n    k8s_version: v1.18.5\n    trident_version: v20.10.0\n  ownerReferences:\n  - name: trident\n    uid: d255c7b2-6b94-4819-aa88-7a85ad506015\n    apiVersion: trident.netapp.io/v1\n    controller: true\n    kind: TridentProvisioner\n"
time="2020-12-17T07:25:51Z" level=debug msg="Updating Trident Provisioner CR after failed installation." name=trident namespace=trident
time="2020-12-17T07:25:51Z" level=error msg="error syncing 'trident/trident': reconcile failed; error installing Trident using CR 'trident' in namespace 'trident'; err: reconcile failed; could not create the Trident service account; could not create service account; yamlDocument ---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: trident-csi\n  labels:\n    app: controller.csi.trident.netapp.io\n    k8s_version: v1.18.5\n    trident_version: v20.10.0\n  ownerReferences:\n  - name: trident\n    uid: d255c7b2-6b94-4819-aa88-7a85ad506015\n    apiVersion: trident.netapp.io/v1\n    controller: true\n    kind: TridentProvisioner\n was not created after 30.00 seconds, requeuing"
time="2020-12-17T07:25:51Z" level=info msg=-------------------------------------------------
time="2020-12-17T07:25:51Z" level=info msg=-------------------------------------------------
time="2020-12-17T07:25:51Z" level=error msg="error syncing 'trident/trident': reconcile failed; error installing Trident using CR 'trident' in namespace 'trident'; err: reconcile failed; could not create the Trident service account; could not create service account; yamlDocument ---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: trident-csi\n  labels:\n    app: controller.csi.trident.netapp.io\n    k8s_version: v1.18.5\n    trident_version: v20.10.0\n  ownerReferences:\n  - name: trident\n    uid: d255c7b2-6b94-4819-aa88-7a85ad506015\n    apiVersion: trident.netapp.io/v1\n    controller: true\n    kind: TridentProvisioner\n was not created after 30.00 seconds, requeuing"
time="2020-12-17T07:25:51Z" level=info msg="'TridentProvisioner' CR updated." name=trident namespace=trident
time="2020-12-17T07:25:51Z" level=info msg="Trident deployment not found."
time="2020-12-17T07:25:51Z" level=debug msg="No operator based CSI Trident deployment found."
time="2020-12-17T07:25:51Z" level=info msg="Reconciler found Trident installation." callingCRName=trident callingCRNamespace=trident callingResourceType=resourceTridentProvisionerCR controllingCRBasedOnStatus=trident controllingCRBasedOnStatusNamespace=trident isCSI=true namespace= operatorBasedCSIDeployments="[]"
time="2020-12-17T07:25:51Z" level=debug msg="Controlling CR: 'trident', Controlling CR Namespace: 'trident'"
time="2020-12-17T07:25:51Z" level=debug msg="'trident' in namespace 'trident' is a controlling CR based on status."
time="2020-12-17T07:25:51Z" level=warning msg="Could not get server resources, defaulting to Kubernetes flavor." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
time="2020-12-17T07:25:51Z" level=debug msg="Initialized Kubernetes API client." cli=kubectl flavor=k8s namespace=trident timeout=30s version=v1.18.5
time="2020-12-17T07:25:51Z" level=debug msg="Initialized installer." namespace=trident
time="2020-12-17T07:25:51Z" level=info msg="Trident deployment not found."
time="2020-12-17T07:25:51Z" level=info msg="Trident daemonset not found."
time="2020-12-17T07:25:51Z" level=debug msg="Alpha snapshot CRD not present." CRD=volumesnapshots.snapshot.storage.k8s.io
time="2020-12-17T07:25:51Z" level=debug msg="Alpha snapshot CRD not present." CRD=volumesnapshotclasses.snapshot.storage.k8s.io
time="2020-12-17T07:25:51Z" level=debug msg="Alpha snapshot CRD not present." CRD=volumesnapshotcontents.snapshot.storage.k8s.io
time="2020-12-17T07:25:51Z" level=debug msg="Kubernetes version 'v1.18.5' is supported."
time="2020-12-17T07:25:51Z" level=warning msg="Could not get server resources, defaulting to Kubernetes flavor." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
time="2020-12-17T07:25:51Z" level=debug msg="Initialized Kubernetes API client." cli=kubectl flavor=k8s namespace=trident timeout=30s version=v1.18.5
time="2020-12-17T07:25:51Z" level=debug msg="Initialized installer." namespace=trident
time="2020-12-17T07:25:51Z" level=debug msg="Using default Trident image 'netapp/trident:20.10.0', version 'v20.10.0'"
time="2020-12-17T07:25:51Z" level=info msg="Current deployment version '' is not same as the Trident image version 'v20.10.0'; need to update the installation"
time="2020-12-17T07:25:51Z" level=info msg="Starting Trident installation." namespace=trident
time="2020-12-17T07:25:51Z" level=info msg="Trident service account not found."
time="2020-12-17T07:25:51Z" level=debug msg="Deleting unlabeled Trident service account by name as it can cause issues during installation."
time="2020-12-17T07:25:51Z" level=debug msg="Parsed YAML into unstructured object." group= kind=ServiceAccount version=v1
time="2020-12-17T07:25:51Z" level=error msg="API resource not found." group= kind=ServiceAccount version=v1
I1217 07:25:59.485214       1 request.go:645] Throttling request took 3.397962475s, request: GET:https://10.96.0.1:443/apis/crd.projectcalico.org/v1?timeout=32s
time="2020-12-17T07:26:03Z" level=error msg="Could not get dynamic resource, failed to list server resources." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request" group= kind=ServiceAccount version=v1
time="2020-12-17T07:26:03Z" level=error msg="Object creation failed." err="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request" yamlDocument="---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: trident-csi\n  labels:\n    app: controller.csi.trident.netapp.io\n    k8s_version: v1.18.5\n    trident_version: v20.10.0\n  ownerReferences:\n  - name: trident\n    uid: d255c7b2-6b94-4819-aa88-7a85ad506015\n    apiVersion: trident.netapp.io/v1\n    controller: true\n    kind: TridentProvisioner\n"
time="2020-12-17T07:26:03Z" level=debug msg="Object not created, waiting." err="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request" increment=692.495874ms yamlDocument="---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: trident-csi\n  labels:\n    app: controller.csi.trident.netapp.io\n    k8s_version: v1.18.5\n    trident_version: v20.10.0\n  ownerReferences:\n  - name: trident\n    uid: d255c7b2-6b94-4819-aa88-7a85ad506015\n    apiVersion: trident.netapp.io/v1\n    controller: true\n    kind: TridentProvisioner\n"
time="2020-12-17T07:26:04Z" level=debug msg="Parsed YAML into unstructured object." group= kind=ServiceAccount version=v1
time="2020-12-17T07:26:04Z" level=error msg="API resource not found." group= kind=ServiceAccount version=v1
I1217 07:26:09.684534       1 request.go:645] Throttling request took 5.301402273s, request: GET:https://10.96.0.1:443/apis/batch/v1?timeout=32s

sakaia on 17 Dec 2020

Can you please confirm if you installed bundle.yaml and not operator.yaml instead, otherwise operator's cluster-role/cluster-role bindings will not be created.

kubectl get clusterrole and kubectl get clusterrolebinding works fine.

$ kubectl get clusterrole -n trident
trident-operator                                                       2020-12-17T06:05:25Z

$ kubectl get clusterrolebinding -n trident
trident-operator                                                ClusterRole/trident-operator                                                       179m

Also would appreciate if you could confirm if you are running vanilla Kubernetes, is that correct?

Currently I am using vanilla Kubernetes. At this moment it seems workfine.

sakaia on 17 Dec 2020

Hello @sakaia

This seems to be the problem:

time="2020-12-17T07:25:19Z" level=error msg="Could not get dynamic resource, failed to list server resources." error="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request" group= kind=ServiceAccount version=v1
time="2020-12-17T07:25:19Z" level=error msg="Object creation failed." err="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request" yamlDocument="---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: trident-csi\n  labels:\n    app: controller.csi.trident.netapp.io\n    k8s_version: v1.18.5\n    trident_version: v20.10.0\n  ownerReferences:\n  - name: trident\n    uid: d255c7b2-6b94-4819-aa88-7a85ad506015\n    apiVersion: trident.netapp.io/v1\n    controller: true\n    kind: TridentProvisioner\n"

Internally Operator does something similar to kubectl create -f xyz.yaml i.e. create Trident objects such as deployment, serivceaccount, services etc using YAML. And, it verifies if the given resources (like Pod etc) is valid or not and after verification calls the appropriate create on that resource. The step at which resource identification/verification occurs is failing. Is your cluster in a good health and could you please verify if the API services within your Kubernetes cluster are up and running?

Try kubectl get apiservice, check the ones with AVAILABLE set to False.

ntap-arorar on 17 Dec 2020