Trident: Operator reports that its installed multiple times

[Numblesix]

Describe the bug
Operator reports installed in a "loop"

Environment
Provide accurate information about the environment to help us reproduce the issue.

• Trident version: 20.07
• Trident installation flags used: https://github.com/NetApp/trident/blob/stable/v20.07/deploy/crds/tridentprovisioner_cr.yaml
• Container runtime: CRI-0
• Kubernetes version: v1.18.3+012b3ec
• Kubernetes orchestrator: Openshift 4.5.4
• OS: RHEL CoreOS
• NetApp backend types: Ontap NAS and Ontap SAN

To Reproduce
Steps to reproduce the behavior: Install via Operator as described in DOc

Expected behavior
Trident install should only be reported once

Additional context

user@host ➜  ~  oc describe tprov -n trident trident                   
Name:         trident
Namespace:    trident
Labels:       <none>
Annotations:  <none>
API Version:  trident.netapp.io/v1
Kind:         TridentProvisioner
Metadata:
  Creation Timestamp:  2020-08-06T18:32:07Z
  Generation:          1
  Managed Fields:
    API Version:  trident.netapp.io/v1
    Operation:    Update
    Time:         2020-08-06T18:32:07Z
    API Version:  trident.netapp.io/v1
    Manager:         trident-operator
    Operation:       Update
    Time:            2020-08-06T18:33:22Z
  Resource Version:  25931
  Self Link:         /apis/trident.netapp.io/v1/namespaces/trident/tridentprovisioners/trident
  UID:               e2529796-b8de-45fa-bc96-fa2293b2bb13
Spec:
  Debug:                true
  Silence Autosupport:  true
Status:
  Current Installation Params:
    IPv6:               false
    Autosupport Image:  netapp/trident-autosupport:20.07.0
    Autosupport Proxy:  
    Debug:              true
    Image Pull Secrets:
    Image Registry:       quay.io
    k8sTimeout:           30
    Kubelet Dir:          /var/lib/kubelet
    Log Format:           text
    Silence Autosupport:  true
    Trident Image:        netapp/trident:20.07.0
  Message:                Trident installed
  Status:                 Installed
  Version:                v20.07.0
Events:
  Type    Reason     Age                  From                        Message
  ----    ------     ----                 ----                        -------
  Normal  Installed  55s (x182 over 14h)  trident-operator.netapp.io  Trident installed

[balaramesh]

@Numblesix perhaps I'm missing something here. Can you explain what you mean by "Operator reports that it is installed multiple times"? From your tprov describe output it looks like Trident was installed and the status field indicates that

[innergy]

It’s the last line of the output. Repeated 182 times over 14 hours.

[Numblesix]

Hi sorry ,

yeah i find it quite weird that it shows this line hear:

Events:
  Type    Reason     Age                  From                        Message
  ----    ------     ----                 ----                        -------
  Normal  Installed  55s (x182 over 14h)  trident-operator.netapp.io  Trident installed

Shouldnt that be displayed only once ?

[balaramesh]

@innergy 👍
@Numblesix the trident-operator runs a reconciling job every 5 minutes to sync the status of the tprov and make sure Trident is installed as the tprov object dictates. The operator's logs will show the sync job happening every 5 minutes. This is normal and expected

[Numblesix]

Thanks a lot :)

Seemed strange to me but then its okay :)

[gnarl]

We plan to provide a fix for this by improving the message.

[ntap-arorar]

As part of each reconcile loop Operator sets status as well as event message in accordance with the current state of Trident.

There is a logic in place to not to update the CR status if it is same as old status (to avoid continuous CR updates). However, event messages are set each and every time thus resulting in the above behaviour and increment of the age counter.

This behaviour is benign but can cause confusion, to avoid this confusion operator should only update the event message if the status is actually getting changed, else not.

[loxley]

Hi, we experience almost the same problem as described here but we are running Openshift 4.4.

But in our case new secrets are generated over and over again also. I guess on each restart it generates a new secret leading to this many:

➜  oc describe tprov -n trident trident
Name:         trident
Namespace:    trident
Labels:       app.kubernetes.io/instance=**************.com-trident
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"trident.netapp.io/v1","kind":"TridentProvisioner","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"****...
API Version:  trident.netapp.io/v1
Kind:         TridentProvisioner
Metadata:
  Creation Timestamp:  2020-05-14T10:14:44Z
  Generation:          3
  Resource Version:    102031764
  Self Link:           /apis/trident.netapp.io/v1/namespaces/trident/tridentprovisioners/trident
  UID:                 38c0978c-2c9b-4c49-a48f-d6bb947599ee
Spec:
  Debug:          true
  Trident Image:  netapp/trident:20.07
Status:
  Current Installation Params:
    IPv6:               false
    Autosupport Image:  netapp/trident-autosupport:20.07.0
    Autosupport Proxy:
    Debug:              true
    Image Pull Secrets:
    Image Registry:       quay.io
    k8sTimeout:           30
    Kubelet Dir:          /var/lib/kubelet
    Log Format:           text
    Silence Autosupport:  false
    Trident Image:        netapp/trident:20.07
  Message:                Trident installed
  Status:                 Installed
  Version:                v20.07.0
Events:
  Type    Reason     Age                    From                        Message
  ----    ------     ----                   ----                        -------
  Normal  Installed  2m49s (x1165 over 4d)  trident-operator.netapp.io  Trident installed

➜  oc get secrets -n trident | grep -c trident-csi-token
1168

[ntap-arorar]

@loxley

The fix for the issue (generation of multiple secrets) has already been checked into the master and is expected to be part of 20.07.1 release.

Link to the fix: https://github.com/NetApp/trident/commit/051294a25f96babe6b7180a20d807efd12441ef7

[gnarl]

This is fixed in the Trident 20.07.1 release.

[lfstm]

Hi, we are on trident 20.10.1 running on opsh4.6 and still get the following message every x minutes:

Generated from trident-operator.netapp.io
259 times in the last 19 hours
Trident installed

Is this expected/normal behaviour?

[Numblesix]

Yes this is expected with you version of trident once you update to 20.07 this is fixed see the comment above

[lfstm]

Sorry, my bad, corrected the version in my previous post, we are on 20.10.1

[Numblesix]

aah okay :)

Seems we got the issue back im still on 20.10.0 and can confirm that this is there again see below :fire:

```
➜ ~ oc describe tprov -n trident trident (debug/api-osr2-xeop-de:6443/du243)
Name: trident
Namespace: trident
Labels:
Annotations:
API Version: trident.netapp.io/v1
Kind: TridentProvisioner
Metadata:
....
Spec:
Debug: true
Silence Autosupport: true
Status:
Current Installation Params:
IPv6: false
Autosupport Hostname:
Autosupport Image: netapp/trident-autosupport:20.10.0
Autosupport Proxy:
Autosupport Serial Number:
Debug: true
Enable Node Prep: false
Image Pull Secrets:
Image Registry: quay.io
k8sTimeout: 30
Kubelet Dir: /var/lib/kubelet
Log Format: text
Silence Autosupport: true
Trident Image: netapp/trident:20.10.0
Message: Trident installed
Status: Installed
Version: v20.10.0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Installed 102s (x2861 over 9d) trident-operator.netapp.io Trident installed

[Numblesix]

@gnarl please reopen :) or Shall i open a new Issue and reference :) ?

[lfstm]

@Numblesix Thanks for the quick help 🎉

[mattikbk]

Please reopen this. I just upgraded from v20.04.0 to v20.10.1 and the operator still reports installs very frequently:

Normal Installed 2m42s (x9 over 26m) trident-operator.netapp.io Trident installed

[gnarl]

Reopening this issue as Trident is still reporting installs more than once.

[gnarl]

This is fixed with commit a60b7a which is included in the Trident 21.01 release.