Operator-sdk: Ansible: Slower with many pods

[mikkeschiren]

Type of question

Best practice

Question

What did you do?
I have built an operator that creates new pods when deploying - each pod represents a site and have config maps secrets etc managed by the operator. After a while, when many pods up using the operator - it is getting slow in the response, running through in loops all deployments using the operator. So if possible how do I make the operator react only on newly created deployments, and stop it from going though all sites?

What did you expect to see?
A quick response when deploying using an ansible operator

[mikkeschiren]

Maybe related to #2358 ?

[estroz]

/cc @fabianvf

[fabianvf]

could you post your watches.yaml, or ideally a link to your operator?

Is the issue you're seeing that the Ansible is frequently reconciling all of the created Custom Resources, rather than intelligently responding to new events? If so you may want to set reconcilePeriod to 0 in the watches.yaml, which means that resources will only be reconciled when an event comes in, either from a watched Custom Resource, or any of the resources deployed as a result of a reconciliation. We should probably move to this being the default behavior, rather than reconciling on a schedule by default.

[mikkeschiren]

Hi!

Sorry - missed the answer - my watches.yaml looks like this now:

- version: v1alpha2
  group: ozzi.io
  kind: Drupal
  playbook: /opt/ansible/playbook.yaml
  watchDependentResources: true
  maxRunnerArtifacts: 30
  reconcilePeriod: 0
  manageStatus: true

I still get the issue with long waits, and that all deploys is runned through - I have same private stuff in the operator right now, but will try to clear it out to share it also.
The strange thing that happens is that the operator seems to think that one step is changed at every run.

[camilamacedo86]

HI @mikkeschiren,

You are using watchDependentResources: true, it will re-trigger the reconcile if any event (create/delete/update) in the dependent resources be raised. See: https://sdk.operatorframework.io/docs/ansible/reference/dependent-watches/

Has any reasons for you are using watchDependentResources: true?

[mikkeschiren]

Ah - yes I think that is the problem - I think I misunderstood the usage - did some quick tests now and it is looking good. I will get back to you soon. Thank you!

[mikkeschiren]

Yes, after some deeper testing that solved my problem, thanks for the help @camilamacedo86 .