Operator-sdk: How to achieve concurrency using the operator-sdk?

[gpanagiotidis]

Let's say I will need to create 1000 CRs at once, and for each CR the operator logic will have to make some external API requests. This means that the logic execution will be synchronous, first for the first CR, then for the second, third etc, the logic is executed for one object at a time. For 1000 CRs this is going to be a problem because it takes a lot of time to process all those requests.

Is there a way to run the reconcile logic concurrently for different objects to speed up the process?

[lilic]

Let's say I will need to create 1000 CRs at once

Why would you want to do that? Can you explain the use case? Do you actually have this problem right now?

[gpanagiotidis]

Why would you want to do that? Can you explain the use case? Do you actually have this problem right now?

No, I do not have this problem right now, it's more of a question about how I should implement things. I am developing an operator which will be registering users and websites to an external analytics application. The infrastructure is big with thousands of websites and for each website we will have a separate CR, so for each website my operator will be getting some information, let's say username, hostname, access rights etc and applying changes in the external application.

So the use case of my question is basically this: when the operator goes into production, there will be thousands of CRs created for each website that exists in our infrastructure (by another operator), what would be the best way to handle both the operator logic and finalization logic to make it as fast as possible?

I hope my use case is clear, if not I can try to rephrase.

[lilic]

@gpanagiotidis in my personal opinion I do not think that is the pattern that an operator or CRs fit, kubernetes is not a database so you should not be storing this data that way. The CRs are there to configure operators but not to be used as a database. Hopefully, that clears things up.

[gpanagiotidis]

@LiliC We chose to use CRDs and operators instead of controllers watching over openshift's resources such as routes and annotations in order to have a more clear approach. We will not be using kubernetes as a database, the external application has its own database. The operator simply automates the registration process. Each CR of this particular CRD will contain the information of each object. The operator then, depending on the state of the object, will be registering users to the external application, thus to the external database. It's a way to automate things. Whenever a new website is created, a new CR will be created, the operator will register the user and the website in the external application, and the user will have access there without any extra action from the administrators' or the user's side.

Is this not a valid use case for using operators?

[kvaps]

Hi, I somehow did not notice this issue when I was creating these two: https://github.com/operator-framework/operator-sdk/issues/1678, https://github.com/operator-framework/operator-sdk/issues/1679

I faced with the same problem.
About use case: I have about 500 nodes already now and number is growing. I want to manage them using CRD and ansible operator, parallel execution solves this problem partially. The most painful case is restarting of operator, because it tries to handle all the 500 nodes during the restart.

Thank you

[kvaps]

By the way I solved my problem by implementing locking mechanism in raw module, and allowing huge number tasks parallel execution

https://gist.github.com/kvaps/c16af57b57bfcd55afc3b4ccdd3cc287

[gpanagiotidis]

@kvaps How many task do you manage in parallel? Also, what is the problem that the locking mechanism actually solves? When an operator is restarted, it means that all CRs were deleted, so it need to recreate them. If you allow parallel execution then why is this a problem?

[buoto]

Is there a way to run the reconcile logic concurrently for different objects to speed up the process?

@gpanagiotidis In Go version there is a MaxConcurrentReconciles option that can be set during controller creation and it looks like it does what you've described in the first comment.

[kvaps]

@kvaps How many task do you manage in parallel? Also, what is the problem that the locking mechanism actually solves? When an operator is restarted, it means that all CRs were deleted, so it need to recreate them. If you allow parallel execution then why is this a problem?

I'm planning to allow about 50 concurent tasks running in one time. I have chassis with 45 blade servers and I want to manage them as custom resources in kubernetes.
Every task is "go to the chassis over ssh and do something, then collect info".
When I run a lots of jobs in one time, chassis can't handle all connections and starts to dropping them.

Locking mechanism is garanties that only one command can be running to each unit of time on same chassis.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci-robot]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.