Strimzi-kafka-operator: [Enhancement] Add Cruise Control support to allow topic partition rebalancing

[tomncooper]

Currently there is no way to automatically rebalance the replica of a topic across the Kafka cluster. A user has to perform these operations manually after scaling the cluster etc.

Adding support for Cruise Control would allow topic replica rebalancing to be handled automatically by the Operator.

Alternatively, we could create our own rebalancing optimiser, certainly CC is not K8s native, but it seems prudent to build and improve on existing work from the Kafka community.

This would provide a possible solution to discussion of topic replication in #191

[tomncooper]

The current plan is to have a separate CC deployment which is managed a part of the main Kafka CR via a balanceOperator (BO) which is part of the Entity Operator (EO).

Configuration of CC will be via the BO config:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
...
spec:
  kafka:
    Config:
  entityOperator:
      topicOperator: {}
      userOperator: {}
      balanceOperator: {
        <Cruise control config options go here>
      }
    …

A rebalance of the cluster's topic partition replicas would then be triggered by posting a KafkaRebalance CR:

apiVersion: balance.kafka.strimzi.io/v1alpha1
kind: KafkaBalance
metadata:
...
spec:
  kafkaClusterName: “my-cluster”
  operation: rebalance
    goals:
      - goal.one.classpath
        args:
           arg1
      - goal.two.classpath
    …
status: #Generated
    conditions:   
      - lastTransitionTime: 2019-10-29T14:25:55+0000
        reason: “In progress” 
        status: "False"
        type: BalanceOperatonComplete

The first step is to get CC deployed and the CC metrics reporter automatically added to the Kafka pods.

If anyone has any issues or comments on the above it would be great to hear them.

[ppatierno]

My first comment is about the balanceOperator field.
If we are not going to support any other platform different from Cruise Control (CC) so we are not trying to have a general "control" platform configuration applicable to all of them, then maybe we should have cruiseControlOperator to make it clear that it deploys CC and the configuration is about CC. I think that adding support for a new "control" platform could mean just adding a different operator with a completely different configuration.

Even about the KafkaRebalance or KafkaBalance (it has two different names here :-)) I guess that the "goals" are more CC concepts that maybe could not exist in a future "control" platform. I would make clear that this is a CC related resource (even if KafkaCruiseControlRebalance seems to be too long?).

Because using CC means to set metrics.reporters property in the brokers configuration are we going to make it forbidden in the spec.kafka.config section or merging with any other reporters that the user can add? What about the metrics.polling.interval.secs property as well? Should we leave that in the `spec.kafka.config' section or locking down in the CC operator section with a different name? (making it forbidden in the kafka config as well?)

[ppatierno]

@kyguy @tomncooper I have doubts about having the CC inside the EO declaration.

We need a way for:

1. allowing the user to declare that the CC should be deployed. The CC will be a Deployment (so a related pod) running alongside the Kafka cluster. If we want to compare to something we have today, we can mention kafkaExporter which is declared directly in Kafka.spec. This means that we "could" have something like cruiseControl directly under Kafka.spec as well with the CC configuration. The CO is in charge to create the CC deployment when a new Kafka resource is created.
2. because we want to configure rebalance actions/goals for CC via Kubernetes resources and not directly via APIs (interacting with the HTTP REST API in the CC), we have a new KafkaRebalance (KafkaBalace or CruiseControlRebalance) CRD. In this case we need a new CCO (Cruise Control Operator ?) in charge to watching this kind of resource and then interacting with the CC API for configuring goals/rebalancing and so on. This operator "could" be declared in the entityOperator section of the Kafka resource. It sounds to me that the operator will run as a container in the EO pod alongside the TO, UO and the TLS sidecar.

Said that ... I don't like the way the user should declare two things (the CC deployment + the CCO in the EO) if we assume that we don't want to expose the CC API to the user (in that case the user could decide to deploy the CC but not using the CCO but directly the HTTP API).
It's also true that ... is clear enough for the user that having just the CCO under the EO declaration means also deploying the CC but as a separate pod and not as a container inside the EO as the CCO (with TO, UO, and TLS sidecar)? Having a CC "operator" in the EO declaration sounds to me that I am deploying an .... "operator" but not the actual CC as well. Is it just a matter of a good documentation?

Wdyt? @tombentley @scholzj any other thoughts from you?

[tomncooper]

I agree having both a Kafka.spec.cruiseControl and the entityOperator.cruiseControlOperator would be confusing and lead to errors. I think having just the CCO is reasonable.

I see the actual CC deployment like ZK, the user shouldn't have to worry about it. All config and operation is through the cruiseControlOperator and if that is not present in the Kafka CR no CC deployment or Kafka metrics reporter is added.

[ppatierno]

Yeah, but what about things like ResourceRequirements, Logging, probes etc etc ... you can have them for the CCO (as we have for TO and UO) but even for the CC deployment itself. How are we going to declare them in the custom resource for both? The current PR has this kind of stuff for the operator only as far as I can see. This let me think that we should have somehow a kind of separation between what is CC related and CCO related.

[tomncooper]

That is a good point. I was just thinking of CC config and not configs that both the CC deployment and CCO would need. In that case would the CCO config have things like cruiseControl.deployment.logging and cruiseControl.operator.logging as a way to separate them?

[ppatierno]

well is we are using the EO approach, we should have a field consistent with the others TO and UO so cruiseControlOperator but at this point what putting inside. Maybe it makes sense what @kyguy did in the PR? so something like:

entityOperator:
   cruiseControlOperator:
      logging:
        # .....
      resources:
        # .....
      cruiseControl:
          # cruise control config
          # .....
          logging:
             # ....
          resources:
             # ....

but I see this EO section growing :-(

[ppatierno]

One more question, what does classpath and args mean in the goals?

goal.one.classpath
        args:
           arg1

[tomncooper]

The classpath thing was just to indicate that this would be the full name of the Goal class as this is how they are set in the CC configs.

[ppatierno]

Thinking more about the new KafkaConnector resource related to the actual KafkaConnect.
Aren't we sure that we wouldn't have a KafkaCruiseControl CRD for the CC deployment exposing the HTTP REST API if some user wants but at the same time providing a corresponding cruiseControlBalanceOperator for handling the KafkaBalance resources? This would split the configuration. I am just thinking aloud raising all the points that come to my mind ... before having the final design ;-)

[tomncooper]

That is an interesting idea, allowing users to deploy vanilla (or a stimzi-cc image) version of CC. The config would technically be split in the CCO anyway (between CC and CCO) so maybe that is not so much of an issue?

Better to have these discussions now before we hard code anything!

[kyguy]

TL;DR Here is a diagram of what I am proposing. My reasons are stated below

For the deployment of Cruise Control I think it would be most economical to declare cruiseControl:{} separately in the Kafka resource like this:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
...
spec:
  kafka:
    Config:
  cruiseControl: {}
  entityOperator:
      topicOperator: {}
      userOperator: {}
   …

For the following reasons:

1. Cruise Control can only monitor one cluster at a time. It should live and die with the Kafka cluster it is assigned to.
2. It saves Strimzi from managing another CRD and maintaining the link between a Cruise Control custom resource and the Kafka cluster it is monitoring.
3. It saves users from confusing the distinction between a Cruise Control resource and CruiseControlBalance resource.
4. I don't think we should have a CruiseControlOperator (reasons listed further down) and without a CruiseControlOperator it doesn't make sense to declare the Cruise Control deployment in the Entity Operator.

It would be nice to have a separate CRD for Cruise Control but it costs us complexity. The labor saved from having Cruise Control declared in the Kafka resource is worth the space taken in the Kafka resource.

As for the CruiseControlOperator, as stated above, I don't think we should have one since:

1. I think this Cruise Control API translation logic belongs in the Topic Operator, after all Cruise Control will be messing around with topics. We will need to add logic in the Topic Operator anyway to make sure that it doesn't collide/interfere with Cruise Control rebalance operations and vice versa. I am also confident that the Cruise Control API translation logic won't take up more than a couple classes either, so it would fit nicely in the Topic Operator too.
2. It would add another operator Strimzi needs to maintain
3. It would add unnecessary bloat to the Kafka resource. e.g. duplicating settings like ResourceRequirements, Logging, probes for the maintenance of the operator when we could use the Topic Operator

And responding to some of the questions posed above:

Because using CC means to set metrics.reporters property in the brokers configuration are we going to make it forbidden in the spec.kafka.config section or merging with any other reporters that the user can add? What about the metrics.polling.interval.secs property as well? Should we leave that in the `spec.kafka.config' section or locking down in the CC operator section with a different name? (making it forbidden in the kafka config as well?)

We should leave the metrics.reporters and metrics.polling.interval.secs properties unlocked in case users want to add or adjust addition metric reporters. Having these properties configurable shouldn't pose any problem.

Thinking more about the new KafkaConnector resource related to the actual KafkaConnect.
Aren't we sure that we wouldn't have a KafkaCruiseControl CRD for the CC deployment exposing the HTTP REST API if some user wants but at the same time providing a corresponding cruiseControlBalanceOperator for handling the KafkaBalance resources? This would split the configuration. I am just thinking aloud raising all the points that come to my mind ... before having the final design ;-)

I think the Cruise Control API should be locked down by default just as Zookeeper access is since it could be potentially destructive to a Kafka cluster. The Cruise Control API could still be completely exposed via a service using EO certs in the same way Jakub's insecure Zookeeper access hack works[1]. However, by forcing users to use a hack like this to expose full API access, we are highlighting the danger of the action. It would also help prevent users from doing something destructive accidentally.

[1] https://gist.githubusercontent.com/scholzj/6cfcf9f63f73b54eaebf60738cfdbfae/raw/068d55ac65e27779f3a5279db96bae03cea70acb/zoo-entrance.yaml

[kyguy]

There is a balance we need to strike between Kafka resource pollution and Strimzi code complexity. Any thoughts on the two designs outlined above @scholzj ?

[tomncooper]

So if the Topic Operator is going to be managing the logic, and as it is possible to run Strimzi without the TO, should we move the CC config into the TO config? Otherwise what is the functionality of CC without he TO present? Should it still be possible to a rebalance without the TO?

I think your proposal make sense but we need to figure out what the TO is doing (or indeed if it is doing everything) regarding CC.

[kyguy]

So if the Topic Operator is going to be managing the logic, and as it is possible to run Strimzi without the TO, should we move the CC config into the TO config? Otherwise what is the functionality of CC without he TO present? Should it still be possible to a rebalance without the TO?

That is a good point, I hadn't even thought of that! I think that would make more sense. Otherwise, if the TO was not declared, users wouldn't be able to interact with a declared CC, at least through supported means a.k.a the CruiseControlBalanceResource (or whatever we end up calling it).

[kyguy]

Although since CC is running on its own outside of the EO maybe it makes more sense to just have it declared and configured through its own custom resource like the other Kafka apps like Mirror Maker, Connect, etc. I would require more logic in Strimzi to keep CC in-sync with the Kafka cluster it is monitoring but it would keep the Kafka resource a heck of a lot neater

[kyguy]

Third proposal based on above comments. We isolate Cruise Control into into its own resource.

Although this method would add two new CRDs to Strimzi and require more code to coordinate between the separate parts, it might be a better way to organize external Kafka components as the project grows

Cruise Control custom resource:

apiVersion: kafka.strimzi.io/v1beta1
kind: CruiseControl
metadata:
  name: my-cruise-control
...
spec:
  cruiseControl:
    replicas: 1
...

The design for the Cruise Control Balance resource can be fleshed out after we agree on how the Cruise Control deployment should be managed. For now, we can assume it follows the format of the previous comments.

[scholzj]

If you use separate CruiseControl resource, it will be hard to integrate it internally into the Kafka cluster. So it would mean that:

• The user will have to configure the metrics reporter
• The user would need to create a KafkaUser with corresponding rights and configure CC with it
• You will get access to KAfka over the usual ports and not Zookeeper access.

If that is OK, we should do this. If this is not OK, we should integrate it into the Kafka CR the same way as we integrated other things. We also need to describe properly how will we handle the CruiseControlRebalance resources since that ultimately impacts a lot how we do the things. How will they be named? How will they be linked to a particular cluster? How will they be passed to Cruise Control? Who and how will pick them up? TO sounds like a logical choice, but it also seems the most troubled part and migth not be enabled. So we need to clarify these things which I don't see here.

[tombentley]

@scholzj there's nothing stopping the Kafka operator observing the CruiseControl in addition to the Kafka resource. When it see a CruiseControl whose name matches the Kafka that it's reconciling then it configures the metrics reporter etc. The kafka operator can still provision the deployment for running CC if that's simplest. I think the principle of having a separate resource is correct from an API point of view.

@kyguy can we elaborate on the ..., because in my ignorance I would expect the CruiseControl.spec to include things like the goals, but I see above you were thinking of putting those in KafkaBalance. I don't yet understand what the advantage of doing that would be.

[scholzj]

My view on that is very easy:

• Everything n be done in the code. But it adds a lot of complexity (synchronizing both resources, dealing with 1-N relationships, etc.)
• I do not see any value it would add

I do not see why from API point of view this would be correct. And even if it is, we broker it alrewady 7 times. So I'm not sure the 8th time matters. I would definitely prefer consistent UX over it.

[tombentley]

• There's no 1-N relationship. It's 1-1 between Kafka and CruiseControl.
• I don't think it's so complex to set up another watch which triggers reonciliation if either resource changes.
• Seriously, you see _no value_ in having separate APIs for these orthogonal things?
• I don't think mistakes made in the past should be used as a reason to carry on making mistakes in the future.

[scholzj]

There's no 1-N relationship. It's 1-1 between Kafka and CruiseControl.

There is - you will again have stray CruiseControl resources which will need to be dealt with.

Seriously, you see no value in having separate APIs for these orthogonal things?

I do not think these things are orthogonal. These things are clearly tightly coupled and that was the decision driver in the past.

I don't think mistakes made in the past should be used as a reason to carry on making mistakes in the future.

Well, as I said, I do not think these were mistakes. And as I said, consistent UX is also important.

[tombentley]

A Kafka resource ought to represent a Kafka cluster. Period. Not a Kafka cluster and a bunch of related projects which we thought were a good idea at the time. They're not really so tightly coupled that they need to be in the same resource.

[scholzj]

Well, it seems like your definition of Kafka cluster is lot narower than my definition.

[tombentley]

Your definition is pushing the Kafka resource to be a huge kitchen sink of things for configuring all these associated projects. I see that as having a number of problems:

• Its forces all these things to evolve in lock step. So we can't decide to have v1 Kafka without having to have all these other things also v1, even though some of them are a lot less mature within Strimzi.
• We bake into our API these dependencies on other projects. If those other projects make some incompatible changes that affects our API and we have to live with it because of point 1.
• At some point we will hit the default 1.5M value size limit in etcd, either for CRs or the CRDs (because the schemas are pretty big and in the v1 CRD API you get a schema per version).
• I think these huge CRs are actually pretty hostile for documentation and helping people learn this stuff.

It will always be easier to carry on with the simple route and pile everything into Kafka, but I believe we're making life harder for ourselves in the long run by doing that.

[scholzj]

I'm not sure this is the right place to continue this discussion. I don't see anything as a kitchen sink. I see it as a kitchen. In a kitchen you have usually the kitchen sink, the stove, the dishwascher, the fridge because these things are naturally coupled and belong together. You rarely hear people say _I do not want sink in my kitchen. I have already one sink in the bathroom._ So much for the metaphores.

We developed these things as tightly coupled and that is why they share the same Kafka CR. As long as they are tightly coupled, they should IMHO share the same CR and there is nothing wrong with it. If we want to decouple them, like we did with Connect or Mirror Maker. And we had this discussion before with both Connect and Mirror Maker but also with Kafka Exporter. And we decided differently in different cases based on how tight the coupling seemed to be.

But what you are sugegsting here is separate API for tightly coupled thing which makes no sense to me at all. And it especially doesn't make much sense if you do it for one thing out of 6 or 7 just to make things more confusing. If you wanna do this, than you should IMHO start with a plan to split all of these into separate resources over a short time period so that can have at least a bit reasonable UX.

[kyguy]

@kyguy can we elaborate on the ..., because in my ignorance I would expect the CruiseControl.spec to include things like the goals, but I see above you were thinking of putting those in KafkaBalance. I don't yet understand what the advantage of doing that would be.

@tombentley yes the CruiseControl.spec would include a section to configure the cruisecontrol.properties file [1] which would allow for specifying fields like default.goals and goals. Cruise Control would use these fields to cache pre-computed balance proposals.

We are still fleshing out the KafkaBalance resource but the idea of specifying goals here as well is that we can request Cruise Control to ignore these pre-computed balance proposals based on the default goals in favor of generating new balance proposals based on other goals of our choosing.

A more detailed version of a CruiseControl resource would look something like this:

apiVersion: kafka.strimzi.io/v1beta1
kind: CruiseControl
metadata:
  name: my-cruise-control
...
spec:
  replicas: 1
  image: <image>
  readinessProbe: {}
  livenessProbe: {}
  jvmOptions:
    ...
  logging:
    ...
  bootstrapServers: my-cluster-kafka-bootstrap:9093
  cruiseControl:
    properties: [1]
      default.goals:<list_of_goals>
      goals: <list_of_goals>
      intra.broker.goals: <list_of_goals>
      hard.goals: <list_of_goals>
      cpu.balance.threshold: 1.1
      metadata.max.age.ms: 300000
      send.buffer.bytes: 131072
      ...
    capacity: [2]
      brokerCapacities:
        -  brokerId: -1
           capacity:
             DISK: 100000
             CPU: 100
             NW_IN:  10000
             NW_OUT: 10000
        -  brokerId: 0
           capacity:
             DISK: 100000
             ...
    capacityJBOD: [3]
      ...
...

[1] https://github.com/linkedin/cruise-control/blob/master/config/cruisecontrol.properties
[2] https://github.com/linkedin/cruise-control/blob/master/config/capacity.json
[3] https://github.com/linkedin/cruise-control/blob/master/config/capacityJBOD.json

[kyguy]

But what you are sugegsting here is separate API for tightly coupled thing which makes no sense to me at all. And it especially doesn't make much sense if you do it for one thing out of 6 or 7 just to make things more confusing. If you wanna do this, than you should IMHO start with a plan to split all of these into separate resources over a short time period so that can have at least a bit reasonable UX.

A wise person (Gary Tully) once told me: "Feature first, refactor later". So if Cruise Control _is_ tightly-coupled with Kafka, I think we should proceed with the way we have been doing things and add Cruise Control to the Kafka resource as we have done with the Kafka Exporter, especially for the sake of consistency. Afterwards, if we decide we want to change the model and separate tightly coupled components, we can always do a refactor and separate these components into their own CRDs.

The decision comes down to the question of how tightly-coupled Cruise Control and Kafka are. Cruise Control most definitely depends on Kafka, however, Kafka does not depend on Cruise Control. Since Kafka does not depend on Cruise Control does Cruise Control merit a place in the Kafka resource like Zookeeper does?

[scholzj]

Since Kafka does not depend on Cruise Control does Cruise Control merit a place in the Kafka resource like Zookeeper does?

I think if you follow the same principle as Tom suggested even Zookeeper doesn't belong there.

Feature first, refactor later

I don't know about this one. The API is kind of a form of a contract. So while this approach fits me in this particular case, I don't think it should motivate the decision making. There will be a lot of complications to change this later.

[tombentley]

@kyguy

We are still fleshing out the KafkaBalance resource but the idea of specifying goals here as well is that we can request Cruise Control to ignore these pre-computed balance proposals based on the default goals in favor of generating new balance proposals based on other goals of our choosing.

Can you explain the user interaction then? Are you envisioning something like the following:

1. User creates CruiseControl resource (or adds it to the Kafka, depending on how this discussion resolves that)
2. CC starts balancing the cluster according to the default goals
3. User creates a KafkaBalance, changing the goals.
4. CC starts balancing according to those new goals

Why not just change the default goals in the CruiseControl?

Also some other questions (though perhaps these are too nitty gritty for this issue):

• Are there any options we'd forbid users to configure via cruiseControl.properties?
• Why does the operator need DISK capacity if it knows the sizes of the volumes?
• Why would you want CPU ≠ 100?
• For the JBOD stuff: Do we really expect the user to have to configure the directory names by hand like that (IIRC we don't really expose the log.dirs to the user much).

[kyguy]

@tombentley

Are you envisioning something like the following:

That's the general idea. The KafkaBalance resource would either trigger a rebalance or just generate a proposal based on the new goals specified.

Why not just change the default goals in the CruiseControl?

We could do that, but then we would need to restart Cruise Control every time we wanted to generate a proposal based on different goals. I imagine a user might want to weigh a couple of proposals based on different goal priorities before committing to a rebalance.

Are there any options we'd forbid users to configure via cruiseControl.properties?

Yes, options like bootstrap.servers, zookeeper.connect, and webserver.http.port where we would generate the values ourselves to prevent potentially destructive tampering.

Why does the operator need DISK capacity if it knows the sizes of the volumes?

Cruise Control needs to know if a broker has enough disk space before it proposes whether a certain partition should be moved to it. Cruise Control can actually estimate these capacity values but offers a way to hardcode these capacity values for environments where the estimation does not work properly.

Why would you want CPU ≠ 100?

In case a broker was sharing a host with another application or CPU usage was expensive

For the JBOD stuff: Do we really expect the user to have to configure the directory names by hand like that (IIRC we don't really expose the log.dirs to the user much).

I don't. Most of the things posted in the CruiseControl CR example was to demonstrate things that _could_ be configurable, I still need to go through the configurations and pick out what _should_ be configurable. I agree the more defaults and decisions we provide users, the better.

[tombentley]

The KafkaBalance resource would either trigger a rebalance or just generate a proposal based on the new goals specified.

Why not just change the default goals in the CruiseControl?

We could do that, but then we would need to restart Cruise Control every time we wanted to generate a proposal based on different goals. I imagine a user might want to weigh a couple of proposals based on different goal priorities before committing to a rebalance.

How does the user interaction with KafkaBalance work wrt proposals? Or alternatively, how does the life cycle of a KafkaBalance work wrt proposals? Once the user has a proposal (where does that appear to the user? In the status of a Kafka balance?), how do they say "yes, do that please"? Sorry if you already described this somewhere and I missed it.

[ppatierno]

The user would need to create a KafkaUser with corresponding rights and configure CC with it

When authorization is enabled, it's something that the user has to do for Kafka Connect as well in order to access to "internal" topics like connect-cluster-offsets, etc etc. I had this problem for setting up the RHTE demo.

I would agree that handling the Rebalance resource is complicated to add in the TO logic.

I see CC coupled with Kafka in the same way as Kafka Exporter is, because both of them get metrics from the Kafka cluster. It happens in different ways, and CC is more invasive on the Kafka broker side due to the metrics.reporters configuration.

On the other side, other similar operators implementations decided to go for having a Kafka resource with just Kafka and no ZooKeeper for example. In one case ZooKeeper is a completely different resource handled by the same operator in another one, the operator doesn't care about ZooKeeper and you should deploy it upfront.
Looking at what is happening with KIP-500, maybe having a different custom resource for ZooKeeper seems to make more sense.

Tbh I still don't have a strong opinion on which solution is better so I can be convinced on both of them :-)

[scholzj]

Looking at what is happening with KIP-500, maybe having a different custom resource for ZooKeeper seems to make more sense.

Well, there was no KIP-500 when we started. And maybe now it is actually easier to keep it in and just forget the Zookeeper part after KIP-500 hits the code.

[tombentley]

I am not advocating having a ZooKeeper CR; I agree we should keep that in the Kafka CR and deprecate it when the time comes.

[scholzj]

But Zookeeper is as separate API as CC based on what you say. That is one of the problems I have with this ... I think this approach let's try something else here and something totally different next time just makes our UX total mess.

[tombentley]

To me the differences are:

• ZK is a non-optional prerequisite of having a Kafka cluster. CC etc are not.
• We know that ZK is going to go away, so I don't think it's worth spending the time refactoring to have a ZooKeeper CR. If ZK was not going away, I _might_ have a different opinion.

The inconsistent UX argument is a valid point. But I don't think having a single CR is really going to end up as a paragon of good UX either. Specifically I think the following are user hostile:

• lots of deprecated properties
• some properties which we consider stable and mature, and others which we might consider alpha or beta (or supported/not supported), but we have no mechanism to effectively convey such distinctions to users.
• Not getting to Kafka/v1 because the whole thing is constantly evolving as we just keep chucking more functionality in there.

We have already started down this latter path, and I although our UX might look "consistent" now I think it will end up making for a bad UX for the above reasons.

So, for me, the sacrifice of some consistency in the short term is worth the long term benefit of having separate APIs for these features. This _is_ a change of heart, however it would be wrong to have doubts about our current course and not voice them.

[ppatierno]

I am not advocating for a refactoring about moving ZK to a new custom resource. I was just making the point that anyway others made a different decision than us so it sounds to me that there is no clear direction for this kind of decision.

[scholzj]

ZK is a non-optional prerequisite of having a Kafka cluster. CC etc are not.

Kafka is also a non-optional prerequisite for CC. So it is the same logic.

[ppatierno]

I am starting to think that embedding the CC declaration in the Kafka resource is the simpler solution and it could avoid adding two new custom resources (for CC itself and for the rebalance).
If the cruiseControlConfig has the field with a plain CC configuration which contains goals, thresholds, and so on ... the user could just change it for changing the rebalance actions by the CC.
Tbh it's not still clear to me if the CC needs a restart to read the new config or it's able to do that on the fly but I have the feeling that it's not possible but you need to interact with the REST API for changing only a "part of" the configuration.

[tombentley]

I've never argued that a separate CruiseControl CR was going to be _simpler_, rather that it would result in a leaner, cleaner set of APIs which would be more flexible in the future. But if there's a consensus around sticking it in the Kafka resource then don't let my misgivings hold back progress.

[kyguy]

How does the user interaction with KafkaBalance work wrt proposals? Or alternatively, how does the life cycle of a KafkaBalance work wrt proposals? Once the user has a proposal (where does that appear to the user? In the status of a Kafka balance?), how do they say "yes, do that please"? Sorry if you already described this somewhere and I missed it.

The current plan is to trigger proposal generation and rebalance operations using k8s annotations in a similar fashion to how rolling updates are manually triggered. We were thinking of storing the response in the status section of the KafkaBalance resource, assuming it can fit. If the responses/proposals are too large for the status section, we will need to find a more suitable place. In this way, A user could first generate a proposal using an annotation, observe the proposal in the status section, then trigger a rebalance using that generated proposal with another annotation.

[kyguy]

If the KafkaExporter is not going to act as a precedent for our future resource model then CruiseControl surely will. We will be stuck with whichever model we choose here for better or for worse. I agree that consistency is important so if we are to have a CruiseControl resource, I argue that we must separate the KafkaExporter into its own resource shortly after. If that's not feasable, we should use the KafkaExporter as an example and put CruiseControl in the Kafka resource.

[kyguy]

Since we cannot come to a unanimous agreement, can we vote on the decision? Should we restrict the voting between the maintainers since they are the most familiar with the codebase and they sum to an odd number?

[scholzj]

I guess my and Tom's views are clear. Paolo seemed to side with including it in Kafka resource unless Tom managed to convince him.

I'm still not completely clear and how will the KafkaBalance resource work and what will / should be the options inside the cruiseControlConfig section. I agree with tom that the things such as the capacity etc. should be IMHO set by the operator automatically. I expected there to be only the things around the deployment of the CC.

[ppatierno]

I am keen to have the CC configuration in the Kafka resource but we need a clear design for specifying things like Resources requests/limits, logging and so on for both the CC deployment and the CC operator (for handling "balance" resources).

About "balance" resources and CC configuration, my understanding is that goals and related parameters (like thresholds) are in the CC configuration (aka properties file); changing them needs a CC restart. I don't see a CC API for changing them via REST so that CC can do a rebalancing proposal based on the new configuration. Maybe I misunderstood something from the CC doc, @kyguy ? I am just wondering if a "balance" resource is going to work or not ...

[tomncooper]

@ppatierno You can specify the particular goals you wish to use for an individual rebalance as well as other per-rebalance settings that can be applied. This was our thinking behind needing a specific KafkaRebalance CR to allow the user to set these option. We still need to identify which of these settings make sense to expose but allowing the user to choose only the goals they care about for a rebalance makes sense to me.

Also as these operations are particularly long lived we need a location to pin the status updates on the progress of the rebalance so a CR for that makes sense.

To help with understanding the proposed flow of operations I am putting together some flow charts showing what we think should happen.

[ppatierno]

So coming back to the KafkaBalance resource which is meant for starting a rebalance (and getting the status as well), I was thinking that this resource hits one of the endpoints of the Cruise Control REST API, more specifically the /rebalance endpoint.
While it's ok having a custom resource for this one, I was thinking if (not right now) we would open part of the API so making accessible other endpoints like for example /state (state of Kafka Cruise Control) or /load (current cluster load) ...
In this case instead of adding more CRs in the future (i.e. KafkaCruiseControlState or KafkaCruiseControlLoad ...) we can think since the beginning of having a sort of KafkaCruiseControlRequest/Action/Operation and having in its spec the possibility to ask for rebalance, load, state and so on.
So focusing on the rebalance (which is what we want right now) I was thinking more at:

apiVersion: kafkacruisecontrol.kafka.strimzi.io/v1alpha1
kind: KafkaCruiseControlAction
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
spec:
  rebalance:
    goals:
      //
status:
   // 

and maybe in the future

apiVersion: kafkacruisecontrol.kafka.strimzi.io/v1alpha1
kind: KafkaCruiseControlAction
metadata:
  name: my-load-request
  labels:
    strimzi.io/cluster: my-cluster
spec:
  load:
    //
status:
  //

where in the CRD we can use the OneOf on load, rebalance and so on.
This kind of design would avoid to add a new CR if now we start with the rebalance one but we want to add more in the future.

[ppatierno]

After some chats offline, back to the original design using KafkaClusterRebalance.
If we need to expose more Cruise Control REST API endpoint we'll add other custom resources that could be clearer for the user. In this way, it could be even possible using RBAC to allow some users doing some operations on Cruise Control and not others.

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaClusterRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
spec:
    goals:
      - DiskCapacityGoal
      - CpuCapacityGoal
status:
    conditions:
    //

An empty goals array means to evaluate all default goals set in the Cruise Control configuration on startup.
Applying this resource means doing a /rebalance request with "dry-run" so not actually doing the rebalance.
After a while, the status.optimizationResult will contain the JSON with the main information about the proposed plan for optimization.

The user can apply a strimzi.io/do-rebalance=true annotation on the same result to request Cruise Control to apply the provided proposal (a /rebalance request without "dry-run").

The resource can also have an optional spec.verbose boolean to specify that the JSON response should contain ALL the information and not just the main ones.

[dimisjim]

@tomncooper
Closing this means that this functionality has been added, and is gonna get released in the future?

[tomncooper]

Cruise Control intergration was added in the 0.18 release. See this blog post for details.