Strimzi-kafka-operator: [Question] Running Cruise Control UI

[krystyw]

Hi,

I've started using strimzi for deploying Kafka and I've decided to deploy Cruise Control along with it. Lately I wanted to use Cruise Control UI for easier (and prettier) way to check Kafka's status. Unfortunately, I can't figure out a way to do that.

I saw your blog post about it and in that version, the CCFE was deployed with the Cruise Control. In strimzi's version of the image the CCFE isn't there.

I've tried to run CCFE locally on my machine and port-forward to the Cruise Control pod, however in that case the CORS blocks me. Changing the config of CC, in order to allow CORS, isn't allowed by the operator.

My question is: is there a way to use/deploy CCFE with strimzi's Cruise Control deployment?

[scholzj]

Our idea is for users to use the KafkaRebalance resource for controlling Cruise Cotnrol. Not the UI or the REST API.

Not sure if anyone from the people who worked on the Cruise Cotnrol support have any tips or tricks how you can use it.

[scholzj]

CC: @ppatierno @tomncooper @kyguy

[ppatierno]

To be honest I haven't tried it but from what I can see on the CCFE GitHub repo, you can create a Dockerfile with it (https://github.com/linkedin/cruise-control-ui/wiki/CCFE-(Dev-Mode)---Docker) which could be used for your CCFE Deployment as for the container image to run CCFE itself alongside the Strimzi Kafka cluster with CC.
At that point, you should configure CCFE to point to the <cluster-name>-cruise-control Kubernetes service which exposes the CC REST API on port 9090.
I am not a NodeJS expert and the only place in the CCFE code where the target CC is set seems to be this https://github.com/linkedin/cruise-control-ui/blob/master/config/index.js#L35
This is where you should put the CC Kubernetes service.
It's not a huge help but at least a starting point for investigation :-)

[ppatierno]

Actually I found this https://github.com/linkedin/cruise-control-ui/wiki/config.csv-Syntax
If this config.csv is used for configuring CCFE so even the CC URL, you could set it into a ConfigMap and mounting it as a volume in the CCFE Deployment. It would be the way to configure and change the configuration for CCFE if needed.

[kyguy]

Hey @krystyw!

If you are open to creating your own Kafka image you could also:

Add the following line here [1]

RUN curl -L https://github.com/linkedin/cruise-control-ui/releases/download/v0.3.4/cruise-control-ui-0.3.4.tar.gz -o /tmp/cruise-control-ui.tar.gz \
+    && tar zxvf /tmp/cruise-control-ui.tar.gz -C ${CRUISE_CONTROL_HOME}

Build the Kafka image and push it to your K8s cluster (remember to reference your custom Kafka image from your Cluster Operator deployment)

Then in your Kafka resource:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  ...
spec:
  cruiseControl:
    config:
      webserver.ui.diskpath: /opt/cruise-control/cruise-control-ui/dist/

then port-forward the pod

kubectl port-forward svc/my-cluster-cruise-control 9090:9090

This will make the UI accessible via http://127.0.0.1:9090

[1] https://github.com/strimzi/strimzi-kafka-operator/blob/0.18.0/docker-images/kafka/Dockerfile#L43

[krystyw]

Thank you for the answers!

@scholzj Thank you for the reasoning behind not including the CCFE. Right now, we are interested in easy status checking, not the rebalancing feature and that's why we thought about CCFE in the first place.

@ppatierno Using separate pod for the CCFE would result in CORS error from CC as well, I think. I didn't try that, but I'd guess that would be the same as me running the CCFE locally. I'll try it though. The configuration file I've already found before.

@kyguy I wanted to avoid building my own image but I guess that would be a sure solution for the problem.

[spaghettifunk]

I just want to share the work I did based on @kyguy answer. You can have your own image by simply building a multi-stage one based on the strimzi version. What we did was

# --------------- Builder stage ---------------
FROM centos:7 AS builder

ENV CC_VERSION=0.3.4
ENV CRUISE_CONTROL_HOME=/opt/cruise-control

RUN mkdir $CRUISE_CONTROL_HOME

# Install Cruise Control GUI Frontend
RUN curl -L https://github.com/linkedin/cruise-control-ui/releases/download/v${CC_VERSION}/cruise-control-ui-${CC_VERSION}.tar.gz \
    -o /tmp/cruise-control-ui.tar.gz && \
    tar zxvf /tmp/cruise-control-ui.tar.gz -C ${CRUISE_CONTROL_HOME}

# --------------- Final stage ---------------
FROM strimzi/kafka:0.18.0-kafka-2.5.0
COPY --from=builder /opt/cruise-control/* /opt/cruise-control

We pushed this to our registry and then simply changed the Kafka object in two places

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: kafka
  namespace: kafka
spec:
  kafka:
    replicas: 3
    image: own-repository/kafka:2.5.0
   # ...
   # ... other settings here
   # ...
  cruiseControl:
    image: own-repository/kafka:2.5.0
    config:
      webserver.ui.diskpath: /opt/cruise-control/dist/

Last touch was to create a new service that would expose a LB. We needed this as we wanted to have it available within the company. This is the result

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "0.0.0.0/0"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-1:xxxx:certificate/xxxxx"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
    external-dns.alpha.kubernetes.io/hostname: kafka-cruise-control.my-domain.com.
    external-dns.alpha.kubernetes.io/ttl: "60"
  labels:
    strimzi.io/cluster: kafka
    strimzi.io/kind: Kafka
    strimzi.io/name: kafka-cruise-control
  name: cruise-control-lb
  namespace: kafka
spec:
  ports:
    - name: http-9090
      port: 443
      protocol: TCP
      targetPort: 9090
  type: LoadBalancer
  selector:
    strimzi.io/cluster: kafka
    strimzi.io/kind: Kafka
    strimzi.io/name: kafka-cruise-control

This Kafka operator is absolutely amazing! Thanks for the hardwork 🚀

[superbeer]

extend from above comment

 config:
      webserver.ui.diskpath: /opt/cruise-control/dist/
      webserver.ui.urlprefix: /*