Pulsar: Kubernetes deployment: Pods from a service are being scheduled on the same node

Created on 13 Oct 2019 · 5Comments · Source: apache/pulsar

I was trying to deploy pulsar using GKE. While I was deploying the ZooKeeper service I noticed that its pods were being scheduled on the same node. I am really new to Kubernetes but this looks a little odd to me:

deployment/kubernetes/google-kubernetes-engine/zookeeper.yaml

spec:
    serviceName: zookeeper
    replicas: 3
    template:
        metadata:
            labels:
                app: pulsar
                component: zookeeper
                cluster: pulsar-gke
            annotations:
                pod.alpha.kubernetes.io/initialized: "true"
                prometheus.io/scrape: "true"
                prometheus.io/port: "8080"

        spec:
            # Make sure multiple pods of ZK don't get scheduled on the
            # same node, unless there are no other available nodes
            affinity:
                podAntiAffinity:
                    preferredDuringSchedulingIgnoredDuringExecution:
                      - weight: 1
                        podAffinityTerm:
                            labelSelector:
                                matchExpressions:
                                  - key: "app"
                                    operator: In
                                    values:
                                      - zookeeper
                            topologyKey: "kubernetes.io/hostname"

I read this Kubernetes affinity example and I think that the key used in matchExpressions should be component, not app. When I use component as the key, all the ZooKeeper pods are scheduled on different nodes.

If I am correct, this error is present in all the Kubernetes deployment files.

Is there any Kubernetes expert willing to take a look at this?

Thank you!

typbug

Source

stupidusername

All 5 comments

@stupidusername you are correct. To fix it, you might want to add a componentkey to the podAffinityTerm, not change to change app since the pod has both those labels. Doing that would be more important if you wanted to create multiple Pulsar instances in one Kubernetes cluster.

Here is an example of what I am using (I also have release):

 affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - pulsar
          - key: release
            operator: In
            values:
            - useast1-gcp
          - key: component
            operator: In
            values:
            - zookeeper

I noticed this issue in the Helm chart and have been meaning to submit a PR to fix it.

cdbartholomew on 13 Oct 2019

👍1

Actually, looking at the Helm chart more closely, there is a different problem with the anti-affinity rules, but the effect is the same. Pods don't get scheduled on different nodes.

      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: "app"
                operator: In
                values:
                - "{{ template "pulsar.name" . }}-{{ .Values.zookeeper.component }}"
              - key: "release"
                operator: In
                values:
                - {{ .Release.Name }}
              - key: "component"
                operator: In
                values:
                - {{ .Values.zookeeper.component }}

The issue here is that app is a concatenation of 'pulsar' and the component name, but the pods aren't labeled that way.

I will create a PR to clean both of these up.

cdbartholomew on 13 Oct 2019

they put component in this one here