Strimzi-kafka-operator: [Bug] Kafka pod's cannot be created with custom (non-default) cluster domain

[Ranger-X]

Describe the bug
I try to create a Kafka cluster through strimzi-kafka-operator in minikube. When operator tried to connect to zookeeper (on kafka's pod-creation phase) it's failed with error:

Failed to connect to Zookeeper
        minikube-kafka-zookeeper-0.minikube-kafka-zookeeper-nodes.infra-kafka.svc:2181.
        Connection was not ready in 300000 ms.

I think that the operator cannot resolve the zookeeper's service name correctly, because it does not include the cluster domain suffix (in my environment it's a "yc.minik", not default "cluster.local").

I checked these thoughts like this (inside zookeer's container):

[kafka@minikube-kafka-zookeeper-0 kafka]$ nslookup minikube-kafka-zookeeper-0.minikube-kafka-zookeeper-nodes.infra-kafka.svc
Server:     10.96.0.10
Address:    10.96.0.10#53

** server can't find minikube-kafka-zookeeper-0.minikube-kafka-zookeeper-nodes.infra-kafka.svc: NXDOMAIN

[kafka@minikube-kafka-zookeeper-0 kafka]$ nslookup minikube-kafka-zookeeper-0.minikube-kafka-zookeeper-nodes.infra-kafka.svc.yc.minik
Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   minikube-kafka-zookeeper-0.minikube-kafka-zookeeper-nodes.infra-kafka.svc.yc.minik
Address: 172.17.0.15

To Reproduce
Steps to reproduce the behavior:

1. Launch minikube with custom dns-domain suffix, like: minikube start --dns-domain='yc.minik'
2. Install strimzi kafka operator through Helm or other way (but remember to set deployment's env-variable KUBERNETES_SERVICE_DNS_DOMAIN='yc.minik'): helm install kafka-operator -n infra-kafka --set-string kubernetesServiceDnsDomain='yc.minik' strimzi/strimzi-kafka-operator
3. Create something like:

cat << EOF | kubectl create -n infra-kafka -f -
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: minikube-kafka
  namespace: infra-kafka
spec:
  kafka:
    version: 2.4.0
    replicas: 1
    resources:
      limits:
        cpu: "200m"
        memory: "1Gi"
      requests:
        cpu: "50m"
        memory: "64Mi"
    listeners:
      plain: {}
      tls: {}
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.message.format.version: "2.4"
    storage:
      type: ephemeral
  zookeeper:
    replicas: 1
    resources:
      limits:
        cpu: "200m"
        memory: "1Gi"
      requests:
        cpu: "50m"
        memory: "64Mi"
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}
EOF

1. See that no kafka's pod will created by operator

Expected behavior
Kafka cluster created and running successfully.

Environment (please complete the following information):

• Strimzi version: 0.17.0
• Installation method: Helm chart
• Kubernetes cluster: Kubernetes 1.16.8
• Infrastructure: Minikube

YAML files and logs
Reports files attached.
report-13-04-2020_22-07-46.zip

[scholzj]

The hostname minikube-kafka-zookeeper-0.minikube-kafka-zookeeper-nodes.infra-kafka.svc is just a DNS name configured through the hedless service. It should be resolvable even without the cluster suffix ragerdless whether it is .cluster.local or .yc.minik.

[Ranger-X]

Hmm. Thank you. I will look for problems in the minikube's DNS (mis)configuration.

[scholzj]

I had a look at this and I can confirm the issue. But it looks like Minikube issue to me. It does not seem to propagate the new cluster domain into the resolv.conf file:

[kafka@my-cluster-zookeeper-0 kafka]$ cat /etc/resolv.conf 
nameserver 10.96.0.10
search myproject.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Which has still cluster.local even when you changed the domain. I found this older bug https://github.com/kubernetes/minikube/issues/1674 which was closed by the ci-robot, so I'm not sure whether it is still valid.

[Ranger-X]

I also investigated this problem more deeply and agree that this is issue in the minikube.

For anybody who googled this page :-) I solved the issue by running minikube like this:

minikube start --vm-driver=virtualbox --kubernetes-version=1.16.8 --cpus 4 --disk-size 20g --memory 16384 --dns-domain='yc.minik' --extra-config='kubelet.cluster-domain=yc.minik'

So, an important option is --extra-config='kubelet.cluster-domain=yc.minik'. After this I successfully run Kafka cluster in the Minikube :-)

@scholzj Thank you very much for the right direction about this issue :-)

[scholzj]

Thanks for investigating anf updating the issue. Much appreciated!