Strimzi-kafka-operator: [Bug] Cluster DNS not correctly set in Kafka advertised addresses

[zzvara]

Describe the bug
The Kubernetes cluster DNS (defaults to cluster.local) is not correctly set by the operator for Kafka brokers, as the brokers will advertise inter-namespace correct but intra-namespace incorrect DNS.

To Reproduce
Steps to reproduce the behavior:

1. Deploy operator:
   helm install --namespace kube-system --generate-name --set "watchAnyNamespace=true,kubernetesServiceDnsDomain=sigma" strimzi/strimzi-kafka-operator
2. Deploy Kafka cluster:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: development-default
  namespace: development
spec:
  kafka:
    replicas: 2
    version: 2.4.0
    listeners:
      plain: {}
    storage:
      type: persistent-claim
      size: 20Gi
      class: ceph-volatile
    jvmOptions:
      -Xms: 2048m
      -Xmx: 2048m
    config:
      auto.create.topics.enable: "true"
  zookeeper:
    replicas: 1
    jvmOptions:
      -Xms: 1024G
      -Xmx: 1024G
    storage:
      type: persistent-claim
      size: 1Gi
      class: ceph-volatile

1. Note that operator namespace differs from Kafka namespace.
2. Confirm that operator environment variables contain your cluster DNS - "sigma" this case.
   KUBERNETES_SERVICE_DNS_DOMAIN: sigma
3. Confirm that Kafka brokers would not have correct DNS advertised ("sigma" is missing):
   advertised.listeners=REPLICATION-9091://development-default-kafka-0.development-default-kafka-brokers.development.svc:9091,PLAIN-9092://development-default-kafka-0.development-default-kafka-brokers.development.svc:9092
4. From kube-system namespace network, instantiate a Kafka Producer (2.4.0 - same version) and dial Kafka. Notice the error.
5. Consumer bootstrap servers: development-default-kafka-brokers.development.svc.sigma:9092

java.net.UnknownHostException: development-default-kafka-1.development-default-kafka-brokers.development.svc
    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
    at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
    at java.net.InetAddress.getAllByName(InetAddress.java:1192)
    at java.net.InetAddress.getAllByName(InetAddress.java:1126)
    at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)

Expected behavior
Kafka to advertise address development-default-kafka-1.development-default-kafka-brokers.development.svc.**sigma**

Environment:

• Strimzi version: 0.16.2
• Installation method: See above.
• Kubernetes cluster: Kubernetes 1.17 installed on CoreOS with Kubespray
• Infrastructure: Bare metal, storage provisioned by Ceph

Small note:

• The Kafka Consumer is instantiated by a development PC that has VPN access to the cluster through an OVPN pod sitting on kube-system namespace and exposed through NodePort.
• I have confirmed that from any kube-system pod the Kafka borkers can not be telnetted on development namespace by simply calling telnet development-default-kafka-1.development-default-kafka-brokers.development.svc 9092.

[scholzj]

I do not see this as a bug. The DNS name service.namespace.svc should be resolvable witin your cluster. On the other hand adding the suffix is causing problems because while cluster.local is the default suffix is can be changes and in some cases it is changed. So routing the traffix only through service.namespace.svc makes it work without any problems in all the clusters which are not using cluster.local suffix.

[zzvara]

Could you support your claim?

The DNS name service.namespace.svc should be resolvable within your cluster.

I think it is correct only for namespaces, not the cluster. At least the default DNS on Kubernetes 1.17 will not pass it through.

On the other hand adding the suffix is causing problems because while cluster.local is the default suffix is can be changes and in some cases it is changed.

What are the use cases that would support such a case? Could you provide one example when the cluster DNS changes in runtime?

So routing the traffix only through service.namespace.svc makes it work without any problems in all the clusters which are not using cluster.local suffix.

What is the effect of KUBERNETES_SERVICE_DNS_DOMAIN then?

[scholzj]

Could you support your claim?

Support what claim?

I think it is correct only for namespaces, not the cluster. At least the default DNS on Kubernetes 1.17 will not pass it through.

Not sure what exactly do you mean with this. The internal and external interfaces are supposed to be used from other namespaces within the same cluster. If you want access from outside the cluster - not running in any namespace, you can use the external listener.

What are the use cases that would support such a case? Could you provide one example when the cluster DNS changes in runtime?

I didn't said anything about runtime and I do not know what do you mean with it. Different clusters have different suffixes. cluster.local is default, but not the only one used.

What is the effect of KUBERNETES_SERVICE_DNS_DOMAIN then?

That is the thing which we used in the past. But it doesn't work well because user do not know they need to configure it. So now they don't need it anymore because we do not use the suffix anymore.

[zzvara]

Support the following claim:

The DNS name service.namespace.svc should be resolvable within your cluster.

Not sure what exactly do you mean with this.

I meant that when you want to resolve services from different namespaces, it seems that it is not enough to specify an internal domain name ending with svc. You have to add the cluster domain name. The default cluster domain name is cluster.local. In my case, it is sigma.

To reiterate my problem: I'm calling Kafka that is sitting on development namespace from an application that is sitting on kube-system namespace. You see the following error from within the application:

java.net.UnknownHostException: development-default-kafka-1.development-default-kafka-brokers.development.svc

If you would try to resolve development-default-kafka-1.development-default-kafka-brokers.development.svc.**sigma**, it would work. The problem is that Kafka brokers advertise their address in the following form: development-default-kafka-1.development-default-kafka-brokers.development.svc (without sigma)

To me, this seems like a bug since Kafka is not advertising the correct address, moreover, it seems to be as you are saying, that we have a configuration parameter, namely KUBERNETES_SERVICE_DNS_DOMAIN environment variable, and parameter also present in the Helm chart that is misleading users.

[scholzj]

Try it for example in Minikube to see that it normally works. The images have /etc/resolve.conf modified to configure the search domains. Within a namespace, you can use just service. Outside you can use service.namespace or service.namespace.svc.

[scholzj]

I will close this since there was no update for more than 10 days. If you have something more feel free to open a new issue, reopen this one or get in touch with us on Slack or mailing list.

[logart]

Hi, @scholzj

Sorry if this is not the right place to ask questions.
I am experiencing the same issue after update to 0.18.

Could you please help me with correct configuration.

# nslookup kafka-kafka-1.kafka-kafka-brokers.namespace-dev.svc
Server: *
Address:    *

** server can't find kafka-kafka-1.kafka-kafka-brokers.namespace-dev.svc: NXDOMAIN

** server can't find kafka-kafka-1.kafka-kafka-brokers.namespace-dev.svc: NXDOMAIN

/ # nslookup kafka-kafka-1.kafka-kafka-brokers.namespace-dev.svc.cluster.local
Server: *
Address:    *
Name:   kafka-kafka-1.kafka-kafka-brokers.namespace-dev.svc.cluster.local
Address: address

/ # nslookup kafka-kafka-1.kafka-kafka-brokers.namespace-dev
Server: *
Address:    *
** server can't find kafka-kafka-1.kafka-kafka-brokers.namespace-dev: NXDOMAIN
** server can't find kafka-kafka-1.kafka-kafka-brokers.namespace-dev: NXDOMAIN

/ # nslookup kafka-kafka-1.kafka-kafka-brokers
Server: *
Address:    *
** server can't find kafka-kafka-1.kafka-kafka-brokers: NXDOMAIN
** server can't find kafka-kafka-1.kafka-kafka-brokers: NXDOMAIN

/ # nslookup kafka-kafka-1
Server: *
Address:    *
** server can't find kafka-kafka-1.cluster.local: NXDOMAIN
** server can't find kafka-kafka-1.namespace-dev.svc.cluster.local: NXDOMAIN
** server can't find kafka-kafka-1.namespace-dev.svc.cluster.local: NXDOMAIN
** server can't find kafka-kafka-1.cluster.local: NXDOMAIN
** server can't find kafka-kafka-1.svc.cluster.local: NXDOMAIN
** server can't find kafka-kafka-1.svc.cluster.local: NXDOMAIN

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-26T06:17:09Z", GoVersion:"go1.14", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

So you can see in the logs the only address I could resolve is with .cluster.local at the end.
I am not pushing to update it on your side. Just thinking if this is new feature of k8s and I should update or there could be some other reason why addresses that you've provided(service.namespace or service.namespace.svc) does not work.

Thank you for any help.

[scholzj]

Thsi should be normally configured by your Kubernetes our of the box:

sh-4.2$ cat /etc/resolv.conf 
search myproject.svc.cluster.local svc.cluster.local cluster.local us-east-2.compute.internal
nameserver 172.30.0.10
options ndots:5

[logart]

Thank you for your reply. For everyone who will run into this issue I will post a comment.
This works perfectly fine on linux, but on the OS X where I(and I believe @zzvara had an issue) it seems like .local is reserved for bonjour(mDNS) software.
I was not able to change it so I've decided to put kafka ip addresses directly into /etc/hosts.
If there will be a better solution I would gladly apply it.
P. S. This is indeed not a bug of neither kafka-operator nor kubernetes. Just conflict with osx.

[scholzj]

Out of curiosity - how are you running the Kubernetes on OS-X? I did never had any problems with this with Minikube.

[zzvara]

For me, this issue happens through an OVPN connection, where the OVPN client asks the OVPN server container to resolve the Kafka brokers. The OVPN server fails to do so since it is in a separate namespace from the Kafka brokers AND the Kafka brokers do not provide an FQDN!

This should be fixed. Brokers should provide FQDN so that everyone would be happy across the cluster.

[logart]

@scholzj I am not. We are running k8s on linux but connect to it through openvpn for debug from local os x machines.

[Oldsouldier]

I'm experiencing the same issue as @zzvara . Since the service domain is customizable per deployment, why not use it?

[scholzj]

Since the service domain is customizable per deployment, why not use it?

Not sure I follow - can you be more specific?

[Oldsouldier]

KUBERNETES_SERVICE_DNS_DOMAIN as mentioned above.

EDIT i see this is unofficially deprecated?

[scholzj]

I don't think it is deprecated. It is just not used where it is not needed. I also assume that impelmenting this https://github.com/strimzi/proposals/blob/master/005-improving-configurability-of-kafka-listeners.md migth give users more flexibility to deal with some more scenarios.

[zzvara]

@scholzj This is still a bug and hinders the usability of Strimzi Kafka through VPN connections. Moreover, the bug has been confirmed by other users as well.

Using FQDNs are generally a good practice to make sure that complex network settings are supported. I don't see why omitting the cluster domain name would be of any help. Probably the issue could be fixed with a one-digit line of code?

[scholzj]

@zzvara Just FYI: Since 0.20.0, this is now configurable on the listener level with the useServiceDnsDomain option:

    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
        configuration:
          useServiceDnsDomain: true

Docs: https://strimzi.io/docs/operators/latest/full/using.html#property-listener-config-dns-reference