Confluent-kafka-dotnet: Authorization error from windows client when using DNS A record DNS instead of broker hostname using SASL_PLAINTEXT

[sysThematic]

Description

We've a kafka cluster running on linux hosts. Configured with SASL_PLAINTEXT. Both linux hosts and windows are on the same Active Directory Domain.

We have Confluent Open Source version 5.3.0, Apache version equiv. 2.3.0 running on Redhat hosts.

As told with issue #2549 we want to use a librdkafka's round robin DNS support and use a DNS record rather than broker's hostname.
When I use kafkacat on linux and it is working as expected. I can consume&produce with a DNS A record other than broker hostname.
I can also consume&produce with java clients when I use client.dns.lookup property.

However, it fails for confluent-kafka-dotnet 1.4.3 librdkafka 1.4.2 on Windows.
Is there a known issue on SASL via SSPI on Windows?

When we use dnsaliasnode01 instead of node01 (which is the part of broker's principle) we get the following error:

[KAFKA_ERROR] Message:sasl_plaintext://dnsaliasnode01:9092/bootstrap: Failed to initialize SASL authentication: InitializeSecurityContext failed: Target unknown (0x80090303) (after 0ms in state AUTH_REQ), Error:sasl_plaintext://dnsaliasnode01:9092/bootstrap: Failed to initialize SASL authentication: InitializeSecurityContext failed: Target unknown (0x80090303) (after 0ms in state AUTH_REQ)

How to reproduce

Checklist

Please provide the following information:

• [ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [X] Confluent.Kafka nuget version.
• [X] Apache Kafka version.
• [X] Client configuration.
• [X] Operating system.
• [X] Provide logs (with "debug" : "..." as necessary in configuration).
• [ ] Provide broker log excerpts.
• [X] Critical issue.

[edenhill]

The broker hostname (prior to resolve), as set by bootstrap.servers or through the broker's advertisted.listeners, is passed to the SSPI security context as part of the principal:
<sasl.kerberos.service.name>/<hostname>

https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_sasl_win32.c#L509

Are you specifying the exact same bootstrap.servers for kafkacat as you are your windows application? If so then there is something going on in SSPI.
Do note that with SSPI we are using the currently logged on user's credentail to authenticate.

cc @rnpridgeon

[edenhill]

The host resolution should work the same in librdkafka for both Linux and Windows, the mapping of CNAME to principal that client.dns.lookup=resolve_canonical_bootstrap_servers_only allows in the Java client is not supported in librdkafka, so if your Kerberos auth requires the hostname portion of the principal to match the secondary DNS entry then you are out of luck.

[sysThematic]

We don't have any kafka broker configuration for second dns record (dnsaliasnode01)
both listeners and advertised listeners shows node name.
listeners=SASL_PLAINTEXT://node01.domain.com:9092
advertised.listeners=SASL_PLAINTEXT://node01.domain.com.:9092

We are using the same value for bootstrap.servers on windows and for -b parameter for kafkacat.
If we use hostname of the broker on windows client it is working, and we have ACLs for logged on user of windows client, so I don't think we have problem there.

cc @serdarsert

Note:deleted previous comment and make some editing.

[edenhill]

If you can connect on Linux, but not on Windows, with bootstrap.servers=dnsaliasnode01, my guess is that Windows SSPI cares about the hostname part of the principal while Linux KRB5 does not.

Maybe there's something that needs to be done with SETSPN.

[rnpridgeon]

The krb5 client libraries are performing this translation for librdkafka. It's actually configurable in the linux host's krb5.conf.

see dns_canonicalize_hostname
https://web.mit.edu/kerberos/krb5-devel/doc/admin/conf_files/krb5_conf.html#libdefaults

and service principal canonicalization
https://web.mit.edu/kerberos/krb5-devel/doc/admin/princ_dns.html#service-principal-canonicalization

Java ships with its own kerberos client implementation which is why you configure it with the application directly.

I believe you can control this on a windows machine by setting the FEATURE_USE_CNAME_FOR_SPN_KB911149 register to True.

[sysThematic]

@edenhill @rnpridgeon Thank you both for real quick and informative answers.
I think it will be more complex to change OS settings instead of using default values.
We will handle this in our code, and change the config value back to hostname which is a part of the broker's principal via dns calls.
@edenhill I'm not closing the issue but I've my answers.
You can close the issue if you don't think this is worth implementing on librdkafka or maybe it is not in the scope of this project.

[edenhill]

@sysThematic Can you elaborate on your workaround?

[sysThematic]

We have two A records for the same IP, but only one of them has PTR record.
node01 192.168.0.11
dnsaliasnode01 192.168.0.11
PTR:
192.168.0.11 node01
We will use dnsaliasnode01 in our config, and get its IP from dns and perform a reverse lookup to get bootstrap.server config.

Like @rnpridgeon mentioned we will implement this feature from krb5:

If dns_canonicalize_hostname is set to true (the default value before release 1.19), the client performs forward resolution by looking up the IPv4 and/or IPv6 addresses of the hostname using getaddrinfo(). This process will typically add a domain suffix to the hostname if needed, and follow CNAME records in the DNS. If rdns is also set to true (the default), the client will then perform a reverse lookup of the first returned Internet address using getnameinfo(), finding the name associated with the PTR record.