Librdkafka: SASL with cluster thrashes the kinit cache (RHEL 6/7)

[crodier]

With a three node SASL cluster (latest master today 4/29, RHEL 6 or 7) it appears the SASLREFRESH steals the kinit credentials, causing connections to other cluster nodes to fail after the refresh.

Details:

First off, thanks for the terrific library, and apologies if this is a mistake in configuration.
I will also start looking at the source and trying to correct, but am in need of guidance.

SASL is up and running here, to a three node Kafka cluster, both Java and C++ are working.

• I can bootstrap to any node in the cluster with SASL.
• The handshake occurs, kinit works, and I receive the Metadata regarding the three nodes, to any node, as the start of the bootstrap.

But, later, I receive FAIL errors:

FAIL: sasl_plaintext://myhostA:9093/bootstrap: Failed to initialize SASL authentication: SASL handshake failed (start): SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Matching credential not found)

My best guess: the three 'kinit' command are overwriting the krb cache, thrashing the cache and prevent each other from connecting.

I am asserting this, because I see a SASLREFRESH to a different node in the cluster, right before the FAIL error to a different cluster node.

Also, if I run both a producer and a consumer, from the same app, this error seems to be more obvious that the krb cache is being thrashed by the producer and consumer threads.

LOGS:

consumer.txt
producer_and_consumer.txt

Best regards (and thx again),
Chris

[edenhill]

The code actually caches the kinit command, so it will run the first broker's SASL command for all of them, which really isnt the right thing to do..

Thanks for reporting this, will push a fix soon!

[crodier]

Amazing response time. I can't say enough about the platform and code, and we will look into a support license. In finance, multi-tennant setup, we will rely on SASL/Kerberos to prevent different groups in the firm from seeing each-other.

[edenhill]

@crodier See if that helps.
This has not been properly tested on my part yet.

[crodier]

@edenhill is that Thor's hammer!? It is fixed :+1:

I can't say thanks enough, And it will be a classic story for the Monday team meeting.

[crodier]

Running a producer and consumer, two threads in the same process, seems to thrash the cache now at startup, but eventually connects everywhere. I believe we are ok running one of them, but not both in one, or multiple in one. This may be a feature request but I thought would list it here as related. I'm looking for better details and at this issue.

[edenhill]

Can you enable debug=security to see what the exact kinit commands are for the two instances, if they look okay (i.e., dont seem to collide) then there might be an issue with kinit, otherwise librdkafka.
Thanks

[crodier]

Looking myself, here is the log I am reviewing thx again. It works and the messages are sent, but it seems to hiccup in authenticating, with both a producer and consumer starting simultaneously.

The interesting part in this test run is 'cluster2' (the second cluster node.) It fails to authenticate on line 22. Then if you scroll to the bottom of the log file before it begins reading offsets, you can see it successfully authenticates with 'cluster2' (second node), and is then alive.

kafaMultiThreadedSaslDebugSecurity.txt

[edenhill]

I suggest logging rd_kafka_name(rk) in your log callback as to be able to separate the different instances.
Looking at the log it is not clear what kinit command is for which client instance.

[edenhill]

They all seem to share the same principal (seti/[email protected]), is this intentional?

[crodier]

Principal the same on both threads, yes is intentional. It need not be, but I expect this is common. An app is writing to one topic, reading from another, shares a keytab to know who that app is. Is it me? If it helps to use distinct principals, I could test it.

Here is the callback, naming the producer and consumer. That is a big help thx, agree. Seems the consumer and producer are fighting to connect to the second cluster machine, appl147-stc (second consumer when I scrubbed it in the earlier log.) Then at the end, both of them authenticate to it.

logProducerConsumer.txt

using the -cache flag on kinit was one idea, the other was to use the kerberos library directly and not system(cmd) for these, but I'm still spinning up.

Thanks,
Chris

[edenhill]

This does seem like a race condition, question is where: is it in libsasl2 (Cyrus) or in kinit?
I dont think this is a librdkafka issue per-say since it relies on the afforementioned two to perform these tasks.
Unfortunately I dont know enough about how this works under the hood so I'm sort of lost for ideas, my suggestion at this point is to debug kinit and libsasl2 (the GSSAPI module).
Are you using the MIT or Heimdal GSSAPI module for libsasl2 btw? I've only used the MIT one.
See if changing module helps.

[crodier]

Even with one consumer, it seems like the kinit cache is being thrashed, but whatever the cause, we observe failures with only the consumer or producer boostrap connecting to multiple cluster nodes.

I don't believe kinit supports re-initializing with the same credential to multiple places, simultaneously. Leading me to believe we need to protect that section of the code with a lock, until we can introduce calls to the GSSAPI directly, if we want it to be multi-threaded.

Attaching logs for only the consumer side, connecting to a three node cluster. The fix I would like to try, would be to protect the authentication to each cluster node, and will be trying today. Could you possibly push a similar fix as to locking around the SASL authentication, one node at a time, preventing librd or anything from doing that simultaneously?

I am on RHEL 6, using MIT. We won't be able to switch to Heimdal, and this is the one that ships with Red Hat, believe every major linux distro. is on MIT now. My belief is that you can't call kinit this way, not a threading issue, rather that if you kinit as one thread, and then another, for the same principal with a different host, you overwrite the entry there (it will fail, by design, if you are using the same principal to three hosts, which we are?)

consumerBootstrapThreeClusterSasl.txt

[crodier]

http://web.mit.edu/kerberos/krb5-1.12/doc/user/user_commands/kinit.html

Under the "-c" cache_name argument:

"Otherwise, any existing contents of the default cache are destroyed by kinit."

In quotes makes me think that we are destroying credentials at the point when the TGT is acquired, before login has completed. Good guess?

[edenhill]

Interesting, does -c help?

[edenhill]

Easiest way to test this is by writing a script that adds -c and then configuring sasl.kerberos.kinit.cmd= to point at that script

[crodier]

I tried, but that doesn't work for different reasons. The problem is the kinit is called three times for the consumer. that can't be controlled from config (changing the cache used by each Kinit, or making it wait to finish one before the others.) I am looking at blocking around that section of SASL authentication (the kinit calls.)

[edenhill]

You are right, so do you think a unique cache file for each client instance would work?
Either we expose a new sasl.kinit.cache.path= property or generate one on the fly.

[crodier]

My understanding is that unfort. the unique cache will fail because SASL won't look in anything but the default cache (without more work.) When I added a cache as the kinit command, I was unable to login to any of the nodes in the cluster, even with one bootstrap node.

The near term solution I plan to try, would be block around the kinit call with a mutex. I don't perceive a case where the kinit call will ever fail to return with either success or an error, and give the lock back. Then, let the second kinit access the authentication code. While maybe not ideal, it seems ok, and the alternative appears to be integrating calls to the MIT krb5 library (heavy lifting.) If you are interested in the mutex approach and can push it to a branch, I can grab the branch and test. Otherwise I plan to build and attempt the mutex myself, if even as a quick validation exercise of the kinit interaction (and helps decide if it is worth pursuing as a long term solution.)

[edenhill]

So you will need to lock everything from kinit to the end of the SASL authentication, including ping ponging back and forth with the broker, right?
Even under good conditions that is a long time (from a locking perspective), under bad conditions (with network latency, timeouts, etc) this will not work well since INSTANCES * BROKERS number of threads will be blocking on the current broker thread to finish SASL auth.

[crodier]

We agree locking is sub-optimal. My idea was to validate the theory. I was able to validate it this AM.

This locking change in rdkafka_transport.c works (not surprisingly) with zero failures to the cluster.

`
pthread_mutex_t mutex1 = PTHREAD_MUTEX_INITIALIZER;

/**

• CONNECT state is failed (errstr!=NULL) or done (TCP is up, SSL is working..).
• From this state we either hand control back to the broker code,
• or if authentication is configured we ente the AUTH state.
  */
  void rd_kafka_transport_connect_done (rd_kafka_transport_t *rktrans,
  char *errstr) {
  rd_kafka_broker_t *rkb = rktrans->rktrans_rkb;

if WITH_SASL

if (!errstr &&
    (rkb->rkb_proto == RD_KAFKA_PROTO_SASL_PLAINTEXT ||
     rkb->rkb_proto == RD_KAFKA_PROTO_SASL_SSL))
{

    pthread_mutex_lock( &mutex1 );
    printf("BIG LOCK LOCKED\n");

    char sasl_errstr[512];
    if (rd_kafka_sasl_client_new(rkb->rkb_transport, sasl_errstr,
                     sizeof(sasl_errstr)) == -1)
    {
        errno = EINVAL;
        rd_kafka_broker_fail(rkb, LOG_ERR,
                     RD_KAFKA_RESP_ERR__AUTHENTICATION,
                     "Failed to initialize "
                     "SASL authentication: %s",
                     sasl_errstr);

    pthread_mutex_unlock( &mutex1 );
    printf("UNLOCKED BIG LOCK in error section\n");

    return;

    }

    rd_kafka_broker_lock(rkb);
    rd_kafka_broker_set_state(rkb, RD_KAFKA_BROKER_STATE_AUTH);
    rd_kafka_broker_unlock(rkb);


    pthread_mutex_unlock( &mutex1 );
    printf("UNLOCKED BIG LOCK\n");

    return;
}

endif

rd_kafka_broker_connect_done(rkb, errstr);

}

`

[edenhill]

Okay, good to know that solves the problem.

I discussed this issue with @ijuma and it turns out the Java client only refresh the token (kinit .. -R ..), I dont know if this makes any difference for us though.

[crodier]

This led me to looking at the "kinit" command, when using a keytab. Here is what may be proper fix , posting for your feedback while I also review.

(note the removal of service name and hostname and the hyphen S in the kinit command)

rd_snprintf(cmd, sizeof(cmd), "%s %s %s",
rk->rk_conf.sasl.kinit_cmd,
keytab,
rk->rk_conf.sasl.principal);

The fix, above, is to remove the "-S serice_name/hostname" from the kinit (at least this attempt.)

This works. (!). No failures, even with the race.

But it leads me to asking if we need to run three "kinits", when this is only to get the TGT. We want to use that same TGT against the cluster, is my improved plan. At least in the Keytab use case. I don't perceive any uses for multiple TGTs with the same keytab. This leads me to ask if the kinit should be a singleton or similar, where the kinit is run once and only once to save time.

I'm looking at that now, but above works, no SASL failures with three hosts.

[edenhill]

Are you sure the client can acquire a new ticket (if there is none cached locally) with -R and not -S?

[crodier]

Sry, not proposing -R. I'm removing the -S in code above. Second improvement may be to only kinit one time, as we don't need three TGTs?

[crodier]

It works with one kinit ! , with keytabs, if you remove the "-S".

With keytabs, do we need to kinit three times, with -S / (is that a side effect of clustering in use?)

I believe we want to remove the -S with keytabs (service name will come from the keytab is my understanding.) Without keytabs, I need to investigate, but '-S' is likely necessary.

With keytabs and a cluster, we benefit from a mutex and only one kinit, speed wise (but could run three without the -S). Mutex wise, whichever broker transport arrives first should be ok (inside the rd_kafka_sasl_kint_refresh.) Would you like me to work on a pull request? May be red tape here for me to commit.

[crodier]

I believe best of all worlds, would be:

• make "-S servicename" optional, via a property "sasl.kinit.servicename".
• Only set the -S if this property is set, default to not using it.
• with keytabs (probably without) put a mutex around kinit, and only kinit once (but need to be careful regarding refresh, need to refresh it, and then again, only once. this is the tricky bit.)
• default to kinit only once as you should only need one TGT for the entire Kafka Cluster
• later could investigate kinit multiple times, if it becomes necessary (but should not with keytabs supported)

Thanks for directing me during this fix; we are close or this sorts it out.

[edenhill]

Thanks for your effort @crodier, I can randomly reproduce this now using the sasl_test.py test suite.
Will try out your suggested changes and let you know.

[edenhill]

Kerberos is not one of my strongest games.

Why is it that -S servicename is optional and the kinit seems to work flawlessly without it?
Is there even a need for specifying the service name?

I want to fix this issue prior to (the imminent) release but I'm a bit apprehensive that this might break SASL for other users whos setup are different (in whatever way).

One way around it would be for the kinit command format to be fully configurable, e.g.:
sasl.kerberos.kinit.cmd="kinit -S %{sasl.kerberos.servicename} ...-k %{sasl.kerberos.keytab} .."

Also, the fix now will only add a lock around each kinit invocation, the singleton approach will not be done now since it is more complex.

[crodier]

Thanks Magnus, seems plenty strong. I am not an expert with Kerberos either sadly. The lock around each invocation works, thanks, as well as your proposed fix, and your proposed workaround seems like a plan.

Long story on -S, but basically the kinit is the same without your -S, , and then, it doesn't matter which broker connection kinit last, it will always be usable. I believe it is used when you want to use an alternate service name to what is in the keytab, or if not using keytabs, but I don't use it when I kinit and yes it does seem unnecessary, perhaps an option that _could_ be used, but no one should be honestly. Unless we are aware of any users of Kerberos in prod, possible I am the first in the wild?

One more wrinkle, and it should be usable in production, for your review..

We need to refresh tickets, run kinit -R or krenew? http://linux.die.net/man/1/krenew

Have you given the ticket renewal process any thought? I stumbled upon it in testing today.
The kinit ticket will expire (defaults to one day.) We need a solution to re-request the ticket, otherwise after a day, your ticket expires, and you can't send messages. This can be worked around in /etc/krb5.conf, by changing the ticket_lifetime, but I doubt users would expect to alter this (instead expecting the ticket to be renewed on a refresh cycle.)

Unfortunately, this seems to warrant the singleton, with a timer, but I realize this is more work. Without it, most people may launch, lose their ticket a day later (maybe on launch day +1.)

Strategy wise, I've seen GSS Java work, but not via SASL, and I'm fuzzy on the kinit to SASL interaction. It seems like SASL is a hack, when you actually only care about and want Kerberos via the GSSAPI, directly, is my take right now. The use case for something other than Kerberos, does not exist in my view, e.g. the other SASL authentication mechanisms. I suppose that is a +1 for dropping SASL and using the GSS api directly in librdkafka, if that makes any sense, but is conjecture for your review.

[edenhill]

-sasl.kerberos.kinit.cmd                  |  *  |                 |         kinit | Kerberos kinit command path. <br>*Type: string*
+sasl.kerberos.kinit.cmd                  |  *  |                 | kinit -S "%{sasl.kerberos.service.name}/%{broker.name}" -k -i %{sasl.kerberos.principal} -k "%{sasl.kerberos.keytab}" | Full kerberos kinit command string, %{config.name} are replaced corresponding config object values, %{broker.name} returns the broker's hostname. <br>*Type: string*

Does that change seem okay to you?

[edenhill]

I will not fix the issue with keys timing out for the upcoming release, we'll need more time to figure out how this is actually supposed to work
So until then you can provide a wrapper script for sasl.kerberos.kinit.cmd that backgrounds and refreshes tickets as necessary. When we know what that looks like we can move it into the library C code.
Sounds ok?

[edenhill]

Please review these two commits (and test!) as soon as possible, thanks!

[edenhill]

I changed the kinit.cmd default behaviour to be like the previous defualt, using -t %{...keytab}.
Seems more backwards compatible.

[edenhill]

https://github.com/edenhill/librdkafka/commits/sasl_kinit

[crodier]

Perfect. But we must reinit the cache on a timer. The tgt expires. Try
running for 24 hours, see if fails? Should start failing after 12-24 hours.
On May 19, 2016 1:26 PM, "Magnus Edenhill" [email protected] wrote:

-sasl.kerberos.kinit.cmd | * | |
kinit | Kerberos kinit command path.
_Type: string_
+sasl.kerberos.kinit.cmd | * | |
kinit -S "%{sasl.kerberos.service.name}/%{broker.name}" -k -i
%{sasl.kerberos.principal} -k "%{sasl.kerberos.keytab}" | Full
kerberos kinit command string, %{config.name} are replaced
corresponding config object values, %{broker.name} returns the
broker's hostname.
_Type: string_

Does that change seem okay to you?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/edenhill/librdkafka/issues/635#issuecomment-220394703

[crodier]

Sorry was out. I will pull and try this morning.
On May 23, 2016 4:42 AM, "Chris Rodier" [email protected] wrote:

Perfect. But we must reinit the cache on a timer. The tgt expires. Try
running for 24 hours, see if fails? Should start failing after 12-24 hours.
On May 19, 2016 1:26 PM, "Magnus Edenhill" [email protected]
wrote:

-sasl.kerberos.kinit.cmd | * | | kinit | Kerberos kinit command path.
_Type: string_
+sasl.kerberos.kinit.cmd | * | | kinit -S "%{sasl.kerberos.service.name}/%{broker.name}" -k -i %{sasl.kerberos.principal} -k "%{sasl.kerberos.keytab}" | Full kerberos kinit command string, %{config.name} are replaced corresponding config object values, %{broker.name} returns the broker's hostname.
_Type: string_

Does that change seem okay to you?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/edenhill/librdkafka/issues/635#issuecomment-220394703

[crodier]

Looks great ty. I will leave it running, suspect we still need a strategy for kinit refresh on a timer. Most kafka processes will run for longer than the standard TGT lifetime. Pls. let me know if I can look. When SASL refresh occurs, a check to see if it has been longer than a default, configurable "sasl.kinit.refresh.hours" value, and then kinit refresh, is one idea.

[crodier]

Looks great in testing. Librdkafka support goes to 11. Thank you again Magnus!