Confluent-kafka-dotnet: Support for OAuthBearer SASL authentication mechanism

[kuehlmeyer]

Description

It seems there's no support for the SASL mechanism OAuthBearer (see https://docs.confluent.io/current/kafka/authentication_sasl/authentication_sasl_oauth.html#) in the .NET version of the Confluent.Kafka lib. The class SaslMechanism has only the members Gssapi, Plain, ScramSha256, and ScramSha512 in 1.0.0-RC3.

Are there plans to support OAuthBearer in the near future? Or is there another way to use it for .NET producers and consumers?

How to reproduce

See source of SaslMechanism class.

Checklist

Please provide the following information:

• [ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [1.0.0-RC3 ] Confluent.Kafka nuget version.
• [ ] Apache Kafka version.
• [ ] Client configuration.
• [Windows 10] Operating system.
• [ ] Provide logs (with "debug" : "..." as necessary in configuration).
• [ ] Provide broker log excerpts.
• [ ] Critical issue.

[mhowlett]

yes, a PR for this is in the late stages of review in librdkafka, you can expect it in a post-1.0 release (which will occur more frequently than 0.11.6 -> 1.0!): https://github.com/edenhill/librdkafka/pull/2189

[Fallibilist]

@mhowlett
Any progress on this?

[mhowlett]

it's here: https://github.com/edenhill/librdkafka/releases/tag/v1.1.0

I omitted to regenerate the strongly typed .NET Config classes for the v1.1 release, but you can still use the relevant config via librdkafka style properties. Will fix this in the upcoming 1.2 release.

https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

[edenhill]

For OAUTHBEARER to be meaningful the application must provide a token refresh callback and token acquisition logic, the built-in unsecure JWT is just for testing, and not for actual production use.

I believe the .NET client currently lacks the new APIs required to perform the token operations.

[Fallibilist]

@edenhill
Thanks for the help!

So it is currently possible with the .NET client? It is straightforward enough to create the callback to fetch the JWT, but I'm not certain how to pass the callback to the producer config.

Currently I'm setting the config via a Dictionary in the builder constructor which won't allow me to pass in a callback as a value for the "oauthbearer_token_refresh_cb" option which librdkafka uses.

[edenhill]

It is not currently supported by the .NET client.

[mhowlett]

ahh yes, I forgot we need to add the callback.

[edenhill]

And set_token*()

[Fallibilist]

Is there any way to access the underlying C/C++ functions within C# without forking the repository and recompiling?

[mhowlett]

I think it might be, but I could be forgetting about something. Have a look in NativeMethods.cs to see how to do the p/invokes.

[thtp]

Any progress? It seems like a quite simple issue, but there has been no progress for almost a year.

[weberms]

Any updates? OAuth or SASL/SSL support is a blocker for us to adopt KAFKA in our environment.

[mhowlett]

noted. i'm implementing a big feature which is behind schedule and i need every minute of every day is what's happening. after that a few quick wins for .net will go in, including this.

[weberms]

Thanks for the quick response. Any estimate when that might be? Next month, 3 months, etc.?

[mhowlett]

yes :-).

you could use use a #1295 PR build from appveyor as an interim measure: https://ci.appveyor.com/project/ConfluentClientEngineering/confluent-kafka-dotnet/history

or build a nuget package yourself.

on the C# side, there's not a lot to it, so you can trust the stability of the feature. i don't know what the api will be precisely yet, but changes will be cosmetic and not a big deal to swap over to whatever we decide on.

there is a lot of demand, so you can be sure it's coming.

[mhowlett]

this is now merged into 1.5.x, and will be in a point release soon.

[weberms]

Matt, I upgraded to v1.5.1 but I don't see the OAuth option added to the SaslMechanism enumeration.

Also will you be creating a sample of how to properly use OAuth for consumers and producers?

[mhowlett]

the option is definitely there: https://github.com/confluentinc/confluent-kafka-dotnet/blob/v1.5.1/src/Confluent.Kafka/Config_gen.cs#L233

unfortunately we won't get time to do an example (ideally more a tutorial with code) in the near term. there are integration tests you can refer to as examples in: https://github.com/confluentinc/confluent-kafka-dotnet/tree/master/test/Confluent.Kafka.IntegrationTests/Tests

[KunalAdu]

@weberms : Can you share the sample reference for OAuth Consumer and producer ?