Confluent-kafka-dotnet: Subscribing to non-existant topic silently fails

[qed-]

With automatic topic creation is disabled on the broker, calling Consumer.Subscribe() on a non-existant topic, or a list of topics where one does not exist, does not return an error.

No error callbacks or partition assignment callbacks are invoked.

Using Nuget package 0.9.5

[edenhill]

This is intentional since the client can be started before the topic is created (through other means).

You could log a warning in your application if it gets an empty assignment, and possibly check with the metadata API if the expected topics are missing or not.

[qed-]

It doesn't get any assignment at all.
In the case of subscribing to multiple topics shouldn't it be assigned the ones that do exist?

[edenhill]

Right, if none of the topics in the subscription exist then the client will not send a JoinRequest and thus not get an assignment.
But if there is at least one existing topic it should indeed join and get an assignment (possibly empty if there are other consumers in the same group).

You can troubleshoot this by setting the debug property to cgrp.

[qed-]

Seems like it's not going to try and join until it has all topics. This is the advanced consumer example, trying to subscribe to topics test, test2 and test3. Only test and test2 exist:

7|2017-05-04 12:03:49.366|rdkafka#consumer-1|CGRPQUERY| [thrd:main]: kafka-bv-dev01-01.xxxx.com:9092/1: Group "advanced-csharp-consumer": querying for coordinator: intervaled in state up
7|2017-05-04 12:03:49.366|rdkafka#consumer-1|CGRPCOORD| [thrd:main]: kafka-bv-dev01-02.xxxx.com:9092/2: Group "advanced-csharp-consumer" coordinator is kafka-bv-dev01-02.xxxx.com:9092 id 2
7|2017-05-04 12:03:50.315|rdkafka#consumer-1|SUBSCRIPTION| [thrd:main]: Group "advanced-csharp-consumer": effective subscription list changed from 0 to 2 topic(s):
7|2017-05-04 12:03:50.315|rdkafka#consumer-1|SUBSCRIPTION| [thrd:main]: Topic test with 1 partition(s)
7|2017-05-04 12:03:50.316|rdkafka#consumer-1|SUBSCRIPTION| [thrd:main]: Topic test2 with 1 partition(s)
7|2017-05-04 12:03:50.316|rdkafka#consumer-1|JOIN| [thrd:main]: Group "advanced-csharp-consumer": join with 2 (3) subscribed topic(s)
7|2017-05-04 12:03:50.316|rdkafka#consumer-1|CGRPMETADATA| [thrd:main]: consumer join: metadata for subscription only available for 2/3 topics (0ms old)
7|2017-05-04 12:03:50.316|rdkafka#consumer-1|JOIN| [thrd:main]: Group "advanced-csharp-consumer": postponing join until up-to-date metadata is available
7|2017-05-04 12:03:50.316|rdkafka#consumer-1|CGRPCOORD| [thrd:main]: kafka-bv-dev01-01.xxxx.com:9092/1: Group "advanced-csharp-consumer" coordinator is kafka-bv-dev01-02.xxxx.com:9092 id 2
7|2017-05-04 12:03:50.802|rdkafka#consumer-1|JOIN| [thrd:main]: Group "advanced-csharp-consumer": join with 2 (3) subscribed topic(s)
7|2017-05-04 12:03:50.802|rdkafka#consumer-1|CGRPMETADATA| [thrd:main]: consumer join: metadata for subscription only available for 2/3 topics (500ms old)
7|2017-05-04 12:03:50.803|rdkafka#consumer-1|JOIN| [thrd:main]: Group "advanced-csharp-consumer": postponing join until up-to-date metadata is available

[edenhill]

That looks like a bug in librdkafka.

One possible workaround for now is to specify the topic names as regexes (prefix with "^" and suffix with "$", e.g.: "mytopic" becomes "^mytopic$")

[qed-]

Thanks, have raised an issue there https://github.com/edenhill/librdkafka/issues/1201

[mhowlett]

closing as this is being tracked in https://github.com/edenhill/librdkafka/issues/1201