Confluent-kafka-dotnet: Producer OnStatistics Event Never Fires

[keggster101020]

Description

Hi,

I'm attempting to use the producer.OnStatistics event to consume the tx and rx events so I can know how many messages my system has missed as well as the round trip time (rtt) for the producer. I tried using the consumer event but it seems as though there is a constant gap of (1) between tx and rx (ie: for consumer tx-rx = 1). Also my consumer rtt seems rather high (80ms average). I wanted to validate these against the producer statistics however, the OnStatistics event will only stay alive for the life of the producer. Since the producer is not kept alive like the consumer is (due to polling) is it possible to consume the producer statistics? I can never get the json of the producer statistics (ie: I can't step into the lambda function that is assigned to the producer event).

Thanks for the help!

How to reproduce

1. Build a producer in a using block.
2. Assign the OnStatistics event to a lambda function.
3. Notice you're unable to step into the assigned function.

Checklist

Please provide the following information:

• [x] Confluent.Kafka nuget version: 0.11.0
• [] Apache Kafka version:
• [] Client configuration:
• [x] Operating system: windows 10
• [] Provide logs (with "debug" : "..." as necessary in configuration)
• [] Provide broker log excerpts
• [] Critical issue

[mhowlett]

didn't quite get what you're asking here. The producer does have a poll loop, just like the consumer but for the producer we do the polling behind the scenes on a background thread for you (you can also configure it to manually poll if you like).

also might be worth mentioning that there is an interceptor API in librdkafka. you'll need to write native code, but you can arbitrarily intercept messages and do whatever custom logic you like there. this is used by Confluent Control Center to track message delivery like you are describing.

[keggster101020]

@mhowlett Thanks for the reply, essentially I'm trying to compare the OnStatistics events for both the producer and the consumer as the consumer is giving me a difference of 1 between tx and rx always.

From my understanding for dotNet: the consumer would look like the following:

//inside some exposed poll method
using(var consumer = new Consumer(...)){

    consumer.OnStatistics += (_,json) => {...}

    while(!cancelled) consumer.Poll(1000) //this will keep the consumer from calling .Dispose()

}

Since Dispose() isn't called on the consumer due to the explicit polling, the events can be kept alive.
However, for the producer, you would have

//inside some exposed produce method
using(var producer = new Producer(...)){

    producer.OnStatistics += (_,json) => {/*never enter*/}

    while(messages to produce){
        producer.ProduceAsync(...)
    }

    producer.Flush(...)

} //producer.Dispose() is called here. Erasing the OnStatistics event.

So if the total time it takes to run through produce is less than statistics.ms configuration time, OnStatistics will not fire. A workaround for this would be to keep one producer object similar to the consumer object in a way, but I could not find any examples showing the producer being used in a single instance so I'm not sure if this is considered safe.

[mhowlett]

you should keep a single Producer instance alive as long as you need it - you typically do not want to dispose / create producers all the time (thanks for noting your confusion here, others probably are too, will keep in mind when writing docs).

Note that there is a constructor for Producer that takes 3 arguments that allows you to run it in foreground poll mode (note though that this is being depreciated in 0.11.4, to be replaced by the dotnet.producer.enable.background.poll configuration property). But you probably don't want to use this. You probably just want to keep your producer around for a long time.

[keggster101020]

Ahh, that makes a lot more sense then. I'm assuming that this is the same for the consumer (although it's instance is being kept alive via poll) to create once and reuse.

Thanks for clearing up the confusion 😄