Php-rdkafka: [producer shutdown] To purge or not to purge? To flush or not to flush?

[simPod]

Hi, what are the options on producer shutdown with v4? I found this section in docs https://github.com/arnaud-lb/php-rdkafka#proper-shutdown but it doesn't explain all options.

I understand I should call flush() to attempt to produce all queued messages in order not to lose them. But what if I eg. don't care about losing them?

So I was wondering what is the difference between:

• calling only purge()
• calling purge() and flush() consequently
• doing nothing, no purge() nor flush()

[nick-zh]

Hey @simPod

Very good question, so if you don't care about losing messages, why send them in the first place :smile:
Just kidding, but yes then you don't need to purge or flush.

There are two ways to purge, check it at the bottom of the doc:
https://arnaud.le-blanc.net/php-rdkafka-doc/phpdoc/rdkafka.examples-producer.html

I can see how somebody might want to use the first flag, i am not sure what a use case would be for the second flag.
So ì personally see RD_KAFKA_PURGE_F_QUEUE as safe abort. Probably your application will only mark it as sent, when you get the delivery report for example, so purge would be safe to do, for an application that handles messages this way.

As you pointed out correctly by you, flush makes sure all queued messages go out, so you don't lose any.

As always with Kafka, a lot of settings and implementation details depend on your use case.
I hope this helps.

I see how proper shutdown might me misleading, it's more of a suggestion / note for those who don't want to lose messages.

[nick-zh]

@simPod closing this for now, feel free to reopen if something is still unclear.

[simPod]

_Sorry, the notification got too far in my inbox :/_

Thanks for reply!

So it is recommended for the application call at least either flush or purge. What if app does not call any? Is there any difference compared to calling purge? Messages might be lost in both cases, right?

[Steveb-p]

What if app does not call any?

Depends on the phprdkafka major version.
In 3.x, background thread will keep retrying messages that failed to send for whatever reason (until time out). This has virtually no effect when Kafka is OK, but if broker is down for some reason it causes PHP processes to linger and choke. If you use PHP-FPM for example, it would either continuosly spawn new processes (which would starve your server of resources) or block new requests (since there would be no processes available to handle incoming requests).

In 4.x it was decided to allow user (that is, you) to control how this mechanism should behave - you have full control when to stop waiting for messages to be sent (and received) by Kafka. If you do not wait (that is, call flush and wait for thread queue to become empty) then thread terminates alongside the main PHP process. Whether message is received by Kafka depends on whether aforementioned thread managed to push the message or not - if you're not waiting at all, then there's no way of knowing (aside from looking into broker).

IIRC flush does not remove messages from internal message queue when timeout (argument) is reached, but I might be wrong. That's what purge is for I believe? @nick-zh might want to correct me on this one.

As a side note: background thread is something that librdkafka does. This thread, being a child of PHP process, lives and dies with it. As I said, in 3.x this thread prevented the main PHP process from ending, hence the change.

[nick-zh]

@Steveb-p is totally right, flush and purge were added to give the user (us) full control.
As stated correctly, the previous behaviour was to accomplish the same as flush but in php-rdkafka:3.x it was possible that messages were lost during shutdown in rare cases.

Recommendations can only be given for certain use cases, i am listing some, but i am sure there are many more :smile:

Use case 1
You want to make sure messages are sent out. Your application does not track if sending a message was successful or not and has no internal requeuing.

• you need to call flush, be sure that it returns `RD_KAFKA_RESP_ERR_NO_ERROR

Use case 2
You want to make sure messages are sent out. Your application tracks if sending a message was successful (application tracks delivery reports) and has persistent internal requeuing for messages that were not delivered successfully.

• you can safely call purge(RD_KAFKA_PURGE_F_QUEUE)
• you need to call flush, be sure that it returns `RD_KAFKA_RESP_ERR_NO_ERROR

Use case 3
You are sending millions of messages in one process, losing a few is acceptable.

• you can skip purge and flush (i personally would still call flush)

Additionally if exactly once is important, use idempotent producer.

tl;dr Since the default behaviour was to flush, in most cases you want that too.
Hope this sheds some light, but again, most settings, etc. really depend on your use case.

[simPod]

All clear, one last question: is there a difference between a) calling purge b) calling nothing? Why would I call purge? Does it make shutdown of underlying process faster?

[Steveb-p]

All clear, one last question: is there a difference between a) calling purge b) calling nothing? Why would I call purge? Does it make shutdown of underlying process faster?

purge clears internal queue. Then new messages can be placed in queue in the same process.

Simply allowing the process to exit means that your operating system will close all related threads (phprdkafka in this case) resulting in memory being "wiped".

As said before, in 3.x thread closing was prevented and "not calling purge" would result in process still trying to send them until time out.

[nick-zh]

With calling purge:
Any message that was produced (internally queued) but not yet sent to the broker will not be sent.
If you are ok losing them or your application tracks it and sends them later, using purge can make sense.

Without purge:
When you flush, every message you produced will be sent out (unless of errors ofc).

Note: Base behaviour in php-rdkafka:3.x was to not purge.

EDIT: I forgot, in terms of faster, Kafka and the producers / consumers are anyway really fast, unless you have tons of messages queued (which you probably don't), this won't make a noticeable difference

[nick-zh]

@Steveb-p lolz too fast man :smile:

[simPod]

Both answers gave me some more insight but I still feel answer to my question is somehow unclear 😄

Considered I don't care about messages being sent and the only thing I want is to exit the producer process, should I for some reason call purge or I can simply exit without calling any of purge and flush?

I'm aware purge will clean up the queue but exiting the process will clean up the memory as well. So there's no reason to call purge, am I right?

[Steveb-p]

I'm aware purge will clean up the queue but exiting the process will clean up the memory as well. So there's no reason to call purge, am I right?

That's exactly what will happen.

The only difference is, calling purge will explicitly remove any leftover messages that weren't sent or failed to send.
Process ending will "clear" the queue - or rather, background thread will be stopped and removed, which basically achieves the same thing.

[simPod]

Thanks a lot guys!