Logstash: DLQ Input should be able to delete read messages

[andrewvc]

It is commonly the case that users will have a secondary pipeline for processing DLQed messages. The DLQ input should be able to mark processed messages as 'dead', and eligible for immediate deletion.

This would enable use cases like https://discuss.elastic.co/t/why-does-logstash-not-clear-dlq-after-successful-fixes/113618

[andrewbanchich]

+1000

[reswob10]

Can you please make this an optional switch? It can default to on, but can be set to off at the discretion of the admin.

[dannygoulder]

+1

[victor-mendes-eduardo]

+1

[keshrisohit]

+1

[alexPolisevschi]

+1

[jakub-bacic]

+1

[SuchiSmita-N]

Just curious, Is it still ongoing ? We have enabled DLQ but the processed messages are not deleted automatically.

[opiumfor]

+1

[Alsheh]

+1

[zmays]

Is there a current solution for when the DLQ fills up? I've added a pipeline for DLQ, but it filled up and stopped processing events. I was under the impression it was supposed to rotate the file automatically from docs here:

File Rotationedit
Dead letter queues have a built-in file rotation policy that manages the file size of the queue. When the file size reaches a preconfigured threshold, a new file is created automatically.

By default, the maximum size of each dead letter queue is set to 1024mb. To change this setting, use the dead_letter_queue.max_bytes option. Entries will be dropped if they would increase the size of the dead letter queue beyond this setting.

Which, upon reading now is a little confusing, it says that a new file will be automatically created, and that entries will be dropped.

[vivekkarne]

+1

[DonatasR]

+1

[delhisanjay]

Facing same issue i.e. DLQ does not clear queue content even after successful processing. Do we have fix for above issue --

[bendoerr]

Maybe it should be renamed. "Dead Letter Dump"?

[jerome83136]

+1

[naimmalek]

+1000

[arpit-ghiya]

+1

[bendoerr]

Any bets on how long this issue will stay open without comment from the maintainers?