Fluent-bit: Please add support for Azure Event Hub

[igorgatis]

Is your feature request related to a problem? Please describe.
My company would like to use Azure Event Hub to collect system logs so they can be processed by Elastic Search and Databricks pipeline.

Describe the solution you'd like
Simple output plugin for Azure Event Hub.

Describe alternatives you've considered
I consider writing my own c plugin using https://github.com/Azure/azure-event-hubs-c.

Additional context
My company uses AKS and Azure Analytics is not the best tool for logging management.

[ahacking]

FYI: I am using the kafka output plugin with Azure Event Hubs. The output plugin uses librdkafka so you just have to configure the required connection string and messages arrive as expected. Just don't try and use message level compression as that is not supported in Azure Event Hubs.

[lukasmrtvy]

+1
Logstash and Filebeat support it:

• https://www.elastic.co/guide/en/logstash/current/plugins-inputs-azure_event_hubs.html
• https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-azure.html

[edsiper]

We just implemented Azure Blob support (will be part of 1.6 release in ~2 weeks), I am wondering if it solves the needs since the data can be streamed internally:

https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-capture-overview

[lukasmrtvy]

Superb !

One thing, if I remember correctly, there is slightly delay in capturing from event hub to azure blob ( it takes up to 10 minutes, realtime alerts might be a real problem ) and Event Hubs Capture is charged, it means that one have to pay for event hub and event hub capture...

There is another option for getting Metrics and Logs from Azure via Azure Monitor REST.
For example:

• https://github.com/tomkerkhove/promitor
• https://grafana.com/grafana/plugins/grafana-azure-data-explorer-datasource
• https://grafana.com/grafana/plugins/grafana-azure-monitor-datasource

[edsiper]

FYI: here you can find the doc for the new plugin for 1.6 (GIT master):

https://docs.fluentbit.io/manual/v/master/pipeline/outputs/azure_blob

note: you can choose between blockblob and appendblob

[lukasmrtvy]

OP ( and me ) asked for input plugin, not output..

Anyway,
Azure Storage ( blob ) output plugin is also good thing for storing logs to something more scalable/cheaper for archiving purposes.