Beats: Generic HTTP output

[tsg]

This issue is meant to measure the interest in a potential generic HTTP output for the Beats.

Since the Elasticsearch output is already HTTP, it can be used by custom applications that want to ingest data from the Beats. If this is not a good option for you, we'd be interested in hearing your use case and why it's not an option.

Related discuss ticket: https://discuss.elastic.co/t/output-beat-events-as-plain-http-post/57923

[alefebvre-ls]

@tsg In case it's of any help to you, here's another discussion that actually pre-dates the one you've included:

https://discuss.elastic.co/t/http-output-to-logstash/33269/16

[SeijiSuenaga]

This would be useful for me, but the linked discussion contains a very valid concern: this feature is a slippery slope of additional requirements. My use case would require a custom Authorization header and probably request bulking.

[moooofly]

I need this generic HTTP output, here is my use case

topology

|<-----  internet ---->|                      |<---  intranet -->|  
proxy <--> filebeat <--> (custom http server)  <--> kafka <--> ELK

comments

• Both kafka brokers and ELK are in a private network accessible only to organization's staff.
• Both proxy and filebeat are placed on public cloud providers.
• For the sake of security, filebeat can not access kafka directly, but allow to POST the content of log to the custom http service which is connecting to kafka brokers.

@tsg

[Pectojin]

In our organisation we ship logs from a lot of different customers to Logstash. We've got a no-trust environment where we cannot count on our customers to submit logs to only their respective log indexes. The customer submitting logs to another customers index is not acceptable.

We've secured this by using the Logstash HTTP input behind an NGINX proxy terminating TLS. In addition to terminating TLS we also validate the HTTP URI so that each index has it's own URI, part of which is an authentication secret.

This setup is incredibly easy to implement for our customers either directly in their application or using fluent-bit HTTP output. Additionally it has the added benefit of being much more tangible than configuring hundreds of client certificates and trying to keep track of them. With HTTPS as protocol both encryption and authentication can be communicated in a single URL.

It performs quite well despite not supporting batch processing. We really only rely on HTTP with JSON payload and TLS.

I don't see a clear path for how to implement this flow using any other options in the beats agents or Logstash. Currently if we need beats capabilities the output must then be thrown at a fluent-bit agent to actually submit logs, which is not ideal.

Addressing the fear of scope creeping, I don't think it's unreasonable to implement a beats plugin that only implements the functionality that Logstash HTTP input supports. With that purpose stated it'd be fine to decline any new feature requests that aren't supported by Logstash.

[djmcgreal-cc]

Since the Elasticsearch output is already HTTP, it can be used by custom applications that want to ingest data from the Beats. If this is not a good option for you, we'd be interested in hearing your use case and why it's not an option.

Seemingly suggest it's possible to use the ES output to send to an HTTP endpoint? That HTTP would need to behave like ES, e.g. replying to the requests with sensible JSON.

Is that right?
Thanks, Dan.

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic[bot]]

This issue doesn't have a Team:<team> label.