Vector: New `http` source

[binarylogic]

An HTTP server as a source

• [ ] Configurable port
• [ ] HTTPs support?
• [ ] JSON, NDJSON, msgpack, and new line delimited decoding support

Anything else I'm missing? This is just a preliminary issue, we'll get it spec'd out further before we begin work.

[dneray]

I am also very interested in this feature, for an initial implementation I would be happy with:

• plain http support (I can run a reverse proxy alongside to handle tls)
• configurable IP + port to bind to
• json
• access to any http headers used to make the request

I think this would be enough to provide an avenue for integration with fluentbit/fluentd and logstash. I do agree that ultimately there should be support for all the listed formats, https and possibly basic auth?

[leshow]

I wrote some of the very beginnings of this, spawning a server with hyper & a configurable ip/port, and forwarding received messages to the Sender half of the channel provided to sources. I could submit a WIP PR and get some feedback on it? I realize you haven't fully gathered requirements but the implementation is so minimal right now I thought I could get the ball rolling.

[gedkins]

My recommendation would be to start from the logplex source and refactor it to make more and more things configurable (e.g. body format parsing, path, etc). Then it would be really nice if the specific sinks like logplex could be small wrappers around the new generic http source.

I've submitted a pull request that addresses the first part of this. I've got separate changes on the way for https support and for refactoring the logplex sink to use the shared code.

Note there may be an issue with this implementation, which we haven't looked into yet, regarding telling vector to reload the configuration - I tried it once and saw it complain about the port conflicting with itself.

[binarylogic]

This is done via https://github.com/timberio/vector/pull/1650