Influxdb: Feature Request: add web sockets for improved performance and no polling

[ReinsBrain]

Feature Request

__Proposal:__

Add web sockets to reduce handshaking for frequent POSTs to the same HTTP API endpoint. Also, utilize web sockets to maintain an open connection to push new elements to the client request (channel), perhaps based on (insert) triggers, from the data storage layer.

__Current behavior:__
Currently influxdb HTTP API uses standard HTTP(S) request/response model where POSTs and GETs require an HTTP handshake each time.

__Desired behavior:__
Streaming POSTs and GETs

__Use case:__
Aside from improved bandwidth, GET queries can stream new records allowing graphical displays that are highly responsive/fluid

[phemmer]

Desired behavior:
Streaming POSTs and GETs

What about a simpler method, such as streaming via chunked transfer encoding?
Websockets are generally used when you need 2-way interactive communication, and are a lot harder to implement. Pretty much everything supports chunked transfer encoding.

[e-dard]

@ReinsBrain are you batching your writes? A good sized for batched writes is around 10,000 points per batch. This can significantly improve write performance and reduce the number of requests you need to make.

[ReinsBrain]

@e-dard: perhaps GETs are more interesting for websockets although you wouldn't need to batch your posts with web sockets. Where websockets become interesting are for GETs - you'd have a sort of PUBSUB model with many listeners getting up-to-date data without the polling. @phemmer: 2-way comms are one use-case but receiving event-driven information is also a major use-case for it.

I think websockets would really be ideal for infrequent (or non-periodic) event logging - granted many event types saved to influxdb are probably mostly periodic measurements.

[phemmer]

2-way comms are one use-case but receiving event-driven information is also a major use-case for it.

No it isn't. event-driven information is a misuse of websockets. There are other better technologies for one-way streaming information.

[ReinsBrain]

@phemmer : where do you find that use-case cited as a mis-use of websockets? I've never heard of that although I would be very interested to find out. Rather I've read and understood the opposite. For example, from https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API they suggest: With this API, you can send messages to a server and receive event-driven responses without having to poll the server for a reply.

From RFC 6455 they suggest a few use-cases:

The same technique can be used for a variety of web applications: games, stock tickers, multiuser applications with simultaneous editing, user interfaces exposing server-side services in real time, etc.

I think the typical kind of logging data pushed to influxdb can be thought of, in many ways, similar to the "stock tickers" example they've given in the RFC. In specific regards to GETs, we are really looking at the PubSub pattern which is well-suited for distributing event-oriented data. You might even look at the Wamp sub-protocol (of websockets) which has significant parts dedicated to PubSub.

I'm feeling quite certain that websockets are a natural match for InfluxDB's HTTP API - particularly where data being logged to influxdb may be infrequent or sporadic. But perhaps I've missed something and I would be grateful if you could please point me any information that demonstrates the use of websockets I've suggested is an antipattern :)

Also, @phemmer : presuming we want to stay on HTTP(S) ports, what other streaming technologies would you recommend in place of websockets?

[e-dard]

@ReinsBrain

presuming we want to stay on HTTP(S) ports, what other streaming technologies would you recommend in place of websockets?

HTTP/2 springs to mind and is something we're thinking about for the future.

[ReinsBrain]

Hmm, I wasn't up on HTTP/2 - seems like we only just recently found websockets hitting most browsers. HTTP/2 seems imminently there so I would welcome whichever push method you folks opt for and I'll need to brush up on HTTP/2. I'm still super curious as to why event-driven data is a mis-use of websockets :)

Given all above, perhaps I might edit the feature request asking for push (over pull) updates to event data - then there is no need to close the issue.

[Tomcat-Engineering]

HTTP/2 is already supported by all common browsers, but which aspect of it helps in this situation? HTTP/2 Server Push is "not a mechanism for the server to notify things to the browser" (from https://en.wikipedia.org/wiki/HTTP/2_Server_Push).

Server Sent Events (SSE) is a much simpler way to do event streaming over HTTP than to use websockets. It would be a great fit for this application. It is a lot easier to implement on the server side than websockets. There is a useful description here.

The only drawback is that IE still doesn't support it, despite other most browsers implementing it about 6 years ago, so if you are writing webapps for third parties then it probably isn't an option. Microsoft say it is "under consideration".

Although websockets are often used for event streaming, I agree with @phemmer that they are generally overkill for this type of one-way-dataflow application. However, I guess people use them because there isn't an alternative with such good cross-browser support.

[ReinsBrain]

SSE is interesting but as @Tomcat-Engineering mentioned, websockets have wide support. Also, much work has been done to support the protocol in proxy servers and across wwan so it has wider access across many networks. I think that, while websockets may be more difficult to implement (although there are many libraries making it light work), it has broad coverage and is available today.

Perhaps we can nudge MS a little harder to adopt SSE : https://wpdev.uservoice.com/forums/257854-microsoft-edge-developer/suggestions/6263825-server-sent-events-eventsource

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please reopen if this issue is still important to you. Thank you for your contributions.

[sriramk4493]

Are there any alternatives for such use cases?