Ktor: Feature request: support for RFC 7692 (Compression Extensions for WebSocket)

[volkert-fastned]

It would be very useful if ktor-websockets gained support for Compression Extensions for WebSocket, as defined in RFC 7692. The WebSocket Subprotocol "ocpp2.0" (Open Charge Point Protocol version 2.0) requires this feature to be implemented at the server side, specifically using the DEFLATE algorithm, which is included as the "permessage-deflate" extension in section 7 of the aforementioned RFC.

I guess there's probably a way to manually implement the handling of DEFLATE (de)compression according to this extension in application code on top of Ktor, but it would be much nicer if Ktor would gain integrated and transparent support for this extension in its WebSocket functionality, so this would work out of the box without requiring any application-specific programming.

Thank you in advance for considering this feature.

[volkert-fastned]

So according to paragraph 5.8 of RFC 7692, "PMCEs use the RSV1 bit of the WebSocket frame header to indicate whether a message is compressed or not". Unfortunately, the RSV1 bit of the frame header (referred to as frame-rsv1 in RFC 6455) appears to be disregarded in Ktor's FrameParser class, which seems to abstract the frame header stuff away, preserving only the opcode value, which is then used to determine the frame type.

Even Ktor's Raw WebSocket API doesn't seem to expose the detailed frame header metadata required to determine whether a WebSocket message is compressed or not.

Hacking Per-Message Compression Extensions (PMCEs) on top of Ktor doesn't seem possible without patching Ktor. It would therefore indeed seem better if Ktor were to be enhanced to provide built-in transparent support for RFC 7692 WebSocket (de)compression out of the box.

[avently]

Any news?
For those who don't now about this compression see https://www.igvita.com/2013/11/27/configuring-and-optimizing-websocket-compression/#parameters

So we can expect 50-75% of data reduction. My app uses ~5GB per day. Even 50% reduction will be really good.

Here is a PR for OkHttp. For now it's not merged because of the upcoming release of Okhttp v4.0. https://github.com/square/okhttp/pull/4871
Would be great to have the feature implemented in Ktor itself since it's uses custom decryption mechanism and should be supported by Ktor's frame parser.

[jinsankim]

Any plan to implement this feature yet? Since I faced it, I couldn't go ahead w/ Ktor because my project needs rfc 7692 for websocket client.

[r4zzz4k]

JFYI, OkHttp introduced permessage-deflate in 4.5.0 this week. Related PR: square/okhttp#5871

[volkert-fastned]

JFYI, OkHttp introduced permessage-deflate in 4.5.0 this week. Related PR: square/okhttp#5871

Thanks for sharing that. However, Ktor offers OkHttp only as a WebSocket client engine. It still wouldn't help in getting WebSocket compression supported in Ktor's WebSocket server engine.

[volkert-fastned]

Sorry for accidentally closing and reopening this issue. Is mis-tapped on my laptop. Why doesn't GitHub pop up a confirmation window for actions like this? 😳

[r4zzz4k]

Sure, I've just shared a place where RFC 7692 was recently in client-side, nothing more than that :)

[volkert-fastned]

Sure, I've just shared a place where RFC 7692 was recently in client-side, nothing more than that :)

Of course, no problem. 🙂 And who knows, perhaps it will provide a convenient shortcut to add WebSocket compression support at least to the client-side of Ktor's WebSocket functionality, before it can also be implemented server-side.

But in our specific use case, we really need it server-side. :sweat_smile:

@e5l I know you guys are very busy with Ktor, with a lot of issues currently still open, but could you perhaps give an idea on a possible time table for a feature like this? And would you be open to Pull Requests for this in the shorter term? Thanks.

[e5l]

Hi @volkert-fastned, I'm working on fixing native problems and problems with IO, but I'm really welcome to work on the PR :)

Sorry for the delay

[volkert-fastned]

Okay, I started work on this here: https://github.com/ktorio/ktor/pull/2014

Everybody, feel free to contribute and/or comment. :slightly_smiling_face:

[edvorg]

@volkert-fastned I'm also interested in getting this feature implement, do you need any help this? How much effort do you estimate would be required to get it done?

[volkert-fastned]

@edvorg All help is welcome, of course. :smiley:

I'm not working on this feature full-time.

By the way, I'm only working on the permessage-deflate compression specifically, since that's the default/example implementation that is defined at https://tools.ietf.org/html/rfc7692

In the IANA registry, there is one other vendor-specific compression extension defined, but I won't be working on that for now: https://www.iana.org/assignments/websocket/websocket.xml

Also, for HTTP compression Ktor currently uses the standard DEFLATE implementation provided in java.util.zip. I intend to use that for implementing WebSockets permessage-deflate support as well. However, that standard Java implementation currently does not support the option to specify the LZ77 sliding window size. There was a feature request opened for that almost 6 years ago: https://bugs.openjdk.java.net/browse/JDK-8056093 (I'm not holding my breath for that to be implemented any time soon.)

So I think the first version won't be supporting that either, since it would require an alternative library, such as LZ77Compressor from Apache Commons.

Let's get the simplest version implemented first. :sweat_smile:

Would that be okay for you as well?

[oleg-larshin]

Please check the following ticket on YouTrack for follow-ups to this issue. GitHub issues will be closed in the coming weeks.

[volkert-fastned]

Fixed and included in Ktor release 1.5.0. See https://youtrack.jetbrains.com/issue/KTOR-688 and https://ktor.io/changelog/1.5

Thanks for implementing this, Ktor developers! :ok_hand: