Elasticsearch: Feature: Zstandard compression support and possibility to choose a compression algo

[Cornoualis]

Hi !

It would be awesome to add support to Zstandard, an awesome (de)compression algorithm by Facebook.

The benchmarks show impressive speed/compression rate, compared to zlib.

It would be very nice to have the possibility to have an "algorithm" setting in the snapshot API...

Example:

PUT /_snapshot/my_backup
{
    "type": "fs",
    "type": "fs",
    "settings": {
      "location": "/mount/backups/my_backup",
      "compress": true,
      "algorithm": "zstd"
  }
}

There are bindings provided for many languages and of course, for java/jvm.

The idea would be to let the user choose an algorithm, which implies that the user has the zstd binaries installed (like elastic requires the jre/jdk to be installed).

[jpountz]

On the one hand I like the idea because I think Zstandard is a great compression algorithm. On the other hand, we have been striving to not use libraries that use JNI bindings or Java's sun.misc.Unsafe in order to keep troubleshooting easy. Overall this has more cons than pros to me. If compressing snapshots is what you are after, maybe another way to make it work would be to enable compression on the volume that store the snapshots at the filesystem level.

[Cornoualis]

Salut and thank you for your answer!
I understand what you mean. I'll give a try on OpenZFS+LZ4 compression...

[denji]

The benefit of zstd is achieved using dictionaries. @Cornoualis OpenZFS not fully stable cross platform.