Adios2: Blosc Transform

[ax3l]

Hi,

opening since I did not find an issue on it. Are we still planning to add a blosc transform again to ADIOS2?

We are using it all the time in PIConGPU as parallel compressor (https://arxiv.org/abs/1706.00522) and like the modern methods it implements as a meta-compressor (e.g. zstd and bitshuffle pre-conditioners) as well.

I think to remember we talked last time if we want to implement (thread/OpenMP) parallelization of transformation steps in general.

[williamfgc]

@ax3l any particular blosc version being targeted?

[ax3l]

Just the latest and potentially the upcoming blosc2 api's (still experimental but will support accelerated ARM and PPC in the near future).

On June 12, 2019 9:59:51 PM GMT+02:00, William F Godoy [e-mail] wrote:

@ax3l any particular blosc version being targeted?

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/ornladios/ADIOS2/issues/1486#issuecomment-501433074

--
Sent from my smartphone.
Please excuse my brevity.

[williamfgc]

@ax3l just trying to squeeze blosc support into the upcoming release. Still there are a few issues that we haven't tested. See #1538

[chuckatkins]

@ax3l @williamfgc so can we close this as "fixed" for now and use new issues for problems and / or feature requests to the blosc compression added in #1538?

[williamfgc]

@chuckatkins I am fine either way, @ax3l your call.

[ax3l]

First of all: thanks a lot, that's awesome!

I would leave this open until the loop for >2GByte chunks (actually BLOSC_MAX_BUFFERSIZE) is added as well. We did the same in ADIOS1 as I just checked: https://github.com/ornladios/ADIOS/pull/135 (credits to @psychocoderHPC)

In case you look for a test case: in production we use zstd with 2-16 threads and bitshuffle pre-conditioning on a daily basis for best results.

[ax3l]

Just a note: blosc2 released its first beta release: https://github.com/Blosc/c-blosc2/releases/tag/v2.0.0-beta.1 https://blosc-doc.readthedocs.io

[williamfgc]

@ax3l thanks for sharing. Does blosc2 support 64-bit data sizes or is still 32-bit (2Gb) max?

[ax3l]

Seams to be the same size yet, also since they try to be quite compatible for easy adoption:

https://github.com/Blosc/c-blosc/blob/v1.17.0/blosc/blosc.h#L42
https://github.com/Blosc/c-blosc2/blob/v2.0.0-beta.1/blosc/blosc2.h#L66

Update: maybe schunks help us http://blosc.org/posts/blosc2-first-beta/

Just finished a Spack package for it and will submit it later on. ( https://github.com/spack/spack/pull/12430 )

[psychocoderHPC]

@williamfgc In the old issue, the developer said that Blosc2 will support 64bit buffer but what @ax3l posted looks like that this is not the case.

I opened an issue to get a clear answer from the devs https://github.com/Blosc/c-blosc2/issues/72

[williamfgc]

@psychocoderHPC thanks, that would give us a clear picture. I know I've been pushing this for a while, but my understanding is that the real value of this effort is to support 64-bit sizes either at the adios2 level or the blosc level. We'll need to try and see which approach makes more sense for your efforts.

[ax3l]

Ok, looks like super-chunks (schunks) are the way to go in c-blosc2 (example in the linked issue).

[ax3l]

We should also verify that we are able to read BP3 files with c-blosc transform (using our default zstd compression) in the ADIOS2 BP3-engine.