Lz4: Error 44 : Unrecognized header : file cannot be decoded #192

[NoraDing]

When I try to decompress the *.lz4 log file which generated by spark running, the error shows as below

"Error 44 : Unrecognized header : file cannot be decoded"

So I want to know which headers are marked as 'Unrecognized', and why these errors show so frequently?

Can anyone take a look at this issue?
Much appreciate .

[Cyan4973]

I have no idea how spark generates LZ4 files.

My current guess is that Spark may produce well-formed LZ4 blocks but encapsulated into some Spark-specific header/footer, that only Spark can understand.

This feels related to : https://issues.apache.org/jira/browse/HADOOP-12990

[a-k-goncharov]

I'm struggling from the E44 too. The compressor is being launched on a ARM Cortex A7 single-board computer. The compressing code is really primitive:

const size_t outbufSize = LZ4_compressBound( insize );
size_t outlen = 0;

char *outbuf = new char[ outbufSize ];
if( outbuf == nullptr )
{
   std::cout << "[Sampler::compress] memory allocation error" << std::endl;
   return false;
}

outlen = LZ4_compress_default( inbuf, outbuf, insize, outbufSize );

std::cout << "[Sampler::compress] in-memory archive size is " << outlen
          << " / " << outbufSize
          << " bytes (source - " << insize << " bytes)" << std::endl;

bool ok = true;
int fd = open( (const char *)path,
                O_WRONLY | O_CREAT | O_TRUNC, // | O_DIRECT,
                S_IRWXU );
if( fd > 0 )
{

    size_t written = 0;

    while( written < outlen )
    {
        ssize_t single_write;

        single_write = write( fd,
                              (void *)( outbuf + written ),
                              outlen - written );

        if( single_write < 0 )
        {
            if( single_write == -1 )
            {
                std::cout << "[SAVE]: Not enough space on SD. "
                          << "Data aquisition has been stopped. "
                          << std::endl;
                std::cout << "Error code: "
                          << single_write << std::endl;
            }
            else
            {
                std::cout << "[SAVE]: write error: "
                          << single_write
                          << std::endl;
            }

            ok = false;
            break;
        }

        written += static_cast< size_t >( single_write );
    }

    (void) close( fd );
}


return ok;

First of all the data becomes compressed and right after that stored on a SD-card. There are no errors if compressing and decompressing of the data is being performed on the same computer. "No errors" - I mean the memcmp check is passed - don't know about the headers.
The issue occurs when I download the stored files onto "big" Linux-computer and try to decompress them there. I always get this E44 error.
Could you please give me some tips what's going on? The liblz4 version is the last - 1.8.3.

Thanks in advance!

[Cyan4973]

LZ4_compress() generates an LZ4-compressed block, without any metadata.
Locally, this probably does not matter, because metadata will likely be stored and transmitted by other means.

But when transferred to another computer, there is a problem : all metadata are lost.
To compensate, LZ4 defines an additional format, a __frame__, which encapsulates metadata, and makes it possible to transport and decompress data anywhere.

The frame is a different format.
It's generated by LZ4F_compressFrame() for example.

The lz4 command line utility is designed to decompress frames. Only. It can't decompress individual blocks.

If you do your own decoder yourself, you may be able to decompress blocks, using LZ4_decompress_safe() typically. That requires to "transport" metadata one way or another. The minimum amount of metadata needed to decompress a block is :

• the size of the compressed block
• the maximum possible size of the decompressed block

So it's not much, and is definitely possible. But that requires to be done explicitly by a program of your own.

[a-k-goncharov]

Thank you, sir!
You saved me hell a lot of time. Now everything works just fine and as expected.
I'm becoming a permanent library user.
Thank you soo much!

[sn0041]

当我尝试解压缩spark运行时生成的* .lz4日志文件时，错误显示如下

“错误44：无法识别的标头：文件无法解码”

因此，我想知道哪些标头被标记为“无法识别”，以及为什么这些错误如此频繁地显示？

谁能看一下这个问题？
非常感谢 。

我也遇到这个问题了 请问怎么解决

[Cyan4973]

As mentioned above, you have to make your own decoder program, and use LZ4_decompress_safe() (or a wrapper to it) to decode the block(s). You will also need the size of the compressed block, and the maximum possible size of the decompressed data, for each block.