Lz4: Protection fault when decompressing highly compressible frames > 2GB

[kilodyne]

dctx->dictSize is being cast from a size_t to an int at points along the LZ4F_decompress->LZ4_decompress_generic call chain. When a frame containing extremely compressible data that will expand to more than 2GB is sent to LZ4F_decompress, a protection fault is generated in LZ4_decompress_generic at this line: memcpy(op, dictEnd - copySize, copySize);

I removed the (int) casts for the dictionary size throughout and that seems to fix the problem. I'm not familiar enough with the code to know if there are any side effects of that, however.

The code below is a new main() that can be pasted into frametest.c to reproduce the problem. If the random fill block is enabled there is no fault.

int main()
{
    for (size_t testSize = 1900LL MB; 1; testSize += 100LL MB)
    {
        LZ4F_preferences_t preferences;
        memset(&preferences, 0, sizeof(preferences));
        preferences.frameInfo.blockSizeID = LZ4F_max4MB;
        preferences.frameInfo.contentSize = testSize;
        size_t maxCompressed = LZ4F_compressFrameBound(testSize, &preferences);
        void* uncompressed = malloc(testSize);
        if (uncompressed == 0)
        {
            DISPLAY("No uncompressed buffer");
            return -1;
        }
        DISPLAY("Filling %llu\n", testSize);
        memset(uncompressed, 0, testSize);
#if 0
        char* pos = (char*)uncompressed;
        char* end = pos + testSize;
        while (pos + sizeof(int) < end)
        {
            *(int*)pos++ = rand();
        }
#endif
        void* compressed = malloc(maxCompressed);
        if (compressed == 0)
        {
            DISPLAY("No compressed buffer");
            return -1;
        }
        size_t compressedSize = LZ4F_compressFrame(compressed, maxCompressed, uncompressed, testSize, &preferences);
        if (LZ4F_isError(compressedSize))
        {
            DISPLAY("Compression failed: %llu\n", compressedSize);
            return -1;
        }
        DISPLAY("Compressed %llu to %llu\n", testSize, compressedSize);
        LZ4F_dctx* context;
        size_t result = LZ4F_createDecompressionContext(&context, LZ4F_VERSION);
        if (LZ4F_isError(result))
        {
            DISPLAY("Decompression context failed (error %llu : %s)", result, LZ4F_getErrorName(result));
            return -1;
        }
        size_t uncompressedSize = testSize;
        LZ4F_decompressOptions_t options;
        memset(&options, 0, sizeof(options));
        result = LZ4F_decompress(context, uncompressed, &uncompressedSize, compressed, &compressedSize, &options);
        if (LZ4F_isError(result))
        {
            DISPLAY("Decompression failed (error %llu : %s)", result, LZ4F_getErrorName(result));
            return -1;
        }
        if (uncompressedSize == testSize)
        {
            DISPLAY("Decompression succeeded\n");
        }
        LZ4F_freeDecompressionContext(context);
        free(compressed);
        free(uncompressed);
    }
    return 0;
}

[Cyan4973]

Thanks for the notification and very clear reproduction case @kilodyne !
We'll look into it !

[felixhandte]

@kilodyne, thanks for the report!

I confirmed the issue with your example reproduction and I've put up a fix. Should get that fixed for you soon!

[kilodyne]

@felixhandte great, thank you for working on this so quickly.