Json: corrupted size vs. prev_size (aborted)

[maxsupermanhd]

Trying to read 37,5kb json feature.txt

Have malloc-ed chunk of memory of exact same size including NULL-terminating byte. Casting from void* to char* and passing to json::parse().

auto features = json::parse((char*)content);

No exception thrown, nothing bad expected to happen. Instead was failing with abort.

Backtrace available here: https://pastebin.com/qTMyiMUp

Compiled, linked and made on Ubuntu 18.04.04 with CMake and make.

[nlohmann]

Can you please try to create a std::string and check if it works then?

[maxsupermanhd]

Same issue with std::string.
Log: https://pastebin.com/JWShSxkH

P.S. string seems to be valid and shows up full 37544 bytes.

[nlohmann]

I can't reproduce this. This code runs fine:

#include <iostream>
#include <fstream>
#include "json.hpp"

using json = nlohmann::json;

int main()
{
    std::ifstream f("/Users/niels/Downloads/feature.txt");
    auto j = json::parse(f);
    std::cout << j << std::endl;
}

Can you share the code calling the parser? Can you run the code with address sanitizer?

[maxsupermanhd]

https://github.com/maxsupermanhd/WMT/blob/JSON-test/lib/wmt.cpp#L1078
Here you go, build howto is in README.md. I can send you core dump if you wish.
File required: 10c-da-bdctw10-v1max.zip
Run with ./bin/wmt 10c-da-bdctw10-v1max.zip -z 5 -feh -v999

[nlohmann]

I could compile and run without errors. This is the output:
log.txt

The only odd line (in my eyes) was:

12:44:56 ERROR ../lib/wmt.cpp:924: No id field in structure JSON! Counting objects...

[maxsupermanhd]

Well, on your system it works... :thinking: It was failing on line 71... Did you built JSON-test branch? If so maybe I should update headers up to master. I'll investigate more...

P.S. This line supposed to be here, some files have this field, some not.

[nlohmann]

I now tried the JSON-test branch with Linux (previous test was macOS). With ASAN, I get the following error:

==3182==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x621000003b15 at pc 0x7ffff74b9c14 bp 0x7ffffffcc480 sp 0x7ffffffcc470
WRITE of size 1 at 0x621000003b15 thread T0
    #0 0x7ffff74b9c13 in WMT_ReadDroidsJSON(WZmap*) /home/vagrant/WMT/lib/wmt.cpp:1227
    #1 0x7ffff74bb1da in WMT_ReadDroidsFile(WZmap*) /home/vagrant/WMT/lib/wmt.cpp:1322
    #2 0x7ffff74bbc8b in WMT_ReadMap(char*, WZmap*) /home/vagrant/WMT/lib/wmt.cpp:1389
    #3 0x55555555c75f in main /home/vagrant/WMT/src/main.cpp:280
    #4 0x7ffff70641e2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x271e2)
    #5 0x55555555978d in _start (/home/vagrant/WMT/build/bin/wmt+0x578d)

0x621000003b15 is located 0 bytes to the right of 4629-byte region [0x621000002900,0x621000003b15)
allocated by thread T0 here:
    #0 0x7ffff7697ae8 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10dae8)
    #1 0x7ffff751d681 in def_alloc_func /home/vagrant/WMT/lib/miniz.h:1443
    #2 0x7ffff7544a7e in mz_zip_reader_extract_to_heap /home/vagrant/WMT/lib/miniz.h:5464
    #3 0x7ffff755290a in zip_entry_read /home/vagrant/WMT/lib/zip.cpp:669
    #4 0x7ffff74b9b0d in WMT_ReadDroidsJSON(WZmap*) /home/vagrant/WMT/lib/wmt.cpp:1221
    #5 0x7ffff74bb1da in WMT_ReadDroidsFile(WZmap*) /home/vagrant/WMT/lib/wmt.cpp:1322
    #6 0x7ffff74bbc8b in WMT_ReadMap(char*, WZmap*) /home/vagrant/WMT/lib/wmt.cpp:1389
    #7 0x55555555c75f in main /home/vagrant/WMT/src/main.cpp:280
    #8 0x7ffff70641e2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x271e2)

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/vagrant/WMT/lib/wmt.cpp:1227 in WMT_ReadDroidsJSON(WZmap*)
Shadow bytes around the buggy address:
  0x0c427fff8710: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c427fff8720: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c427fff8730: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c427fff8740: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c427fff8750: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c427fff8760: 00 00[05]fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c427fff8770: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c427fff8780: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c427fff8790: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c427fff87a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c427fff87b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==3182==ABORTING

So it seems there is an issue executing

        ssize_t readed = zip_entry_read(map->zip, (void**)&content, &readlen);

Then I think that invalid buffer explodes inside the JSON library, but the error is not related to it.