Json: msgpack unit test failures on ppc64 arch

[dkopecek]

Hi, I was trying to build[1] the newest release on multiple architectures (Fedora build architectures). There seems to be some msgpack related unit test failure on ppc64. See the attached build log:

json-ppc64-build.log.txt

[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=18476755

[nlohmann]

Thanks for the report. Could this be an endianess issue?

[dkopecek]

Yeah, that would be my guess too. Consider:

src/unit-msgpack.cpp:210: FAILED:
  CHECK( json::from_msgpack(result) == j )
with expansion:
  1 == 256

which looks like 0x0001 == 0x0100

[dkopecek]

ppc64 is big endian

[user1095108]

if you decide to do the swaps don't forget about bit_cast<>().

[nlohmann]

I currently do not have the means to test this. Is there a way to have a big endian system with Vagrant?

[user1095108]

Apparently the users, test this for you, you don't need to test yourself.
Otherwise, try qemu, install a big endian distro and start testing.

2017-04-01 1:08 GMT+02:00 Niels Lohmann [e-mail]:

I currently do not have the means to test this. Is there a way to have a
big endian system with Vagrant?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nlohmann/json/issues/524#issuecomment-290856054, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AH6jVM6a_CIFxnG3AQFUuhJPrTh_kJCBks5rrYdigaJpZM4Mh1xK
.

[nlohmann]

In the end, I need a test setup to at least verify a fix.

I could get qemu to work, but I failed to get a recent Linux with a recent compiler to work, let alone properly integrate it into my network. Hints greatly appreciated!

[user1095108]

Its very involved, I don't even remember how I did it myself, but I managed
at last. You should use Debian and follow some online tuts.

On Apr 3, 2017 22:31, "Niels Lohmann" [e-mail] wrote:

In the end, I need a test setup to at least verify a fix.

I could get qemu to work, but I failed to get a recent Linux with a recent
compiler to work, let alone properly integrate it into my network. Hints
greatly appreciated!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nlohmann/json/issues/524#issuecomment-291264540, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AH6jVNy1ltorQmhawYxdQLn7SktGNza1ks5rsVckgaJpZM4Mh1xK
.

[nlohmann]

Hey @dkopecek, could you please try https://github.com/nlohmann/json/blob/feature/manual_lexer/src/json.hpp ? I could not test it with a big-endian system myself.

It is a reimplementation of the binary readers/writers which should respect the byte order.

The main functions are:

// from http://stackoverflow.com/a/1001328/266378
static bool little_endianess()
{
    int num = 1;
    return (*reinterpret_cast<char*>(&num) == 1);
}

// writes a number of type T to vector v
template<typename T>
void write_number(T n)
{
    std::array<uint8_t, sizeof(T)> vec;
    std::memcpy(vec.data(), &n, sizeof(T));

    for (size_t i = 0; i < sizeof(T); ++i)
    {
        // reverse byte order prior to conversion if necessary
        if (is_little_endian)
        {
            v.push_back(vec[sizeof(T) - i - 1]);
        }
        else
        {
            v.push_back(vec[i]);
        }
    }
}

// reads a number of type T from input
template<typename T>
T get_number()
{
    std::array<uint8_t, sizeof(T)> vec;
    for (size_t i = 0; i < sizeof(T); ++i)
    {
        get();
        check_eof();

        // reverse byte order prior to conversion if necessary
        if (is_little_endian)
        {
            vec[sizeof(T) - i - 1] = static_cast<uint8_t>(current);
        }
        else
        {
            vec[i] = static_cast<uint8_t>(current);
        }
    }

    T result;
    std::memcpy(&result, vec.data(), sizeof(T));
    return result;
}

[user1095108]

This code is inefficient considering that endianess is a compile-time
constant. You could do away with the if.

On Apr 5, 2017 06:58, "Niels Lohmann" [e-mail] wrote:

Hey @dkopecek https://github.com/dkopecek, could you please try
https://github.com/nlohmann/json/blob/feature/manual_lexer/src/json.hpp ?
I could not test it with a big-endian system myself.

It is a reimplementation of the binary readers/writers which should
respect the byte order.

The main functions are:

// from http://stackoverflow.com/a/1001328/266378static bool little_endianess()
{
int num = 1;
return (*reinterpret_cast(&num) == 1);
}

// writes a number of type T to vector vtemplatevoid write_number(T n)
{
std::array std::memcpy(vec.data(), &n, sizeof(T));

for (size_t i = 0; i < sizeof(T); ++i)
{
    // reverse byte order prior to conversion if necessary
    if (is_little_endian)
    {
        v.push_back(vec[sizeof(T) - i - 1]);
    }
    else
    {
        v.push_back(vec[i]);
    }
}

}

// reads a number of type T from inputtemplate
T get_number()
{
std::array for (size_t i = 0; i < sizeof(T); ++i)
{
get();
check_eof();

    // reverse byte order prior to conversion if necessary
    if (is_little_endian)
    {
        vec[sizeof(T) - i - 1] = static_cast<uint8_t>(current);
    }
    else
    {
        vec[i] = static_cast<uint8_t>(current);
    }
}

T result;
std::memcpy(&result, vec.data(), sizeof(T));
return result;

}

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nlohmann/json/issues/524#issuecomment-291754409, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AH6jVK0SirO4aXnunKTw7iKvxEGUVaMCks5rsx9agaJpZM4Mh1xK
.

[nlohmann]

I could not find such a constant in the standard, and I would like to avoid dozens of #ifdefs to cover all compilers. I'm curious whether this fix works in the first place.

[user1095108]

you can still have a constexpr variable, if you wish, but the real bonus
would be to avoid the if statement.

Example:

#if defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ || \
defined(__BIG_ENDIAN__)
constexpr auto big_endian = true;

endif

2017-04-05 7:37 GMT+02:00 Niels Lohmann [e-mail]:

I could not find such a constant in the standard, and I would like to
avoid dozens of #ifdefs to cover all compilers. I'm curious whether this
fix works in the first place.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nlohmann/json/issues/524#issuecomment-291759332, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AH6jVHN1biNPfYQQ403e-hqpH0GsO3z7ks5rsyiDgaJpZM4Mh1xK
.

[nlohmann]

Is that macro portable?

[user1095108]

checks boost's endian.hpp and adapt the code there, it's more complete than
my example.

2017-04-05 8:51 GMT+02:00 Niels Lohmann [e-mail]:

Is that macro portable?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nlohmann/json/issues/524#issuecomment-291770626, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AH6jVKDeqG1WBhabe7_LRr2IWgsRUBQuks5rszn9gaJpZM4Mh1xK
.

[nlohmann]

This was exactly what I wanted to avoid.

[user1095108]

Avoid boost, yes, but you can still learn from their code.

2017-04-05 8:55 GMT+02:00 Niels Lohmann [e-mail]:

This was exactly what I wanted to avoid.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nlohmann/json/issues/524#issuecomment-291771329, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AH6jVI_xjEyzutaV-6xgTYEerhT3Ukgrks5rszrxgaJpZM4Mh1xK
.

[nlohmann]

I wanted to avoid "dozens of #ifsefs".

[dkopecek]

@nlohmann Cannot confirm that the fix works, because there are some compilation errors. See the attached
build.log.txt.

[nlohmann]

Sorry, I forgot I also changed the test suite. You need to check out the whole branch.

[dkopecek]

@nlohmann Unit tests fail on ppc64 and ppc64le:

ppc64le: https://kojipkgs.fedoraproject.org//work/tasks/1585/18791585/build.log
ppc64: https://kojipkgs.fedoraproject.org//work/tasks/1587/18791587/build.log

[nlohmann]

Thanks for the feedback. I really need a test system with ppc64.

[dkopecek]

@nlohmann Check out virt-builder:
https://developer.fedoraproject.org/tools/virt-builder/about.html
http://manpages.ubuntu.com/manpages/yakkety/man1/virt-builder.1.html

There are templates for Fedora ppc64, ppc64le, armv7, ... etc. (use virt-builder -l to list available templates).

[nlohmann]

This has nothing to do with the actual error, but the build script is odd:

+ doxygen doc/Doxyfile
warning: tag INPUT: input source `../src/json.hpp' does not exist
warning: tag INPUT: input source `index.md' does not exist
warning: source examples is not a readable file or directory... skipping.
warning: source images is not a readable file or directory... skipping.
warning: source ../src/json.hpp is not a readable file or directory... skipping.
warning: source index.md is not a readable file or directory... skipping.
sh: dot: command not found
error: Problems running dot: exit code=127, command='dot', arguments='"/builddir/build/BUILD/json-feature-manual_lexer/html/graph_legend.dot" -Tsvg -o "/builddir/build/BUILD/json-feature-manual_lexer/html/graph_legend.svg"'
error: Problem extracting size from SVG file /builddir/build/BUILD/json-feature-manual_lexer/html/graph_legend.svg
error: Style sheet 'css/mylayout.css' specified by HTML_EXTRA_STYLESHEET does not exist!

Instead of calling doxygen directly, one should call make -Cdoc instead.

[nlohmann]

I managed to set up a Debian 3.16.39 with powerpc architecture using qemu. With GCC 4.9.2 I could compile all tests. CBOR/MessagePack are still failing, but at least I can now debug myself ;)

[user1095108]

Are you sure that it is a big-endian distro? Last time I checked I only saw
little endian ones. Maybe you need a MIPS distro?

2017-04-08 23:06 GMT+02:00 Niels Lohmann [e-mail]:

I managed to set up a Debian 3.16.39 with powerpc architecture using qemu.
With GCC 4.9.2 I could compile all tests. CBOR/MessagePack are still
failing, but at least I can now debug myself ;)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nlohmann/json/issues/524#issuecomment-292745642, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AH6jVHiD9MWvLThmaC7BCniJKDHDFhcKks5rt_a6gaJpZM4Mh1xK
.

[nlohmann]

It seems so:

[nlohmann]

With the manual_lexer branch, there is only one test failing for CBOR and MessagePack, respectively:

It seems that the test itself assumes little endianess. I shall add a fix.

[nlohmann]

With the last commit (https://github.com/nlohmann/json/commit/6f99d5b2e978ee6ef448fe39a426ba12c101f8f9) all tests succeed.

[nlohmann]

@dkopecek Could you please check whether the fix works for you?

[user1095108]

Hey, don't forget MIPS, now that you're at it.

2017-04-10 21:13 GMT+02:00 Niels Lohmann [e-mail]:

@dkopecek https://github.com/dkopecek Could you please check whether
the fix works for you?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nlohmann/json/issues/524#issuecomment-293049677, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AH6jVMVUgEfhxzwmzwGiXQxZAtm5xxQFks5run9PgaJpZM4Mh1xK
.

[dkopecek]

@nlohmann Oops, sorry, I was planning to but totally forgot :-) I'll check it out tomorrow (CEST timezone) and let you know.

[nlohmann]

What's wrong with MIPS?

[user1095108]

It's big-endian, just like PPC.

2017-04-10 21:37 GMT+02:00 Niels Lohmann [e-mail]:

What's wrong with MIPS?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nlohmann/json/issues/524#issuecomment-293056046, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AH6jVPKZA6JyINcdK88_0wASBsCASFLKks5ruoUNgaJpZM4Mh1xK
.

[nlohmann]

Yes, and I proposed a fix for big endianess. I assume it also works for MIPS, but I already had troubles getting PPC to work.

[user1095108]

Try it if you have time, on MIPS.

2017-04-10 21:40 GMT+02:00 Niels Lohmann [e-mail]:

Yes, and I proposed a fix for big endianess. I assume it also works for
MIPS, but I already had troubles getting PPC to work.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nlohmann/json/issues/524#issuecomment-293056638, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AH6jVKrRqkQwPt97McsCXqCQbNCFjqVoks5ruoWTgaJpZM4Mh1xK
.

[nlohmann]

Sorry, i don't, and would appreciate help on this issue.

[dkopecek]

@nlohmann results:

x86_64: ok
ppc64le: ok
aarch64: ok
ppc64: FAIL
i686: ok
armv7hl: ok

but this time it's a different fail. see the attached ppc64-build.log.txt

[nlohmann]

Thanks for reporting. Could you rerun the tests with

make json_unit
test/json_unit "*"

This runs all tests and does not abort after the first failure.

[dkopecek]

Interesting. With that testing procedure, the tests don't fail on ppc64 and do fail on armv7hl. So there seems to be something non-deterministic in the tests.

armv7hl-build.log.2.txt
ppc64-build.log.2.txt

[nlohmann]

That is indeed strange. The tests should be (of course) deterministic, but since the ARM error is a SIGSEGV, maybe just a different call could have triggered it. Without debugging, I cannot tell much about the issue...

Could I ask you to try the following?

make test-conversions -Ctest
test/test-conversions -s

This should give us more information on when the test crashes.

[nlohmann]

(On the bright side, it seems as if the fix for the endianess issue seems to work)

[dkopecek]

Could I ask you to try the following?

make test-conversions -Ctest
test/test-conversions -s

Doesn't fail at all. Tried several times.

[nlohmann]

Strange... Could you try to execute the tests with Valgrind?

[dkopecek]

Here are the logs. x86_64 is ok but I've attached it anyway.

armv7hl-build.3.log.txt
ppc64-build.3.log.txt
x86_64-build.3.log.txt

[nlohmann]

I think I need to close this issue. I cannot reproduce the problem, and I can do little with the provided logs.