Openpmd-api: JSON

[ax3l]

A (serial-only) JSON IO-backend would be cool.

[ax3l]

A good backend provider for JSON is most likely the popular, header-only: https://github.com/nlohmann/json

[ax3l]

With a JSON backend, validation via a schema with existing tools / defintions such as JSON-LD or json-schema.org could be way easier. (On a side-note, HDF5 seems to explore schema-support as well: https://2018esipsummermeeting.sched.com/event/Eypl)

[ax3l]

Continued from https://github.com/openPMD/openPMD-api/pull/337#discussion_r216041160 on

floating point datatypes already have the same issue and I ignored that so far

It's ok if a backend does not support (conserve) all data types or might even cast all floats to double or all ints to int64_t.

In JSON, depending on the backend, we could signal different precision floats just by significant digits in the mantissa. (Ignore what languages make out of that, e.g. a Java app only knows one double type. In Python, we rely on Numpy to handle floats besides the one builtin.float type properly.)

With integers, we probably have to decide on a fixed type we cast all ints to. My idea is to avoid introducing additional annotations just for the sake of work-arounding JSON limitations, also because no standard JSON reader would understand such. But maybe you can convince me that this is useful at some points.

In the end, we will need the JSON backend in practice to (temporarily) transport int and double data with description of the meta data for small GUI-like applications. It's not a long term storage backend for primary data.

Of course, we need to make a section in the manual writing down limitations of specific backends (e.g. no MPI support or no support for certain types or cast of types during storage, etc.).

[franzpoeschel]

Ok, that sounds sensible.

or might even cast all floats to double or all ints to int64_t.

Not a huge issue in JSON anyway since values are pretty-printed rather than casted.

we could signal different precision floats just by significant digits in the mantissa.

Would the number of bytes be an alternative? That would work for integers, too, and if different FP standards are actually used, we are loosing information anyway.

With integers, we probably have to decide on a fixed type we cast all ints to.

After looking into things a bit more in detail, such a decision would actually be of relevance more for the frontend than the backend. With the current state of affairs, the user actively chooses the C++ datatype to be used:

auto chunk_data = E_x.loadChunk<double>(chunk_offset, chunk_extent);

The frontend then creates a chunk of data for the backend to write to which the backend receives in form of a void pointer. The backend is informed about the C++ internal type that the user chose in Parameter<Operation::READ_DATASET>:

template<>
struct Parameter< Operation::READ_DATASET > : public AbstractParameter
{
    Extent extent;
    Offset offset;
    Datatype dtype;
    std::shared_ptr< void > data;
};

So if I understand it correctly, the discussion is largely irrelevant for the JSON backend: For writing, I just let the JSON library pretty-print the values I get and for reading, I don't have any freedom of decision since the user picks the datatype.

[franzpoeschel]

Ok, there is a small catch: While readDataset gets the datatype as a parameter from the frontend, the backend is responsible for figuring out the correct datatype in readAttribute. Easy solutions would include
1) picking the datatype according to the stored datatype in the json file
2) (as you suggested above for integers) fix one float and one int type to cast all values to

"ulong": {
      "byte_width": 8,
      "datatype": "ULONG",
      "value": 64
    }

In the first solution, this ↑ would then be read as an unsigned long, irregardless of the byte width.
As a third (more fine-grained) solution, one could pick the smallest datatype holding byte_width bytes on the current platform.

Fwiw, the second and third solution approaches both contradict the current assumption behind some of the tests, e.g. SerialIOTest.cpp

REQUIRE(s.getAttribute("uint").dtype == Datatype::UINT);
REQUIRE(s.getAttribute("ulong").dtype == Datatype::ULONG);
REQUIRE(s.getAttribute("ulonglong").dtype == Datatype::ULONGLONG);

The tests currently assume that the first approach has been taken. (A pretty straightforward assumption before the refactoring of the integer type system #337, now maybe too much?)

[ax3l]

1. picking the datatype according to the stored datatype in the json file

I think that is totally fine for now, if we add light-weight annotations on top of that, that can be ignored if one casts to the defaults (probably double and long), it's is even better.

(is that what you draft in 2.? A type attribute per data set would keep/fake the type info but still casts the precision during pretty printing... Talking about pretty-printing, it adds not much value to remember the size of a "ulong":, etc.)

Fwiw, the second and third solution approaches both contradict the current assumption behind some of the tests
The tests currently assume ...

Yes, some tests are "datatype round-trip tests" for long-time storage formats such as HDF5 and ADIOS BP. We have to exclude JSON in those if we go for approach 1. without further annotations first. That's ok!

[ax3l]

Implemented in #338
@franzpoeschel rocks! :guitar: :rocket: :sparkles: :