Json: serialize std::variant<...>

[MattiasEppler]

Hi.
I try to serialize std::variant to json.
I look for an easy and comfortable way. Is there any solution available?

Or any idea how to do this.

[nlohmann]

This section from the README should help you.

[nlohmann]

@MattiasEppler Do you need further assistance?

[MattiasEppler]

Hi.
I already read this section.

I do it in this way

string value = "";

if(const auto pdouble = std::get_if<double>(&data.Value))
    value = std::to_string(*pdouble);
else if(const auto pint= std::get_if<int>(&data.Value))
    value = std::to_string(*pint);

I thought maybe there is an easier solution.

But if not you can close this issue.

Thanks

[nlohmann]

I'm afraid not - at least I am not aware of a generic way to collect all the possible values from a variant.

[theodelrieu]

You could do something like:

namespace nlohmann {
template <typename ...Args>
struct adl_serializer<std::variant<Args...>> {
  static void to_json(json& j, std::variant<Args...> const& v) {
    std::visit([&](auto&& value) {
      j = std::forward<decltype(value)>(value);
    }, v);
  }
};
}

I don't think it's possible to provide a generic from_json here. Even if we serialize the index, it will be known at runtime, so switching on it would require to write N case, N being sizeof...(Args) - 1...

What I would do is setting the index in the json only when serializing an enclosing type:

struct Example {
  std::variant<int, std::string, float> var;
};

void to_json(json& j, Example const& e) {
  j = {{"index", e.var.index()}, {"variant", e.var}};
}

void from_json(json const&j, Example& e) {
  auto const index = j.at("index").get<int>();
  switch (index) {
    case 0:
      e.var = j.get<int>(); break;
    case 1:
      e.var = j.get<std::string>(); break;
    case 2:
      e.var = j.get<float>(); break;
    default: throw std::runtime_error{""};
  }
}

[theodelrieu]

On a second thought, there might be a way to make it generic, I'll try it out and post the results.

[theodelrieu]

Here is a complete example with the generic variant implementation.

Note that it is far from being efficient, it might be improved with something similar to for_each_args.

#include <iostream>
#include <stdexcept>
#include <string>
#include <variant>

#include <nlohmann/json.hpp>

using json = nlohmann::json;

namespace detail
{
template <std::size_t N>
struct variant_switch
{
  template <typename Variant>
  void operator()(int index, json const &value, Variant &v) const
  {
    if (index == N)
      v = value.get<std::variant_alternative_t<N, Variant>>();
    else
      variant_switch<N - 1>{}(index, value, v);
  }
};

template <>
struct variant_switch<0>
{
  template <typename Variant>
  void operator()(int index, json const &value, Variant &v) const
  {
    if (index == 0)
      v = value.get<std::variant_alternative_t<0, Variant>>();
    else
    {
      throw std::runtime_error(
          "while converting json to variant: invalid index");
    }
  }
};
}

namespace nlohmann
{
template <typename ...Args>
struct adl_serializer<std::variant<Args...>>
{
  static void to_json(json& j, std::variant<Args...> const& v)
  {
    std::visit([&](auto&& value) {
      j["index"] = v.index();
      j["value"] = std::forward<decltype(value)>(value);
    }, v);
  }

  static void from_json(json const& j, std::variant<Args...>& v)
  {
    auto const index = j.at("index").get<int>();
    ::detail::variant_switch<sizeof...(Args) - 1>{}(index, j.at("value"), v);
  }
};
}

struct A
{
  int val;
};

bool operator==(A const& lhs, A const& rhs) noexcept
{
  return lhs.val == rhs.val;
}

bool operator!=(A const& lhs, A const& rhs) noexcept
{
  return !(lhs == rhs);
}

void to_json(json& j, A a)
{
  j["val"] = a.val;
}

void from_json(json const& j, A& a)
{
  a.val = j.at("val").get<int>();
}

int main(int argc, char const *argv[])
{
  std::variant<int, std::string, float, A> v(A{42});

  nlohmann::json j(v);

  auto v2 = j.get<decltype(v)>();
  assert(v == v2);

  std::cout << j.dump(2) << std::endl;
  v = std::string("Hello");
  j = v;
  std::cout << j.dump(2) << std::endl;

  // invalidate the index
  j["index"] = 42;
  try
  {
    auto _ = j.get<decltype(v)>();
  }
  catch (std::runtime_error const& e)
  {
    std::cout << "Caught exception: " << e.what() << std::endl;
  }
}

[MattiasEppler]

Oh.... Thanks a lot for spending your time. I will try it.

[nlohmann]

@MattiasEppler Did you try?

[MattiasEppler]

Hi. Oh sorry.
Yes It works. Thanks to theodelrieu.

Will this be added to nlohmann\json?

[theodelrieu]

Will this be added to nlohmann\json?

I guess we could add it, I'd like users to be able to do the following:

std::variant<int, float, MyType> v(2);

json j = v;
auto i = j.get<int>();
auto v2 = j.get<decltype(v)>();

For this to work, we need to add a library-reserved field in the variant's alternative serialization:

template <typename ...Args>
void to_json(json& j, std::variant<Args...> const& v) {
  std::visit([&](auto const& a) { 
                      j = a;
                    }, v);
  j["__nlohmann_variant_index"] = v.index();
}

template <typename ...Args>
void from_json(json const& j, std::variant<Args...>& v) {
  auto const index = j.at("__nlohmann_variant_index").get<int>();
  // recursive ugliness to get the correct index...
}

[MattiasEppler]

In my case its not necessary to serialize the index.
So maybe it should be possible to set serializeIndex = false.

[theodelrieu]

After giving it a bit of thought, I think we should stay away from supporting std::variant natively. There is no correct solution to fit every use-case, i.e. the generic implementation I posted is very inefficient, it instantiates a lot of templates, it's more a hack than anything else.

So maybe it should be possible to set serializeIndex = false.

I'd like to avoid such compile-time toggle to enable/disable some specific conversion as well.

[nlohmann]

I agree with @theodelrieu. In a lot of cases, a variant is not the sole variable to be serialized, so type information may be derived implicitly from other members.

@MattiasEppler Is it OK to close the issue?

[MattiasEppler]

Hi. Yes its ok.

[ibkevg]

Edit: answered below

QUESTION: @theodelrieu @nlohmann after writing the to_json() and from_json() routines to serialize a std::variant I've found that the json constructor can't find my custom to_json() function. I wondered if the problem was that the to/from functions had to be in the std namespace but they don't compile if I do. I noticed above that you stuck the std::variant inside a struct but that isn't possible in my use case.

What do you recommend as the cleanest way to use std::variant? (I should add that my ultimate goal is to be able to json-ify a map<string, variant<uint64_t, double, string>>)
(edit: I'm realizing that an adl_serializer may be involved)

Here is a small example (which I used on the latest Visual Studio 15.9)

#include <nlohmann/json.hpp>
#include <variant>

using json = nlohmann::json;

typedef std::variant<int64_t, std::string>  TestVariant;

void to_json(json& j, TestVariant const& v)
{
    if (auto pInt64 = std::get_if<int64_t>(&v)) {
        j = json{ {"type", 0}, {"int64_t", *pInt64 } };
    else if (auto pStdStr = std::get_if<std::string>(&v)) {
        j = json{ {"type", 1}, {"string", *pStdStr} };
    } else {
        throw std::bad_variant_access();
    }
}

void from_json(json const& j, TestVariant& v)
{
    auto const type = j.at("type").get<int>();
    switch (type) {
    case 0:
        v = TestVariant(j.at("int64_t").get<uint64_t>()); break;
    case 1:
        v = TestVariant(j.at("string").get<std::string>()); break;
    default:
        throw std::runtime_error{ "Unexpected FT:Variant" };
    }
}

void test()
{
    auto v1 = TestVariant(100);

#if 0
    // Gives compile error C2440 below
    json j = v1;
#else
    // This compiles/works
    json j;
    to_json(j1, vInt);
#endif

    std::cout << j << std::endl;
    TestVariant v2 = j; // Gives compile error C2440 below
    cout << std::get<uint64_t>(v2) << endl;
}

error C2440: 'initializing': cannot convert from 'TestVariant' to 'nlohmann::basic_json'
note: No constructor could take the source type, or constructor overload resolution was ambiguous

error C2440: 'initializing': cannot convert from 'json' to 'std::variant'
note: No constructor could take the source type, or constructor overload resolution was ambiguous

ANSWER: here is the glue needed:

namespace nlohmann {

template <>
struct adl_serializer<TestVariant> {
    static void to_json(json& j, const TestVariant& value) {
        ::to_json(j, value);
    }

    static void from_json(const json& j, TestVariant& value) {
        ::from_json(j, value);
    }
};

}

[theodelrieu]

I would advise putting the conversion code inside the specialization, not calling your free function in the global namespace. Otherwise, it seems correct.

[zhongcy]

@nlohmann @theodelrieu Hi, nlohmann has not support C++17 Variant? Have any Variant examples?

[jeffreyscottgraham]

The alternative 'cereal" library has support for boost variant by adding a separate includes file for those that need it.

[jeffreyscottgraham]

Any update on supporting std::variant?
What about boost variant via a separate #include file?

[nlohmann]

Well, there is the code in https://github.com/nlohmann/json/issues/1261#issuecomment-426209912. Apart of this, I would be happy to see a PR!

[chakpongchung]

Also want to see a variant example here

[beached]

just a comment. JSON can have odd ways of expressing what type will be in a member. Take something similar to geojson where we have a type field that tells us what the data field is

[{
  "type": "point",
  "data": [1.0, 2.0]
},
{
  "type": "poly",
  "data": [1.0,2.0,3.0,4.0]
}]

They are the same in terms of JSON types, but when deserializing they could be in a c++ structure like

struct point { double x; double y; };
struct polygon { std::vector<point> points; };

struct geo {
  geo_type type;
  std::variant<point, polygon> data;
};