Pybind11: Need help writting bindings for nlohmann::json

[jdumas]

I'm trying to write pybind11 bindings for nlohmann::json, to be able to seamlessly pass json-like arguments between python and C++. In Python, there isn't really a "json" type: it's just nested lists/tuples/dictionaries. On the C++ side, I have a function taking a nlohmann::json p as parameter.

The difficulty that I am facing here is twofold:

1. I don't want to end up with a custom module.json class on the Python side. So there should be implicit casts both ways between C++ and native Python types.
2. We need to be able to build an instance of nlohmann::json from any python object, so from an arbitrary py::object. This seems more feasible but it implies digging low-level into the python C API.

There was a discussion on the library side about python bindings, but it never really took off, and the example binding provided (with JsonCpp) is pretty outdated.

I just want something easy to use for now, and I don't care much about copying data around. Any suggestion?

[bstaletic]

I've given this a thought myself. nlohmann::json tries to behave like an STL container and for those we already have pybind11/stl.h and pyind11/stl_bind.h.

I don't want to end up with a custom module.json class on the Python side. So there should be implicit casts both ways between C++ and native Python types.

While 100% implicit conversion sounds nice, it comes at a slight performance penalty and will require custom casters. For example, you will have to explain to pybind how to translate json objects and for this, if you want implicit conversions, you will have to dive into the CPython API and write a custom caster.

Another down site of stl.h and implicit conversions is:

py_list = [ 1, 2 ]
cpp_clear_list( py_list )
print( py_list  ) # prints [1, 2]

In other words, changes to STL types bound with stl.h can not be mad opaque.

Same example with stl_bind.h, which requires a bit more boilerplate on both sides, looks like this:

py_list = IntVector( [ 1 ] ) # Custom type created with pybind11::bind_vector from stl_bind.h
py_list.append( 2 ) # But still behaves like a python list
cpp_clear_list( py_list )
print( py_list ) # prints IntVector[] in case you made IntVector opaque
                 # prints IntVector[1, 2] in case you skipped calling PYBIND11_MAKE_OPAQUE

Again, stl_bind.h will also avoid a lot of copies, because there is no conversion between C++ vector and Python list, instead, IntVector (or StringVector or whatever else) is actually a reference to an std::vector like object on the C++ side.

We need to be able to build an instance of nlohmann::json from any python object, so from an arbitrary py::object. This seems more feasible but it implies digging low-level into the python C API.

Are we sure this is the right way to go? What's wrong with requiring that, in order to get a json object on the C++ side, you have to pass python dictionary from the Python side? Similarly, for C++ json array, you have to have a python list on thte other side.

Considering that json arrays in nlohmann::json are std::vectors under the hood, maybe even some types from stl_bind.h could be used, but that is a stretch.

[jdumas]

Are we sure this is the right way to go? What's wrong with requiring that, in order to get a json object on the C++ side, you have to pass python dictionary from the Python side? Similarly, for C++ json array, you have to have a python list on the other side.

Well, but you can build a nlohmann::json from a disct, a list, a nullptr, or even another basic type in C++. I would like to it to be the same on the Python side (i.e. everything that I can pass to json.dumps(...) in python, I should be able to pass to the binding of a C++ function that takes a nlohmann::json argument).

Is there any reason that your first example still prints [1, 2] at the end? Is this because the object is copied and thus not cleaned on the C++ side?

I'm just listing a wishlist at the moment. Doesn't seems there is an ideal solution at the moment, based on what you describe. Again, I care more about ease of use than performances at the moment, since my json structures are usually pretty small and the goal is convenience.

[bstaletic]

Well, but you can build a nlohmann::json from a disct, a list, a nullptr, or even another basic type in C++. I would like to it to be the same on the Python side (i.e. everything that I can pass to json.dumps(...) in python, I should be able to pass to the binding of a C++ function that takes a nlohmann::json argument).

Fair point, but invalid input to json.dumps() should stay invalid for our hypothetical nlohmann_json.dumps().

Is there any reason that your first example still prints [1, 2] at the end? Is this because the object is copied and thus not cleaned on the C++ side?

There is a good reason.

• You've created a list on python side.
• You've passed that list to a C++ function that takes a std::vector<int>
• You're using stl.h, so implicit conversion automagically works.
• Internally, pybind11 copies the data into the C++ vector.
• From this point the vector is completely disconnected from the Python side, so any in-place object mutation are not reflected on the python side.
  
  
  
  • Alternative is to forget about in-place mutations and always return a complete list.
  
  
• The reason behind this is that tracking changes to these objects would create a big overhead and kill performance.

So how can we change the first example so it would work?

py_list = [ 1, 2 ]
py_list = cpp_clear( py_list )
print( py_list ) # prints []

Note that every time you go from Python's list to C++'s you will have a copy. Same goes for vector to list.

[jdumas]

But couldn't the object be copied back into the python list, thus modifying its content? Well, it may not be desirable from a performance standpoint, but technically this could be achieved no?

And yes passing an invalid input to json.dumps() should also fail the autoconversion to nlohmann::json.

[bstaletic]

But couldn't the object be copied back into the python list, thus modifying its content? Well, it may not be desirable from a performance standpoint, but technically this could be achieved no?

That's exactly what the second line of my modified example does. You pass a list, it gets converted to a vector, you modify the vector in C++, you return the vector by value, it gets copied into a list on the left hand side of the assignment.

[jdumas]

Ah yes. I'm fine with that actually. I was asking about copying back in place when doing the conversion from C++ vector back to the python type at the end of the cpp_clear function, but the solution of returning the modified object is also very usable (just a caveat one needs to keep in mind).

[bstaletic]

Once again, that would require a lot of bookkeeping and isn't something pybind11 can do with implicit STL conversion from stl.h. On the other hand, stl_bind.h requires more boilerplate and need bindings for fully specialized STL types, but allows a lot of copies to be avoided.
There is no silver bullet available, nor do I think is possible.

[martinRenou]

@jdumas I also worked a few days ago on a simple implicit conversion from nlohmann::json to py::object and from py::object to nlohmann::json. Here is what I have:

namespace nlohmann
{

    py::object adl_serializer<py::object>::from_json(const json& j)
    {
        py::module py_json = py::module::import("json");

        return py_json.attr("loads")(j.dump());
    }

    void adl_serializer<py::object>::to_json(json& j, py::object obj)
    {
        py::module py_json = py::module::import("json");

        j = nl::json::parse(static_cast<std::string>(
            py::str(py_json.attr("dumps")(obj))
        ));
    }

}

This way you can assign any py::object to nlohmann::json and the other way around. The problem with this implementation is that there is a conversion to string. I plan to improve it soon https://github.com/QuantStack/xeus-python/issues/25.

[jdumas]

Thanks, that's a useful baseline to have! I'll update here if I manage to implement a proper bindings with all this.

[martinRenou]

I have a more advanced version now :)

namespace nlohmann
{

    namespace detail
    {
        py::object from_json_impl(const json& j)
        {
            if (j.is_null())
            {
                return py::none();
            }
            if (j.is_boolean())
            {
                return py::bool_(j.get<bool>());
            }
            if (j.is_number())
            {
                double number = j.get<double>();
                if (number == std::floor(number))
                {
                    return py::int_(j.get<int>());
                }
                else
                {
                    return py::float_(number);
                }
            }
            if (j.is_string())
            {
                return py::str(j.get<std::string>());
            }
            if (j.is_array())
            {
                py::list obj;
                for (const auto& el: j)
                {
                    obj.attr("append")(from_json_impl(el));
                }
                return obj;
            }
            if (j.is_object())
            {
                py::dict obj;
                for (json::const_iterator it = j.cbegin(); it != j.cend(); ++it)
                {
                    obj[py::str(it.key())] = from_json_impl(it.value());
                }
                return obj;
            }
        }

        json to_json_impl(py::handle obj)
        {
            if (obj.is_none())
            {
                return nullptr;
            }
            if (py::isinstance<py::bool_>(obj))
            {
                return obj.cast<bool>();
            }
            if (py::isinstance<py::int_>(obj))
            {
                return obj.cast<long>();
            }
            if (py::isinstance<py::float_>(obj))
            {
                return obj.cast<double>();
            }
            if (py::isinstance<py::str>(obj))
            {
                return obj.cast<std::string>();
            }
            if (py::isinstance<py::tuple>(obj) || py::isinstance<py::list>(obj))
            {
                json out;
                for (py::handle value: obj)
                {
                    out.push_back(to_json_impl(value));
                }
                return out;
            }
            if (py::isinstance<py::dict>(obj))
            {
                json out;
                for (py::handle key: obj)
                {
                    out[key.cast<std::string>()] = to_json_impl(obj[key]);
                }
                return out;
            }
            throw std::runtime_error("to_json not implemented for this type of object: " + obj.cast<std::string>());
        }
    }

    py::object adl_serializer<py::object>::from_json(const json& j)
    {
        return detail::from_json_impl(j);
    }

    void adl_serializer<py::object>::to_json(json& j, py::object obj)
    {
        j = detail::to_json_impl(obj);
    }

}

And it comes with some tests using gtest:

TEST(pyobject_tojson, none)
{
    py::scoped_interpreter guard;
    py::object obj = py::none();
    nl::json j = obj;

    ASSERT_TRUE(j.is_null());
}

TEST(pyobject_tojson, bool_)
{
    py::scoped_interpreter guard;
    py::object obj = py::bool_(false);
    nl::json j = obj;

    ASSERT_TRUE(j.is_boolean());
    ASSERT_FALSE(j.get<bool>());
}

TEST(pyobject_tojson, number)
{
    py::scoped_interpreter guard;
    py::object obj = py::int_(36);
    nl::json j = obj;

    ASSERT_TRUE(j.is_number());
    ASSERT_EQ(j.get<int>(), 36);

    py::object obj2 = py::float_(36.37);
    nl::json j2 = obj2;

    ASSERT_TRUE(j2.is_number());
    ASSERT_EQ(j2.get<double>(), 36.37);
}

TEST(pyobject_tojson, string)
{
    py::scoped_interpreter guard;
    py::object obj = py::str("Hello");
    nl::json j = obj;

    ASSERT_TRUE(j.is_string());
    ASSERT_EQ(j.get<std::string>(), "Hello");
}

TEST(pyobject_tojson, list)
{
    py::scoped_interpreter guard;
    py::object obj = py::list();
    obj.attr("append")(py::int_(36));
    obj.attr("append")(py::str("Hello World"));
    obj.attr("append")(py::bool_(false));
    nl::json j = obj;

    ASSERT_TRUE(j.is_array());
    ASSERT_EQ(j[0].get<int>(), 36);
    ASSERT_EQ(j[1].get<std::string>(), "Hello World");
    ASSERT_EQ(j[2].get<bool>(), false);
}

TEST(pyobject_tojson, tuple)
{
    py::scoped_interpreter guard;
    py::object obj = py::make_tuple(1234, "hello", false);
    nl::json j = obj;

    ASSERT_TRUE(j.is_array());
    ASSERT_EQ(j[0].get<int>(), 1234);
    ASSERT_EQ(j[1].get<std::string>(), "hello");
    ASSERT_EQ(j[2].get<bool>(), false);
}

TEST(pyobject_tojson, dict)
{
    py::scoped_interpreter guard;
    py::object obj = py::dict("number"_a=1234, "hello"_a="world");
    nl::json j = obj;

    ASSERT_TRUE(j.is_object());
    ASSERT_EQ(j["number"].get<int>(), 1234);
    ASSERT_EQ(j["hello"].get<std::string>(), "world");
}

TEST(pyobject_tojson, nested)
{
    py::scoped_interpreter guard;
    py::object obj = py::dict(
        "list"_a=py::make_tuple(1234, "hello", false),
        "dict"_a=py::dict("a"_a=12, "b"_a=13),
        "hello"_a="world",
        "world"_a=py::none()
    );
    nl::json j = obj;

    ASSERT_TRUE(j.is_object());
    ASSERT_EQ(j["list"][0].get<int>(), 1234);
    ASSERT_EQ(j["list"][1].get<std::string>(), "hello");
    ASSERT_EQ(j["list"][2].get<bool>(), false);
    ASSERT_EQ(j["dict"]["a"].get<int>(), 12);
    ASSERT_EQ(j["dict"]["b"].get<int>(), 13);
    ASSERT_EQ(j["world"], nullptr);
}

TEST(pyobject_fromjson, none)
{
    py::scoped_interpreter guard;
    nl::json j = "null"_json;
    py::object obj = j;

    ASSERT_TRUE(obj.is_none());
}

TEST(pyobject_fromjson, bool_)
{
    py::scoped_interpreter guard;
    nl::json j = "false"_json;
    py::object obj = j;

    ASSERT_TRUE(py::isinstance<py::bool_>(obj));
    ASSERT_FALSE(obj.cast<bool>());
}

TEST(pyobject_fromjson, number)
{
    py::scoped_interpreter guard;
    nl::json j = "36"_json;
    py::object obj = j;

    ASSERT_TRUE(py::isinstance<py::int_>(obj));
    ASSERT_EQ(obj.cast<int>(), 36);

    nl::json j2 = "36.2"_json;
    py::object obj2 = j2;

    ASSERT_TRUE(py::isinstance<py::float_>(obj2));
    ASSERT_EQ(obj2.cast<double>(), 36.2);
}

TEST(pyobject_fromjson, string)
{
    py::scoped_interpreter guard;
    nl::json j = "\"Hello World!\""_json;
    py::object obj = j;

    ASSERT_TRUE(py::isinstance<py::str>(obj));
    ASSERT_EQ(obj.cast<std::string>(), "Hello World!");
}

TEST(pyobject_fromjson, list)
{
    py::scoped_interpreter guard;
    nl::json j = "[1234, \"Hello World!\", false]"_json;
    py::object obj = j;

    ASSERT_TRUE(py::isinstance<py::list>(obj));
    ASSERT_EQ(py::list(obj)[0].cast<int>(), 1234);
    ASSERT_EQ(py::list(obj)[1].cast<std::string>(), "Hello World!");
    ASSERT_EQ(py::list(obj)[2].cast<bool>(), false);
}

TEST(pyobject_fromjson, dict)
{
    py::scoped_interpreter guard;
    nl::json j = "{\"a\": 1234, \"b\":\"Hello World!\", \"c\":false}"_json;
    py::object obj = j;

    ASSERT_TRUE(py::isinstance<py::dict>(obj));
    ASSERT_EQ(py::dict(obj)["a"].cast<int>(), 1234);
    ASSERT_EQ(py::dict(obj)["b"].cast<std::string>(), "Hello World!");
    ASSERT_EQ(py::dict(obj)["c"].cast<bool>(), false);
}

TEST(pyobject_fromjson, nested)
{
    py::scoped_interpreter guard;
    nl::json j = R"({
        "baz": ["one", "two", "three"],
        "foo": 1,
        "bar": {"a": 36, "b": false},
        "hey": null
    })"_json;
    py::object obj = j;

    ASSERT_TRUE(py::isinstance<py::dict>(obj));
    ASSERT_TRUE(py::isinstance<py::list>(py::dict(obj)["baz"]));
    ASSERT_TRUE(py::isinstance<py::int_>(py::dict(obj)["foo"]));
    py::list baz = py::dict(obj)["baz"];
    py::int_ foo = py::dict(obj)["foo"];
    py::dict bar = py::dict(obj)["bar"];
    ASSERT_EQ(baz[0].cast<std::string>(), "one");
    ASSERT_EQ(baz[1].cast<std::string>(), "two");
    ASSERT_EQ(baz[2].cast<std::string>(), "three");
    ASSERT_EQ(foo.cast<int>(), 1);
    ASSERT_EQ(bar["a"].cast<int>(), 36);
    ASSERT_FALSE(bar["b"].cast<bool>());
    ASSERT_TRUE(py::dict(obj)["hey"].is_none());
}

[martinRenou]

If you think I can improve it at some places please tell me :)

[jdumas]

Amazing! I just hooked it up and seems to work like a charm =)
May be worth to integrate it directly into pybind11 imho (like the bindings for Eigen types).
I'll close this for now, as the initial issue is pretty much solved.

[jdumas]

Hmm ok so I've tried to register the json to a module:

    py::class_<json>(m, "json")
        .def(py::init<>())
        .def(py::init<py::object>())
    ;
    py::implicitly_convertible<py::object, json>();

Now if I try to define a function that returns a json object like so

    m.def("foo", []() { json j; return j;});

Then I get the following error message:

/home/jdumas/code/pybind11/include/pybind11/cast.h:776:38: error: invalid use of incomplete type ‘struct pybind11::detail::is_copy_constructible<nlohmann::basic_json<>, void>’
 template <typename Container> struct is_copy_constructible<Container, enable_if_t<all_of<
                                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         std::is_copy_constructible<Container>,
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         std::is_same<typename Container::value_type &, typename Container::reference>
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     >::value>> : public is_copy_constructible<typename Container::value_type> {};
     ~~~~~~~~~~                        
/home/jdumas/code/pybind11/include/pybind11/cast.h:776:38: note: declaration of ‘struct pybind11::detail::is_copy_constructible<nlohmann::basic_json<>, void>’

Any idea where the error comes from? I'm pretty sure nl::json is copy-constructible...

[martinRenou]

Hum.. You don't need to define a json class in order to be able to return a nl::json from a function exposed to python. It should be automatically converted to py::object. Did you need a json class?

[jdumas]

Ah, good point. The problem is that the line m.def("foo", []() { json j; return j;}); still doesn't compile with the error message above.

I suppose the problem comes from this piece of code:, in pybind11/cast.h:

// Specialization for types that appear to be copy constructible but also look like stl containers
// (we specifically check for: has `value_type` and `reference` with `reference = value_type&`): if
// so, copy constructability depends on whether the value_type is copy constructible.
template <typename Container> struct is_copy_constructible<Container, enable_if_t<all_of<
        std::is_copy_constructible<Container>,
        std::is_same<typename Container::value_type &, typename Container::reference>
    >::value>> : is_copy_constructible<typename Container::value_type> {};

Basically, in the case of nlohmann::json, we have nlohmann::json === nlohmann::json::value_type, so the class seems to simply inherit from itself without ever going into std::is_copy_constructible.

[martinRenou]

Also, I think you shouldn't need an py::implicitly_convertible<py::object, json>();. If you only add the piece of code I gave, it should just work out of the box :) The tests are passing without the need of anything else.

[martinRenou]

Basically the

    py::object adl_serializer<py::object>::from_json(const json& j)
    {
        return detail::from_json_impl(j);
    }

    void adl_serializer<py::object>::to_json(json& j, py::object obj)
    {
        j = detail::to_json_impl(obj);
    }

part of the code will enable the automatic conversion

[jdumas]

While this compiles ok, I'm running into the following error at runtime on the python side, when calling a function that returns a nl::json type:

TypeError: Unable to convert function return value to a Python type! The signature was
    () -> nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer>
Did you forget to `#include <pybind11/stl.h>`? Or <pybind11/complex.h>,
<pybind11/functional.h>, <pybind11/chrono.h>, etc. Some automatic
conversions are optional and require extra headers to be included
when compiling your pybind11 module.

If I register a class the problem is gone (but I have my own json class, which is less ideal). Note that I did include pybind11/stl.h, but not the other ones.

Back to my initial problem, if I modify the code in pybind11/cast,h to the following:

template <typename Container> struct is_copy_constructible<Container, enable_if_t<all_of<
        std::is_copy_constructible<Container>,
        std::is_same<typename Container::value_type &, typename Container::reference>,
        negation<std::is_same<typename Container::value_type, Container>>
    >::value>> : is_copy_constructible<typename Container::value_type> {};

Then it compiles properly for the type nl::json.

EDIT: My bad, I've removed the last block of code you've posted because it doesn't compile on gcc, failing with the following error message:

error: specializing member ‘nlohmann::adl_serializer<pybind11::object>::from_json’ requires ‘template<>’ syntax

What version of nlohmann/json are you using?

[martinRenou]

3.3.0. It's weird, can you try adding template <> just before the line implementing from_json? Can you give me the code you are using so that I can try?

[jdumas]

Sure. Here is a minimal example:

main.cpp

////////////////////////////////////////////////////////////////////////////////
#include "json.hpp"
#include <pybind11/pybind11.h>
#include <pybind11/eigen.h>
#include <pybind11/cast.h>
#include <pybind11/stl.h>
#include <iostream>
#include <type_traits>
////////////////////////////////////////////////////////////////////////////////

namespace py = pybind11;
namespace nl = nlohmann;

using namespace pybind11::literals;

////////////////////////////////////////////////////////////////////////////////

namespace nlohmann
{

    namespace detail
    {
        py::object from_json_impl(const json& j)
        {
            if (j.is_null())
            {
                return py::none();
            }
            if (j.is_boolean())
            {
                return py::bool_(j.get<bool>());
            }
            if (j.is_number())
            {
                double number = j.get<double>();
                if (number == std::floor(number))
                {
                    return py::int_(j.get<int>());
                }
                else
                {
                    return py::float_(number);
                }
            }
            if (j.is_string())
            {
                return py::str(j.get<std::string>());
            }
            if (j.is_array())
            {
                py::list obj;
                for (const auto& el: j)
                {
                    obj.attr("append")(from_json_impl(el));
                }
                return obj;
            }
            if (j.is_object())
            {
                py::dict obj;
                for (json::const_iterator it = j.cbegin(); it != j.cend(); ++it)
                {
                    obj[py::str(it.key())] = from_json_impl(it.value());
                }
                return obj;
            }
        }

        json to_json_impl(py::handle obj)
        {
            if (obj.is_none())
            {
                return nullptr;
            }
            if (py::isinstance<py::bool_>(obj))
            {
                return obj.cast<bool>();
            }
            if (py::isinstance<py::int_>(obj))
            {
                return obj.cast<long>();
            }
            if (py::isinstance<py::float_>(obj))
            {
                return obj.cast<double>();
            }
            if (py::isinstance<py::str>(obj))
            {
                return obj.cast<std::string>();
            }
            if (py::isinstance<py::tuple>(obj) || py::isinstance<py::list>(obj))
            {
                json out;
                for (py::handle value: obj)
                {
                    out.push_back(to_json_impl(value));
                }
                return out;
            }
            if (py::isinstance<py::dict>(obj))
            {
                json out;
                for (py::handle key: obj)
                {
                    out[key.cast<std::string>()] = to_json_impl(obj[key]);
                }
                return out;
            }
            throw std::runtime_error("to_json not implemented for this type of object: " + obj.cast<std::string>());
        }
    }

    template<>
    py::object adl_serializer<py::object>::from_json(const json& j)
    {
        return detail::from_json_impl(j);
    }

    template<>
    void adl_serializer<py::object>::to_json(json& j, py::object obj)
    {
        j = detail::to_json_impl(obj);
    }

}

////////////////////////////////////////////////////////////////////////////////

PYBIND11_MODULE(pyjson, m) {
    m.doc() = "pyjson module"; // optional module docstring

    // Dummy json
    m.def("dummy_json", []() {
        nl::json j = {
            {"pi", 3.141},
            {"happy", true},
            {"name", "Niels"},
            {"nothing", nullptr},
            {"answer", {
                {"everything", 42}
            }},
            {"list", {1, 0, 2}},
            {"object", {
                {"currency", "USD"},
                {"value", 42.99}
            }}
        };
        return j;
    }, "returns a dummy json object");

}

CMakeLists.txt

cmake_minimum_required(VERSION 3.1)
project(pyjson)

find_package(PythonInterp 3 QUIET)
find_package(Eigen3 REQUIRED)
add_subdirectory(pybind11)

pybind11_add_module(pyjson main.cpp)
target_link_libraries(pyjson PUBLIC Eigen3::Eigen)

# Generate position independent code
set_target_properties(pyjson PROPERTIES POSITION_INDEPENDENT_CODE ON)

# Output location
set_target_properties(pyjson PROPERTIES OUTPUT_NAME "pyjson")
set_target_properties(pyjson PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${PROJECT_SOURCE_DIR})

If I add the template <> as in the code posted over here I get the following error message:

[ 50%] Building CXX object CMakeFiles/pyjson.dir/main.cpp.o
/home/jdumas/public/test_pyjson/main.cpp:114:16: error: template-id ‘from_json<>’ for ‘pybind11::object nlohmann::adl_serializer<pybind11::object>::from_json(const json&)’ does not match any template declaration
     py::object adl_serializer<py::object>::from_json(const json& j)
                ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/jdumas/public/test_pyjson/main.cpp:114:67: note: saw 1 ‘template<>’, need 2 for specializing a member function template
     py::object adl_serializer<py::object>::from_json(const json& j)
                                                                   ^
/home/jdumas/public/test_pyjson/main.cpp:120:10: error: template-id ‘to_json<>’ for ‘void nlohmann::adl_serializer<pybind11::object>::to_json(nlohmann::json&, pybind11::object)’ does not match any template declaration
     void adl_serializer<py::object>::to_json(json& j, py::object obj)
          ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/jdumas/public/test_pyjson/main.cpp:120:69: note: saw 1 ‘template<>’, need 2 for specializing a member function template
     void adl_serializer<py::object>::to_json(json& j, py::object obj)
                                                                     ^
/home/jdumas/public/test_pyjson/main.cpp: In function ‘pybind11::object nlohmann::detail::from_json_impl(const json&)’:
/home/jdumas/public/test_pyjson/main.cpp:67:9: warning: control reaches end of non-void function [-Wreturn-type]
         }
         ^
make[2]: *** [CMakeFiles/pyjson.dir/build.make:63: CMakeFiles/pyjson.dir/main.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:73: CMakeFiles/pyjson.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

If I don't put the template <> I get:

[ 50%] Building CXX object CMakeFiles/pyjson.dir/main.cpp.o
/home/jdumas/public/test_pyjson/main.cpp:113:16: error: specializing member ‘nlohmann::adl_serializer<pybind11::object>::from_json’ requires ‘template<>’ syntax
     py::object adl_serializer<py::object>::from_json(const json& j)
                ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/jdumas/public/test_pyjson/main.cpp:118:10: error: specializing member ‘nlohmann::adl_serializer<pybind11::object>::to_json’ requires ‘template<>’ syntax
     void adl_serializer<py::object>::to_json(json& j, py::object obj)
          ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/jdumas/public/test_pyjson/main.cpp: In function ‘pybind11::object nlohmann::detail::from_json_impl(const json&)’:
/home/jdumas/public/test_pyjson/main.cpp:67:9: warning: control reaches end of non-void function [-Wreturn-type]
         }
         ^
make[2]: *** [CMakeFiles/pyjson.dir/build.make:63: CMakeFiles/pyjson.dir/main.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:73: CMakeFiles/pyjson.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

This is using the tip of the master branch for both pybind11 and nlohmann/json at the time of writing. My compiler is GCC 8.2.1 on Archlinux.

[jdumas]

This compiles if I change the ADL code to the following:

    template <>
    struct adl_serializer<py::object> {
        static void to_json(json & j, const py::object & value) {
            // calls the "to_json" method in T's namespace
            j = detail::to_json_impl(value);
        }

        static void from_json(const json & j, py::object & value) {
            // same thing, but with the "from_json" method
            value = detail::from_json_impl(j);
        }
    };

By then on the python side it fails at runtime when I calll

import pyjson as pj
pj.dummy_json()

giving the error message

Traceback (most recent call last):
  File "<input>", line 1, in <module>
    pj.dummy_json()
TypeError: Unable to convert function return value to a Python type! The signature was
    () -> nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer>
Did you forget to `#include <pybind11/stl.h>`? Or <pybind11/complex.h>,
<pybind11/functional.h>, <pybind11/chrono.h>, etc. Some automatic
conversions are optional and require extra headers to be included
when compiling your pybind11 module.

[martinRenou]

I think you can't change the from_json implementation to

static void from_json(const json & j, py::object & value) {
      value = detail::from_json_impl(j);
}

Because even though you take the py::object as a reference, you won't change the given object by assigning to it.

Can you please try again, with my version of the code, and with this module:

PYBIND11_MODULE(pyjson, m) {
    m.doc() = "pyjson module"; // optional module docstring

    // Dummy json
    m.def("dummy_json", []() {
        nl::json j = {
            {"pi", 3.141},
            {"happy", true},
            {"name", "Niels"},
            {"nothing", nullptr},
            {"answer", {
                {"everything", 42}
            }},
            {"list", {1, 0, 2}},
            {"object", {
                {"currency", "USD"},
                {"value", 42.99}
            }}
        };
        py::object out = j;
        return out;
    }, "returns a dummy json object");
}

I'm explicitely assigning the nl::json into a py::object before returning it. This one should compile.

[jdumas]

I'm sorry but your version of the code for the adl_serializer doesn't even compile on GCC on Linux (as stated above) ... Are you using clang or MSVC perhaps?

[martinRenou]

I'm using GCC 7.3 and it compiles... Which version of the code are you using now? Did you try my version of the adl_serializer (the very first one I gave) with the pyjson module? It does not compile if you just return the nl::json, but doing the conversion explicitly before returning the value works fine, and it won't be slower because this conversion will happen anyway.

[jdumas]

Ok well ... just so we're on the same page, I've created a pyjson repo to try and compile this example on Travis with GCC 6, 7, 8 as well as clang on macOS. As you can see on Travis, it does __not__ compile on any of those. Look at the CMakeLists.txt to see which version of which lib I'm using. I can give you write access to this repo if you want to try your chances against Travis...

[martinRenou]

I'm really sorry for the time wasted.. I only copied and paste here my .cpp file with the implementation, but there was more information in the declaration in my .hpp file:

namespace nlohmann
{
    template <>
    struct adl_serializer<py::object>
    {
        static py::object from_json(const json& j);
        static void to_json(json& j, const py::object& obj);
    };
}

I opened a PR in your test project. It should work now on Travis, at least I hope it will work!
EDIT: Hooray! It works! https://travis-ci.com/jdumas/pyjson/builds/94932105

[jdumas]

No worries. I appreciate your spending time on this to help :D

Btw I find that I still need to register the json class with

    py::class_<nl::json>(m, "json")
        .def(py::init<>())
        .def(py::init<py::object>())
    ;
    py::implicitly_convertible<py::object, nl::json>();

if I want to achieve implicit conversion for the input argument. I.e. I can then write directly:

m.def("input_json", [](const nl::json &j){
    std::cout << j.dump(4) << std::endl;
}, "input a dummy json object");

Ok now the bonus question is: if my function now returns a std::vector<nl::json> j;, can we have automatic conversion work when I write py::object out = j;?

[martinRenou]

I'm happy that it now works for you!

Btw I find that I still need to register the json class

Fair enough :) To be honest, I don't use the implicit conversion directly from Python, so I never encountered any issue without registering the json class.

Concerning your last question, I'm not completely sure. Did you try? If you were trying to implicitely convert std::vector<nl::json> to std::vector<py::object> it would not work. But here it's a bit different, maybe there is some magic in pybind11 that allows this kind of behavior.

[jdumas]

Yes I'm asking because copying a std::vector<nl::json> into a py::object directly doesn't work. But I think I can live with it for now, so I'll close this again =)

[martinRenou]

What you can do is create a std::vector<py::object> vec, populate it with nl::jsons (e.g. vec.push_back(j)). And then pass it to a py::object. This could work.

Or, if you already have an std::vector<nl::json> json_vec, you can create an std::vector<py::object> with:

std::vector<py::object> vec(json_vec.begin(), json_vec.end());

And then pass it to a py::object

[jdumas]

Yes it's not difficult to do. It's just that I'd rather have it done automatically, as when you start nesting types around it gets messy. Probably I can extend the adl_serializer to support that, but that seems like something I shouldn't have to do ideally.

[sriharshav]

In py::object from_json_impl(const json& j) cast py::int_(j.get<int>()) should be with long instead of int py::int_(j.get<long>())

        if (j.is_number())
        {
            double number = j.get<double>();
            if (number == std::floor(number))
            {
                // return py::int_(j.get<int>()); truncates long values
                return py::int_(j.get<long>());
            }
            else
            {
                return py::float_(number);
            }
        }

[martinRenou]

I've put this code in a small package: https://github.com/martinRenou/pybind_json :)

[apollo13]

I tried to separate the header file into a cpp and a header used by a shared library and failed horrible: https://github.com/apollo13/pyjson/blob/master/README.md -- any ideas/hints for me?

[apollo13]

Passing -DPYBIND11_NAMESPACE=pybind11 fixed it, seems to be a result of https://github.com/pybind/pybind11/blob/master/include/pybind11/detail/common.h#L24

[phil-zxx]

@martinRenou, @jdumas Is there still a plan to incorporate this into pybind11 natively?

[martinRenou]

I don't think this was ever the plan... I am not sure we want pybind11 to depend on nlohmann::json, but I might be wrong.

Anyway, you can still install https://github.com/martinRenou/pybind_json on the side :)

[phil-zxx]

True, an explicit dependency might be annoying. Just thought i'd be nice to combine the efforts of two great projects.

Maybe one could add a snippet like this to pybind11:

#ifdef INCLUDE_NLOHMANN_JSON_HPP_
  /*
    Paste relevant code from 
    https://github.com/martinRenou/pybind_json/blob/master/include/pybind_json/pybind_json.hpp
    here
  */
#endif

i.e. only use this code if nlohmann::json has been included prior to including pybind11.

[martinRenou]

Could be a good idea, I am not a pybind11 maintainer so I cannot really judge. Maybe you could ping maintainers?

[martinRenou]

@phil-zxx This work is now part of the official pybind organization: https://github.com/pybind/pybind11_json :)

[phil-zxx]

Great news!