Entt: Serialisation Question

Created on 3 Dec 2019  路  26Comments  路  Source: skypjack/entt

First up, I love the work you've done. I'm working on implementing it in my RPG project built using CRYENGINE. That leads to me having a few more constraints than some projects, which have led to me asking this question.

I am trying to serialise the entities and components out to disk. Sadly, XML is the only format I've got a decent chance of using due to the state of the libraries I am working with. Even then, it's a mess. The code to write an XML file is done, so I only need to read it in now, and that's 80% done.

I have some test output XML:

<entities>
 <entity>
  <components>
   <position typeGUID="d22ce389-15df-4ef3-b6cf-fd7490900765">
    <properties CryXmlVersion="2" X="20.799999" Y="20.799999"/>
   </position>
   <velocity typeGUID="812ddc29-d565-4294-8a1d-ce5af770151b">
    <properties CryXmlVersion="2" DX="0.80000001" DY="0.80000001"/>
   </velocity>
  </components>
 </entity>
 <entity>
  <components>
   <position typeGUID="d22ce389-15df-4ef3-b6cf-fd7490900765">
    <properties CryXmlVersion="2" X="15.6" Y="15.6"/>
   </position>
   <velocity typeGUID="812ddc29-d565-4294-8a1d-ce5af770151b">
    <properties CryXmlVersion="2" DX="0.60000002" DY="0.60000002"/>
   </velocity>
  </components>
 </entity>
</entities>

The idea is simple enough, read in the entities, make a new entity with EnTT, read each component - and use the GUID to figure out which class type I need to instantiate. That's where I am stuck.

Is there already a method to register types somewhere with a key and then have one created by passing in the key? Is this even possible? I don't mind having to register the components with something, as they already have to do that to work with CRYENGINE.

Is there another way to handle this? I did look at the snapshot code, and ran into a few issues as I tried to test it. My archive types didn't seem to have the right functions or something along those lines. I probably need to put more time into that.

I tried using Cereal as a quick and dirty way to get started with the testing but it doesn't compile within my project with the restrictions of CRYENGINE...which for some reason has decided to #define malloc and a few other crimes against good code. It relies on RapidJSON and RapidXML too, which are both looking a little abandoned - plus, I would be adding a third serialisation library to the code >.<

Ideally, I'd just be able to use a factory to create the components, one that is key based (templated). I'm just not sure if you already support that, or how it would work if not supported.

question
Source
picture ivanhawkes

Most helpful comment

With everything I need for the test working, here's the breakdown on the issue with code to show the solution.

Issue - I needed to be able to read and write the entity and component data to XML / JSON files. Initial tests using the snapshot code didn't go well, mainly because I didn't get the functions all defined for the archive manager. It needed one more which I didn't find out about till late in this process. That led me to try and do it using views. I quickly realised that it would be very unwieldy doing it like that as each component needed it's own block of code. The views also didn't return entities unless they had all the components in it's list. I had to switch back to snapshots.

I started with two component classes, Position and Velocity. While EnnT would be happy with POD I needed a little abstraction to make cryengine happy. Each component is therefore derived from a base class called IComponent. This gives me a little leeway on handling things.

struct IComponent
{
    IComponent() = default;
    virtual ~IComponent() = default;

    // This should be pure virtual but the ECS needs to be able to instantiate the struct, so...here's nothing.
    virtual bool Serialize(Serialization::IArchive& archive) { return true; };

    virtual const CryGUID& GetGuid() const
    {
        static CryGUID guid = "{DEADDEAD-DEAD-DEAD-DEAD-DEADDEADDEAD}"_cry_guid;

        return guid;
    }


    virtual const string& GetName() const
    {
        static string name = "ILLEGALVALUE";

        return name;
    }
};

Only the Serialize function is interesting here. It's responsible for serialising data to and from archives. Cryengine provides support code for that, and I was happy to re-use it for this.

The first step in the process is to register the components with the meta system so we can look them up later, construct them, and then assign them to registries.

        entt::registry saveRegistry;

        // Register meta.
        entt::meta<ECS::IComponent>()
            .type("icomponent"_hs);

        entt::meta<ECS::Position>()
            .type("position"_hs)
            .func<&AssignComponentFromMetaAny <ECS::Position> >("assign"_hs)
            .base<ECS::IComponent>()
            .ctor<>()
            .ctor<float, float>();

        entt::meta<ECS::Velocity>()
            .type("velocity"_hs)
            .func<&AssignComponentFromMetaAny <ECS::Velocity> >("assign"_hs)
            .base<ECS::IComponent>()
            .ctor<>()
            .ctor<float, float>();

I made sure to register their base class, just in case. The main thing here though is calling the .func member to register a templated function that will add the instantiated components correctly to a registry. Doing this allows us to avoid having a massive switch statement with a case for each component type. The classes aren't known till runtime, so they need a runtime solution to deal with creating them and assigning them.

        entt::registry loadRegistry;
        TestLoad(loadRegistry);

I need a registry to load the test data into and a function to perform the load.

    void TestLoad(entt::registry& registry)
    {
        // Load the file into an in-memory structure.
        if (XmlNodeRef entitiesNode = GetISystem()->LoadXmlFromFile("chrysalis/parameters/items/test.xml"))
        {
            // Iterate through all the children which should be individual entities.
            for (int i = 0, n = entitiesNode->getChildCount(); i < n; ++i)
            {
                // Grab the entity, which is basically an empty wrapper.
                XmlNodeRef entityNode = entitiesNode->getChild(i);

                // We should create a fresh entity for the components to attach onto.
                auto entity = registry.create();

                // Get the properties node - this holds all the properties for this component.
                if (XmlNodeRef componentsNode = entityNode->findChild("components"))
                {
                    // Iterate through all the components.
                    for (int i = 0, n = componentsNode->getChildCount(); i < n; ++i)
                    {
                        // Grab the entity, which is basically an empty wrapper.
                        XmlNodeRef componentNode = componentsNode->getChild(i);

                        // Using the tag as a unique ID for the class for now.
                        auto hash = entt::hashed_string{ componentNode->getTag() };

                        // Ask the system for the class and default construct a component of that type.
                        auto component = entt::resolve(hash);
                        auto any = component.construct();

                        // Get the properties node - this holds all the properties for this component.
                        XmlNodeRef propertiesNode = componentNode->findChild("properties");

                        // Serialise the properties across to the component.
                        auto& iComponent = any.cast<ECS::IComponent>();
                        Serialization::LoadXmlNode(iComponent, propertiesNode);

                        // Call the component specific function that assigns it to the registry.
                        if (auto const component_func = component.func("assign"_hs))
                        {
                            auto const invoke_res = component_func.invoke(
                                any,
                                any,
                                entity,
                                &registry);
                        }
                    }
                }
            }
        }
    }

You can glance over the XML stuff, it's basically just working through a DOM, getting each entity and reading the component data. It creates a new entity for each node under the main one. It then iterates the components. The XML tag has the hashed string name we used to register in the meta. A call to resolve followed by construct will give us a shiny new component. It exists now, it's just not in a registry attached to an entity yet.

Because I already have code that can serialise from the XML DOM to a component, I get that sorted with a call to the LoadXMLNode routine (not shown). This relies on the iComponent having a Serialize method and polymorphism.

Next, we check the component to see if it has a function (.func) attached to it that matched the name for the template that assigns components to entities. If it exists we invoke that function. The meta is handling the job of figuring out what the component is, and which function and parameters it is called using. That gets our new component added into the registry and attached to it's entity.

To make this a round trip we need to be able to save those entities and components back out to an XML file. I'm using the snapshot system for this, since it can save out correctly even when an entity doesn't include all of the listed component types.

struct Serial
{
    // Input.
    // NOTE: Missing one function, but we're not doing input yet, so haven't chased it up yet.
    template<typename Type>
    void operator()(entt::entity& entity, Type& component)
    {
    }


    // Output.
    void operator()(unsigned int entity)
    {
        // Not really sure why this is getting called. Need it defined to prevent compile errors.
    }


    void operator()(entt::entity entity)
    {
        if (m_entitiesNode)
        {
            // Each entity get's a couple of nodes made for storing it's components.
            XmlNodeRef entityNode = m_entitiesNode->newChild("entity");
            XmlNodeRef componentsNode = entityNode->newChild("components");

            // Store a copy in a map to enable fast lookup when iterating the components.
            m_nodeMap[entity] = componentsNode;
        }
    }


    template<typename Type>
    void operator()(entt::entity entity, const Type& component)
    {
        // Grab the components node from the map and then inject the component into the XML DOM.
        if (XmlNodeRef componentsNode = m_nodeMap[entity])
        {
            SaveComponent(componentsNode, component);
        }
    }


    void StartSave()
    {
        m_entitiesNode = GetISystem()->CreateXmlNode("entities");
    }


    void EndSave()
    {
        m_entitiesNode->saveToFile("chrysalis/parameters/items/test-out-snapshot.xml");
    }

private:
    XmlNodeRef m_entitiesNode;
    std::map<entt::entity, XmlNodeRef> m_nodeMap;
};

        Serial serial;
        serial.StartSave();
        loadRegistry.snapshot()
            .entities(serial)
            .component<ECS::Position, ECS::Velocity>(serial);
        serial.EndSave();

The serial struct is just the minimum code I needed to handling calls from the snapshot. It's not complete and it's definitely not well written...test code.

The snapshot makes calls onto the serial object. I interpret those into a few lines of code to modify an XML DOM. When it's finished, I write that DOM to a file. Not elegant, but it works with the systems I have on hand.

Overall, it's not a lot of code. Knowing what code to write was tough but the real trial was all the times something wasn't exactly right for the templates and MSVC would spew tons of spurious error messages that are eye wateringly hard to read.

Hopefully this helps any people who run into this issue in future.

picture ivanhawkes on 19 Dec 2019
👍2 ❤1

All 26 comments

Hello!

You could try using EnTT's meta for deserialisation. Just register all types with appropriate identifiers, add a bunch of data members and functions and you're done. Here's an implementation for JSON. I think it should be more or less straight forward to implement the same for XML. One day I may try to make it more generic, but right now I don't have time for it.

Innokentiy-Alaytsev picture Innokentiy-Alaytsev on 3 Dec 2019
👍1

First up, I love the work you've done.

Thank you, I'm glad you like it and feel free to give your feedback or suggestions and to join the gitter channel if you need help. :+1:

which have led to me asking this question. [...] just not sure if you already support that, or how it would work if not supported.

I don't know if the snapshot part fits well in this case. I've never used it with xml actually. You may make it work of course but I've not an example for you, I'm sorry.

As @Innokentiy-Alaytsev pointed out, an option is to use the meta system though.
You can register your types with an identifier, then read the name of the component from your xml, hash it (as in entt::hashed_string) and retrieve the meta type. Using it and a good amount of aliasing (as in entt::as_alias), you can create your components directly in the registry and modify their data members by means of an entt::meta_any, with no copies at all. Data members are identified by name as well, so again you can read the name of the data member you want to set, retrieve it and pass the value you read from the xml.

Does it make sense to you?

skypjack picture skypjack on 3 Dec 2019

That makes a decent amount of sense. I'll have a crack at it in a day or two - see how it goes.

ivanhawkes picture ivanhawkes on 4 Dec 2019
👍1

Ok. Feel free to leave the issue open, so you can ask more questions here if you need. Otherwise, join the gitter channel if you want faster answers, there are several people there who can help you.
I'll close this issue within a week if there are no updates or if you don't close it first. :wink:

Let us know how it goes. :+1:

skypjack picture skypjack on 4 Dec 2019

Can you keep it open just a bit longer. I've had family around all week and not been able to get any work done on it yet. Thanks.

ivanhawkes picture ivanhawkes on 14 Dec 2019

Sure, it's not a problem. I know what you mean. :wink:
Ping me if you need help or want a feedback. :+1:

skypjack picture skypjack on 14 Dec 2019

I made a little progress. I've used hash strings to register a class into the meta. I have a little test code to construct an instance of the class based on the string e.g.

        auto postionType = entt::resolve("position"_hs);
        auto any = postionType.construct(20.0f, 30.0f);

I can cast that to the actual type to check it works, and it does. So good, so far.

Now, how do I take that entt::meta_any variable and assign it as a component to an entity? I have a registry created and an entity ready to take a value - I just haven't figured out what function to call to assign it to that registry.

ivanhawkes picture ivanhawkes on 16 Dec 2019

Hello!

The way I do it is by adding a meta-function accepting a registry and an entity and assigning the actual component from the meta-any instance.

template < class TComponent >
void AssignComponentFromMetaAny (
    TComponent const& i_component_instance,
    ::entt::entity    i_entity,
    ::entt::registry* io_registry)
{
    ASCLEPIA_ECS_DEBUG_ASSERT (io_registry, assert::Level::kLogicCheck);
    ASCLEPIA_ECS_DEBUG_ASSERT (
        io_registry->valid (i_entity), assert::Level::kLogicCheck);

    io_registry->assign< TComponent > (i_entity, i_component_instance);
}


::entt::meta< ObjectComponent > ()
    .type (HS ("ObjectComponent"))
    .func< &::asclepia::ecs::AssignComponentFromMetaAny< ObjectComponent > > (
        ::asclepia::ecs::MetaAnnotation::kAssignToEntity)
    .ctor<> ()
    .ASCLEPIA_ECS_NAMED_MEMBER_VARIABLE (ObjectComponent, i, "i")
    .prop (::asclepia::MetaTrait::kIsRequired)
    .ASCLEPIA_ECS_NAMED_MEMBER_VARIABLE (ObjectComponent, f, "f")
    .ASCLEPIA_ECS_NAMED_MEMBER_VARIABLE (ObjectComponent, s, "s");

if (auto const assign_component_func =
        attachment_type.func (MetaAnnotation::kAssignToEntity))
{
    auto const invoke_res = assign_component_func.invoke (
        *attachment_instance,
        *attachment_instance,
        i_entity,
        &io_registry);
        ASCLEPIA_ECS_DEBUG_ASSERT (invoke_res, assert::Level::kLogicCheck);
}

I pass registry by pointer because I didn't think of entt::as_alias_t policy.

Innokentiy-Alaytsev picture Innokentiy-Alaytsev on 16 Dec 2019
👍1

Wow, that looks like it will work. It took a few guesses, but reading the example code for meta helped me figure out the calling conventions and stuff. I was having a spot of bother with getting it working and so simplified it by skipping the template function and putting a couple of test functions on the component. Once I had the calling convention right it just worked. Lovely.

I need to switch it back to using the template since littering my components with a helper function each isn't a great plan. Once I do that and clean up my test code I'll post here with the minimal code I needed to make it work (skipping over the serialisation parts, since that is cryengine dependant).

Thanks for the help! I'm off to have a beer for now.

ivanhawkes picture ivanhawkes on 17 Dec 2019

Hopefully this is the last piece of the puzzle.

I can't seem to make the .func call the template version of the function. To get around it I just added a function with that signature to each of my test components and called it from them. That's not a great idea going forward, since I'll be duplicating and specialising a lot of code - which is exactly what templates are meant to do.

I've added a template at the same namespace scope as my test functions i.e.

template <typename TComponent>
void AssignComponentFromMetaAny(
    TComponent const& i_component_instance,
    ::entt::entity    i_entity,
    ::entt::registry* io_registry)
{
    io_registry->assign< TComponent >(i_entity, i_component_instance);
}

There's a class in the same namespace as the template with a test function I am calling. At one point it registers the meta for one of the components:

        entt::meta<ECS::Position>()
            .type("position"_hs)
//          .func<&ECS::Position::AssignComponentFromMetaAny>("assign"_hs)
            .func<&AssignComponentFromMetaAny <ECS::Position> >("assign"_hs)
            .base<ECS::IComponent>()
            .ctor<>()
            .ctor<float, float>();

To my eye, it looks like it should work, but the compile spews the usual soup of garbage when working with templates:

1>------ Build started: Project: Chrysalis, Configuration: Debug x64 ------
1>Item_uber.cpp
1>Plugin_uber.cpp
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(201,18): error C2672: 'std::invoke': no matching overloaded function found (compiling source file D:\Chrysalis\solutions\win64\Plugin_uber.cpp)
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(196): message : see reference to function template instantiation 'auto entt::internal::invoke::<lambda_11e77117be53435ad121b47c2e0009b6>::operator ()<Type,Type,entt::entity,entt::basic_registry<entt::entity>*>(Type *,Type *,entt::entity *,entt::basic_registry<entt::entity> **) const' being compiled
1>        with
1>        [
1>            Type=Chrysalis::ECS::Position
1>        ] (compiling source file D:\Chrysalis\solutions\win64\Plugin_uber.cpp)
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(659): message : see reference to function template instantiation 'entt::meta_any entt::internal::invoke<Type,void Chrysalis::AssignComponentFromMetaAny<Chrysalis::ECS::Position>(const TComponent &,entt::entity,entt::registry *),Policy,0,1,2>(entt::meta_handle,entt::meta_any *,std::integer_sequence<unsigned __int64,0,1,2>)' being compiled
1>        with
1>        [
1>            Type=Chrysalis::ECS::Position,
1>            TComponent=Chrysalis::ECS::Position,
1>            Policy=entt::as_is_t
1>        ] (compiling source file D:\Chrysalis\solutions\win64\Plugin_uber.cpp)
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(653): message : see reference to class template instantiation 'entt::internal::meta_function_helper<void (const TComponent &,entt::entity,entt::registry *)>' being compiled
1>        with
1>        [
1>            TComponent=Chrysalis::ECS::Position
1>        ] (compiling source file D:\Chrysalis\solutions\win64\Plugin_uber.cpp)
1>D:\Chrysalis\code\ChrysalisCore\Item\ItemSystem.h(222): message : see reference to function template instantiation 'entt::extended_meta_factory<Type,std::integral_constant<void (__cdecl &)(const TComponent &,entt::entity,entt::registry *),void Chrysalis::AssignComponentFromMetaAny<Chrysalis::ECS::Position>(const TComponent &,entt::entity,entt::registry *)>> entt::meta_factory<Type>::func<void Chrysalis::AssignComponentFromMetaAny<Chrysalis::ECS::Position>(const TComponent &,entt::entity,entt::registry *),entt::as_is_t>(const uint32_t) noexcept' being compiled
1>        with
1>        [
1>            Type=Chrysalis::ECS::Position,
1>            TComponent=Chrysalis::ECS::Position
1>        ] (compiling source file D:\Chrysalis\solutions\win64\Plugin_uber.cpp)
1>D:\Chrysalis\code\ChrysalisCore\Item\ItemSystem.h(219): message : see reference to class template instantiation 'entt::meta_factory<Chrysalis::ECS::Position>' being compiled (compiling source file D:\Chrysalis\solutions\win64\Plugin_uber.cpp)
1>D:\Chrysalis\code\ChrysalisCore\Item\ItemSystem.h(216): message : see reference to class template instantiation 'entt::meta_factory<Chrysalis::ECS::IComponent>' being compiled (compiling source file D:\Chrysalis\solutions\win64\Plugin_uber.cpp)
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(196,103): error C2893: Failed to specialize function template 'unknown-type std::invoke(_Callable &&,_Types &&...) noexcept(<expr>)' (compiling source file D:\Chrysalis\solutions\win64\Plugin_uber.cpp)
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.24.28314\include\type_traits(1579): message : see declaration of 'std::invoke' (compiling source file D:\Chrysalis\solutions\win64\Plugin_uber.cpp)
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(196,103): message : With the following template arguments: (compiling source file D:\Chrysalis\solutions\win64\Plugin_uber.cpp)
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(196,103): message : '_Callable=void (__cdecl &)(const TComponent &,entt::entity,entt::registry *)' (compiling source file D:\Chrysalis\solutions\win64\Plugin_uber.cpp)
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(196,103): message : '_Types={Type &, Type &, Type &, Type &}' (compiling source file D:\Chrysalis\solutions\win64\Plugin_uber.cpp)
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(201,18): error C2672: 'std::invoke': no matching overloaded function found (compiling source file D:\Chrysalis\solutions\win64\Item_uber.cpp)
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(196): message : see reference to function template instantiation 'auto entt::internal::invoke::<lambda_11e77117be53435ad121b47c2e0009b6>::operator ()<Type,Type,entt::entity,entt::basic_registry<entt::entity>*>(Type *,Type *,entt::entity *,entt::basic_registry<entt::entity> **) const' being compiled
1>        with
1>        [
1>            Type=Chrysalis::ECS::Position
1>        ] (compiling source file D:\Chrysalis\solutions\win64\Item_uber.cpp)
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(659): message : see reference to function template instantiation 'entt::meta_any entt::internal::invoke<Type,void Chrysalis::AssignComponentFromMetaAny<Chrysalis::ECS::Position>(const TComponent &,entt::entity,entt::registry *),Policy,0,1,2>(entt::meta_handle,entt::meta_any *,std::integer_sequence<unsigned __int64,0,1,2>)' being compiled
1>        with
1>        [
1>            Type=Chrysalis::ECS::Position,
1>            TComponent=Chrysalis::ECS::Position,
1>            Policy=entt::as_is_t
1>        ] (compiling source file D:\Chrysalis\solutions\win64\Item_uber.cpp)
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(653): message : see reference to class template instantiation 'entt::internal::meta_function_helper<void (const TComponent &,entt::entity,entt::registry *)>' being compiled
1>        with
1>        [
1>            TComponent=Chrysalis::ECS::Position
1>        ] (compiling source file D:\Chrysalis\solutions\win64\Item_uber.cpp)
1>D:\Chrysalis\code\ChrysalisCore\Item\ItemSystem.h(222): message : see reference to function template instantiation 'entt::extended_meta_factory<Type,std::integral_constant<void (__cdecl &)(const TComponent &,entt::entity,entt::registry *),void Chrysalis::AssignComponentFromMetaAny<Chrysalis::ECS::Position>(const TComponent &,entt::entity,entt::registry *)>> entt::meta_factory<Type>::func<void Chrysalis::AssignComponentFromMetaAny<Chrysalis::ECS::Position>(const TComponent &,entt::entity,entt::registry *),entt::as_is_t>(const uint32_t) noexcept' being compiled
1>        with
1>        [
1>            Type=Chrysalis::ECS::Position,
1>            TComponent=Chrysalis::ECS::Position
1>        ] (compiling source file D:\Chrysalis\solutions\win64\Item_uber.cpp)
1>D:\Chrysalis\code\ChrysalisCore\Item\ItemSystem.h(219): message : see reference to class template instantiation 'entt::meta_factory<Chrysalis::ECS::Position>' being compiled (compiling source file D:\Chrysalis\solutions\win64\Item_uber.cpp)
1>D:\Chrysalis\code\ChrysalisCore\Item\ItemSystem.h(216): message : see reference to class template instantiation 'entt::meta_factory<Chrysalis::ECS::IComponent>' being compiled (compiling source file D:\Chrysalis\solutions\win64\Item_uber.cpp)
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(196,103): error C2893: Failed to specialize function template 'unknown-type std::invoke(_Callable &&,_Types &&...) noexcept(<expr>)' (compiling source file D:\Chrysalis\solutions\win64\Item_uber.cpp)
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.24.28314\include\type_traits(1579): message : see declaration of 'std::invoke' (compiling source file D:\Chrysalis\solutions\win64\Item_uber.cpp)
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(196,103): message : With the following template arguments: (compiling source file D:\Chrysalis\solutions\win64\Item_uber.cpp)
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(196,103): message : '_Callable=void (__cdecl &)(const TComponent &,entt::entity,entt::registry *)' (compiling source file D:\Chrysalis\solutions\win64\Item_uber.cpp)
1>D:\Chrysalis\code\SDK\EnTT\src\entt\meta\factory.hpp(196,103): message : '_Types={Type &, Type &, Type &, Type &}' (compiling source file D:\Chrysalis\solutions\win64\Item_uber.cpp)
1>Done building project "Chrysalis.vcxproj" -- FAILED.
2>------ Skipped Build: Project: INSTALL, Configuration: Debug x64 ------
2>Project not selected to build for this solution configuration 
========== Build: 0 succeeded, 1 failed, 5 up-to-date, 1 skipped ==========

What could be causing this and how do I fix it?

ivanhawkes picture ivanhawkes on 18 Dec 2019

It seems like at some point you get a pointer to the registry and then take its address again and pass the latter to the function. It's the best guess I have.

Innokentiy-Alaytsev picture Innokentiy-Alaytsev on 19 Dec 2019

Sorry guys if I'm late to the party.
@ivanhawkes what version of EnTT are you using? master or a tag?

I'm from mobile now so I'm not 100% sure but I want to reassure you. At a first glance, the error isn't your fault nor of the library. It's due to a new fresh bug of MSVC instead.
There is a PR open for it and I've copy-merged the changes on branch experimental but it hasn't landed on master yet. Probably it should, even though this will cause some conflicts.

skypjack picture skypjack on 19 Dec 2019

I opted for the master tag, thinking stability would be a good introduction to the project.

It's good to hear it's neither my nor the libraries fault - not so surprising to hear it's come from MSVC. I've been running into trouble a little with that lately. A version or two of cryengine are failing to build thanks to one issue or the other with the compilers. I recently had to delete and re-install MSVC to downgrade my compilers to get other code to build again.

Do you have an ETA on when the experimental stuff might get merged into master? If it's not too far into the future I might just push on with the error and the workaround I am using until a fix comes along.

ivanhawkes picture ivanhawkes on 19 Dec 2019

The issue is on the latest version of msvc afaik. See #373 for more details.
I try to not merge unstable things on master for stability actually but I can do nothing on compilers updates that break backwards compatibility with the language.

experimental will be a huge change from the point of view of working with EnTT across boundaries.
I'm almost finished but the upcoming vacancies could introduce a delay.

Because this issue affects master, I could try to cut a new patch release from master instead.
This should be easier and will make the patch available soon.
I cannot really merge the PR as is because of the conflicts it would have with experimental. However, I can take the relevant parts and reduce this problem to a minimum.

skypjack picture skypjack on 19 Dec 2019

This is the error reproduced also on master.
It confirms that at least MSVC 19.24.28314.0 suffers from this problem. :+1:
May I ask you what version of MSVC are you running currently?

--- EDIT

I've pushed on master also the fix for the issue of MSVC. Currently, all tests pass again.
A test from you would help to know if that was also your error. Looking forward to your answer before to cut a new release.

skypjack picture skypjack on 19 Dec 2019

Mine is MSVC 19.24.28314.0 - so, same version it seems.

I'll do a pull on master and see what happens.

ivanhawkes picture ivanhawkes on 19 Dec 2019
👍1

I can confirm that this fix works for my issue! The .func is happily accepting the templated code now.

Thanks for all the time you've spent helping me get this test running. With the ability to read and write components to XML I can now start prototyping some components and testing systems. I've been looking forward to this for quite a while now.

I'll clean up the test code a bit and post the interesting parts here for anyone who is interested.

ivanhawkes picture ivanhawkes on 19 Dec 2019

With everything I need for the test working, here's the breakdown on the issue with code to show the solution.

Issue - I needed to be able to read and write the entity and component data to XML / JSON files. Initial tests using the snapshot code didn't go well, mainly because I didn't get the functions all defined for the archive manager. It needed one more which I didn't find out about till late in this process. That led me to try and do it using views. I quickly realised that it would be very unwieldy doing it like that as each component needed it's own block of code. The views also didn't return entities unless they had all the components in it's list. I had to switch back to snapshots.

I started with two component classes, Position and Velocity. While EnnT would be happy with POD I needed a little abstraction to make cryengine happy. Each component is therefore derived from a base class called IComponent. This gives me a little leeway on handling things.

struct IComponent
{
    IComponent() = default;
    virtual ~IComponent() = default;

    // This should be pure virtual but the ECS needs to be able to instantiate the struct, so...here's nothing.
    virtual bool Serialize(Serialization::IArchive& archive) { return true; };

    virtual const CryGUID& GetGuid() const
    {
        static CryGUID guid = "{DEADDEAD-DEAD-DEAD-DEAD-DEADDEADDEAD}"_cry_guid;

        return guid;
    }


    virtual const string& GetName() const
    {
        static string name = "ILLEGALVALUE";

        return name;
    }
};

Only the Serialize function is interesting here. It's responsible for serialising data to and from archives. Cryengine provides support code for that, and I was happy to re-use it for this.

The first step in the process is to register the components with the meta system so we can look them up later, construct them, and then assign them to registries.

        entt::registry saveRegistry;

        // Register meta.
        entt::meta<ECS::IComponent>()
            .type("icomponent"_hs);

        entt::meta<ECS::Position>()
            .type("position"_hs)
            .func<&AssignComponentFromMetaAny <ECS::Position> >("assign"_hs)
            .base<ECS::IComponent>()
            .ctor<>()
            .ctor<float, float>();

        entt::meta<ECS::Velocity>()
            .type("velocity"_hs)
            .func<&AssignComponentFromMetaAny <ECS::Velocity> >("assign"_hs)
            .base<ECS::IComponent>()
            .ctor<>()
            .ctor<float, float>();

I made sure to register their base class, just in case. The main thing here though is calling the .func member to register a templated function that will add the instantiated components correctly to a registry. Doing this allows us to avoid having a massive switch statement with a case for each component type. The classes aren't known till runtime, so they need a runtime solution to deal with creating them and assigning them.

        entt::registry loadRegistry;
        TestLoad(loadRegistry);

I need a registry to load the test data into and a function to perform the load.

    void TestLoad(entt::registry& registry)
    {
        // Load the file into an in-memory structure.
        if (XmlNodeRef entitiesNode = GetISystem()->LoadXmlFromFile("chrysalis/parameters/items/test.xml"))
        {
            // Iterate through all the children which should be individual entities.
            for (int i = 0, n = entitiesNode->getChildCount(); i < n; ++i)
            {
                // Grab the entity, which is basically an empty wrapper.
                XmlNodeRef entityNode = entitiesNode->getChild(i);

                // We should create a fresh entity for the components to attach onto.
                auto entity = registry.create();

                // Get the properties node - this holds all the properties for this component.
                if (XmlNodeRef componentsNode = entityNode->findChild("components"))
                {
                    // Iterate through all the components.
                    for (int i = 0, n = componentsNode->getChildCount(); i < n; ++i)
                    {
                        // Grab the entity, which is basically an empty wrapper.
                        XmlNodeRef componentNode = componentsNode->getChild(i);

                        // Using the tag as a unique ID for the class for now.
                        auto hash = entt::hashed_string{ componentNode->getTag() };

                        // Ask the system for the class and default construct a component of that type.
                        auto component = entt::resolve(hash);
                        auto any = component.construct();

                        // Get the properties node - this holds all the properties for this component.
                        XmlNodeRef propertiesNode = componentNode->findChild("properties");

                        // Serialise the properties across to the component.
                        auto& iComponent = any.cast<ECS::IComponent>();
                        Serialization::LoadXmlNode(iComponent, propertiesNode);

                        // Call the component specific function that assigns it to the registry.
                        if (auto const component_func = component.func("assign"_hs))
                        {
                            auto const invoke_res = component_func.invoke(
                                any,
                                any,
                                entity,
                                &registry);
                        }
                    }
                }
            }
        }
    }

You can glance over the XML stuff, it's basically just working through a DOM, getting each entity and reading the component data. It creates a new entity for each node under the main one. It then iterates the components. The XML tag has the hashed string name we used to register in the meta. A call to resolve followed by construct will give us a shiny new component. It exists now, it's just not in a registry attached to an entity yet.

Because I already have code that can serialise from the XML DOM to a component, I get that sorted with a call to the LoadXMLNode routine (not shown). This relies on the iComponent having a Serialize method and polymorphism.

Next, we check the component to see if it has a function (.func) attached to it that matched the name for the template that assigns components to entities. If it exists we invoke that function. The meta is handling the job of figuring out what the component is, and which function and parameters it is called using. That gets our new component added into the registry and attached to it's entity.

To make this a round trip we need to be able to save those entities and components back out to an XML file. I'm using the snapshot system for this, since it can save out correctly even when an entity doesn't include all of the listed component types.

struct Serial
{
    // Input.
    // NOTE: Missing one function, but we're not doing input yet, so haven't chased it up yet.
    template<typename Type>
    void operator()(entt::entity& entity, Type& component)
    {
    }


    // Output.
    void operator()(unsigned int entity)
    {
        // Not really sure why this is getting called. Need it defined to prevent compile errors.
    }


    void operator()(entt::entity entity)
    {
        if (m_entitiesNode)
        {
            // Each entity get's a couple of nodes made for storing it's components.
            XmlNodeRef entityNode = m_entitiesNode->newChild("entity");
            XmlNodeRef componentsNode = entityNode->newChild("components");

            // Store a copy in a map to enable fast lookup when iterating the components.
            m_nodeMap[entity] = componentsNode;
        }
    }


    template<typename Type>
    void operator()(entt::entity entity, const Type& component)
    {
        // Grab the components node from the map and then inject the component into the XML DOM.
        if (XmlNodeRef componentsNode = m_nodeMap[entity])
        {
            SaveComponent(componentsNode, component);
        }
    }


    void StartSave()
    {
        m_entitiesNode = GetISystem()->CreateXmlNode("entities");
    }


    void EndSave()
    {
        m_entitiesNode->saveToFile("chrysalis/parameters/items/test-out-snapshot.xml");
    }

private:
    XmlNodeRef m_entitiesNode;
    std::map<entt::entity, XmlNodeRef> m_nodeMap;
};

        Serial serial;
        serial.StartSave();
        loadRegistry.snapshot()
            .entities(serial)
            .component<ECS::Position, ECS::Velocity>(serial);
        serial.EndSave();

The serial struct is just the minimum code I needed to handling calls from the snapshot. It's not complete and it's definitely not well written...test code.

The snapshot makes calls onto the serial object. I interpret those into a few lines of code to modify an XML DOM. When it's finished, I write that DOM to a file. Not elegant, but it works with the systems I have on hand.

Overall, it's not a lot of code. Knowing what code to write was tough but the real trial was all the times something wasn't exactly right for the templates and MSVC would spew tons of spurious error messages that are eye wateringly hard to read.

Hopefully this helps any people who run into this issue in future.

ivanhawkes picture ivanhawkes on 19 Dec 2019
👍2 ❤1

Just a side note that probably will make you happy. :)
You can avoid the call to assign the updated component to the entity.

It's possible to define a constructor to which you pass a registry and an entity. The constructor can then default-assign the component of the given type to the entity and returns an aliased meta any to work with.
This way, you can read from the XML file, instantiate the right component and modify the data members of the object you get from the received meta any. It will edit directly the instance contained in the registry.
Moreover, you get rid completely of any copy of your components with a nice boost in terms of performance.

skypjack picture skypjack on 19 Dec 2019

Also @roig this issue can be of interest for _you know what_. :wink:

skypjack picture skypjack on 19 Dec 2019

Oh, that sounds interesting, especially since it would allow me to cut out a bunch of code.

ivanhawkes picture ivanhawkes on 20 Dec 2019

Well, that didn't take long to test, and it works! For the record I have added a new .ctor to the meta which passes in an entity and a registry pointer. That takes care of the call to assign the newly constructed component.

Code changes:

// Construct and immediately assign to an entity in a registry.
Position(const entt::entity entity, entt::registry* registry)
{
    registry->assign<Position>(entity, *this);
}

Add that to the position component.

.ctor<const entt::entity, entt::registry*>()

Add that to the line which creates the meta entry.

auto any = component.construct(entity, &registry);

Change the line which constructs the component so it takes the two parameters.

I must say, I'm thoroughly impressed with this body of work. I wouldn't want to be writing all that template code, let alone the funky methods needed for type erasure and all the other cool things the code is doing. Thanks once again!

ivanhawkes picture ivanhawkes on 20 Dec 2019

I spoke too soon. While it does run, it loses the data in the components and saves out garbage values for them. I think the issue is that the component is both constructed and assigned to the registry before any values have been filled in. It seems like it is copying by value rather than moving the actual reference. My call to:

auto& iComponent = any.cast<ECS::IComponent>();

Returns what I presume is just a stale copy of the data rather than a reference to it.

Have I missed a step or some nuance?

ivanhawkes picture ivanhawkes on 20 Dec 2019

Uhm, no, it's not like this. I'm trying to sketch it out of my mind since I've not the code at hand right now.
The idea is that of creating a free function to _construct_ your types directly in the registry and return them by _reference_:

template<typename Type>
Type & assign(const entt::entity entity, entt::registry* registry) {
    return registry->assign<Type>(entity);
}

Then register it as a _constructor_ that returns an aliased meta_any:

.ctor<&assign<Position>, entt::as_alias_t>()

Now, when you do this:

auto any = entt::resolve("Position"_hs).construct(entity, &registry);

What you obtain is an instance of meta_any that doesn't contain a copy of the component. It holds instead a _reference_ to the objects within the registry, that is the one already assigned to your entity. So far, we have no copies at all.
Because you used a base class, you can do this now:

auto& iComponent = any.cast<ECS::IComponent>();

Editing iComponent will modify the instance in the registry.

Side note.

Actually, I did something slightly different though. I've the constructor that returns an alias. When I read the xml or json file, I find the _name_ of the component and construct it as:

auto any = entt::resolve(entt::hashed_string::value(name_from_xml_as_string)).construct(entity, &registry);

The meta_any comes with a type member that returns a meta_type. Moreover, it can be used as an _argument_ to set the parameters.
So, imagine I read from the xml that the parameter x must be set to 20. You can do something like this:

auto type = any.type();
type.data(entt::hashed_string::value(member_to_set_from_xml_as_string)).set(any, value);

Note that:

  • value is the value you read from the xml, so a variable of integer type with value 20, nothing special here. If you want to pass it by reference and avoid copies you can still use std::ref(value) to exploit aliasing.

  • *any it the component on which you want to set the value, that is the one within the registry. Since it's already an alias, you can pass it directly to the setter. A meta_handle is created from it and it still behaves as an alias by copying the reference from the meta_any.

Again, you've zero copies in this process (especially if you use std::ref for the arguments) and you _set_ (as in .set) a data member of a component directly in the registry. No need to _replace_ it later.

I hope I gave you a hint on how this works. Let me know if something is unclear. :+1:
Also, what's the purpose of your base class? Because it's not strictly required to do this actually.

skypjack picture skypjack on 20 Dec 2019

The base class is there to make other parts of cryengine happy. I expect I'll want to expose these components to the editor in some places. There's some functions they all need to expose e.g. Serialize. Since I want to use the existing serialise code to handle marshalling data over for these components it makes sense to just allow them to be derived from it. That gives me the cheap and easy option of casting to an IComponent and calling the Serialize on it. It's probably not technically needed, but it's likely to be useful at some point.

I think I understand the code you posted, and I've popped it into my project and it does work as expected - so, happy days! I'm reading the wiki page on runtime reflection now, and it has the salient information I think I didn't know first time around.

Everything seems to be working, so I'm about ready to make a bunch of component definitions and see how they fit together and solve my problems.

I'm looking into how I can make inventory systems, item systems, and perhaps spell / RPG systems of other sorts. Should be a ton of fun :D

ivanhawkes picture ivanhawkes on 20 Dec 2019

It's probably not technically needed, but it's likely to be useful at some point.

Sure, it makes sense. I was just curious because it hasn't an actual role in the example you posted so it seemed superfluous to me at a first glance. Thanks for answering. :+1:

I've popped it into my project and it does work as expected

:tada:
Aliasing is probably the less known and more powerful feature of the runtime reflection system.
It makes possible to do almost everything with literally no copies, that is a huge boost in terms of performance other than reducing quite a lot the number of headaches.

As a side note, I'll soon allow aliasing also for non-copyable types. It will make possible to pass the registry by reference to our functions from the external context.

Should be a ton of fun

You know where to find me/us. :smile:
Since your work is definitely interesting, feel free to drop you updates here, in the gitter channel or wherever you find it's suitable for the purpose.

I'm looking forward to know your progresses and thanks for using EnTT! :heart:

skypjack picture skypjack on 20 Dec 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

dBagrat picture dBagrat  路  4Comments
xoorath picture xoorath  路  3Comments
skypjack picture skypjack  路  4Comments
netpoetica picture netpoetica  路  6Comments
Kerndog73 picture Kerndog73  路  5Comments