Entt: EnTT Resource loader : same resource, different contexts

[Milerius]

Hello,

I'm thinking of using the EnTT resource module to load my SFML resources. In my head I would like to have access to the resources I need only in my current scene. something like resource[scene_name], is it planned to be able to use sub-sequences of the cache, a bit like a view, but on some resources.

//
using CacheMusic = entt::ResourceCache<sf::Music>;

If for example I would like to have a view of the sf::Music of a certain scene in particular, I want to be able to destroying the scene resources that are specific to it.

I wont store a Cache per scene because in my engine scene are just script, i want to store all the resource at the same place and filter it by scene identifier for exemple with entt::HashedString

What is your point of view ?
I mean it's better if all resources are stored at the same place in memory no ?

a scene in lua:

function update()
    --print("nb entities" .. shiva.entity_registry:nb_entities())
end

function on_key_pressed(evt)
    print("game scene keycode: " .. evt.keycode)
end

function on_key_released(evt)
    print("game scene released keycode: " .. evt.keycode)
end

function leave()
    print("leave game scene")
   -- I want to release resource of this scene for example
  -- Call a c++ callback like release_resource("game_scene")
  -- Internaly get a EnTT::hashed_string, and release the resource associated to this scene
end

function enter()
    print("enter game scene")
end

return {
    on_key_released = on_key_released,
    on_key_pressed = on_key_pressed,
    leave = leave,
    enter = enter,
    update = update,
    scene_active = true
}

scene_system in lua:

local lfs = require "lfs"
local scenes_table = {}
local current_scene

function get_file_name(file)
    return file:match("^.+/(.+)$")
end

function get_file_extension(url)
    return url:match("^.+(%..+)$")
end

function load_scene(require_path, filename)
    if (get_file_extension(filename) == ".lua") then
        local scene_name = string.gsub(get_file_name(filename), '.lua$', '')
        scenes_table[scene_name] = require(require_path .. "." .. scene_name)
        if scenes_table[scene_name].scene_active == true then
            current_scene = scenes_table[scene_name]
            current_scene.enter()
        end
    end
end

function recursive_directory_crossing(path, require_path)
    for file in lfs.dir(path) do
        if file ~= "." and file ~= ".." then
            local f = path .. '/' .. file
            local attr = lfs.attributes(f)
            if attr.mode == "directory" then
                recursive_directory_crossing(f, require_path)
            else
                load_scene(require_path, f)
            end
        end
    end
end

function __constructor__()
    print("scene manager constructor")
    local dir = lfs.currentdir() .. "/assets/scripts/scenes/lua"
    recursive_directory_crossing(dir, "assets.scripts.scenes.lua")
end

function __destructor__()
    print("scene manager destructor")
end

function internal_update()
    if (current_scene.update ~= nil) then
        current_scene.update()
    end
end

function internal_key_pressed(evt)
    if current_scene.on_key_pressed ~= nil then
        current_scene.on_key_pressed(evt)
    else
        print("current scene doesn't have on_key_pressed callback")
    end
end

function internal_key_released(evt)
    if current_scene.on_key_released ~= nil then
        current_scene.on_key_released(evt)
    else
        print("current scene doesn't have on_key_released callback")
    end
end

function internal_change_scene(scene_name)
    if scenes_table[scene_name] ~= nil then
        current_scene.leave()
        current_scene = scenes_table[scene_name]
        current_scene.scene_active = true
        current_scene.enter()
    end
end

scenes_system_table = {
    update = internal_update,
    on_key_pressed = internal_key_pressed,
    on_key_released = internal_key_released,
    on_construct = __constructor__,
    on_destruct = __destructor__,
    current_system_type = system_type.logic_update
}

i can make a map of a structure which handle different cache, but i'm not sure if it's efficient, so that's why i'm asking !

[ArnCarveris]

@Milerius Hi I've done async resource management with EnTT, I hope my experience will help you to find your own approach.

I wont store a Cache per scene because in my engine scene are just script, i want to store all the resource at the same place and filter it by scene identifier for exemple with entt::HashedString

What is your point of view ?
I mean it's better if all resources are stored at the same place in memory no ?

There is my implementation of ResourceRegistry with simple unit test, that allowed me to re-implement resource management for OWMAN as part of EnTT integration.

For example there is cell resource obtaining by entt::HashedString and releasing cell resource by entt::ResourceHandle.

BTW there is resource loaders for texture, sprite and cell.

PS there is ResourceManager.

[skypjack]

:thinking:
I didn't get it honestly. I mean, I've just released a game where resources are conceptually divided in _scenes_, as you mentioned. Btw, I wouldn't put the responsibility of partitioning the resources within a resource manager. Instead, I loaded all the resources in a centralized way, then I created an external tool that stored handles and partitioned them _by scenes_.
This is by far more flexible, mainly because there are some resources that doesn't belong to a scene sometimes and this way you can refer directly to the manager to get them. The other way around would force you to create a sort of _unknown scene_ and it doesn't make much sense.
Does it make sense to you?

[Milerius]

I loaded all the resources in a centralized way, then I created an external tool that stored handles and partitioned them by scenes.

It's exactly what i ask in my question, i will not store several ResourceManager, but load everything in the same place, class, whatever. But my question is more like what is your approach in code for this, how efficient it is etc ?

[skypjack]

As long as you don't need to do prediction or things along this path, the minimal resource manager that comes with EnTT is a flyweight centered cache where you can store all your resources. That is, if you have the same resource used from N scenes, you won't store it N times with the current implementation.
This is also the reason for which you don't get directly the resource, instead you obtain a handle from the cache.

My approach was straightforward actually. I used a few caches to store all the resources (only the ones currently in use when I can't load everything because of the memory usage), that is a cache for the images, a cache for the music, a cache for the fonts and so on.
Scenes are classes or whatever you want that live somewhere else and refer to the caches through a locator. When you set up a new scenes of a given type, you can assign to it all the handles in which it's interested. Otherwise, you can use a slightly different approach and name resources with a prefix (as an example "scene_1/resource_x"_hs), so as to refer to resources from within a scene by means of a fixed prefix.

I can think of a lot of other alternatives actually and none of them requires to _extend_ somehow the cache to introduce the concept of _scene_. It's just not the place to do that from my point of view.

[ArnCarveris]

@Milerius

how efficient it is etc ?

I haven't measured it yet. But OWMAN [Debug] runs stable at 60fps with low-end netbook.
Keep in mind that efficiency is defined by loader implementation.

[skypjack]

Yeah, I don't think the way you're going to use a cache in this terms will affect performance. Caches introduce performance hit through poorly written loaders, bad prediction algorithms and so on. The fact that the cache is conceptually partitioned or not shouldn't be a problem in this sense, in the worst case it's a design flaw.

[Milerius]

Ok I will send you my approach to have some feedback then ! See you latter

[Milerius]

For the loader i'm going on something like:

namespace shiva::sfml
{
    template <typename ResourceType>
    struct loader final : ::entt::ResourceLoader<loader<ResourceType>, ResourceType>
    {
        template <typename ... Args>
        std::shared_ptr<ResourceType> load(Args &&...args) const
        {
            auto resource_ptr = std::make_unique<ResourceType>();
            if (!resource_ptr->loadFromFile(std::forward<Args>(args)...)) {
                throw std::runtime_error("Impossible to load file");
            }
            return resource_ptr;
        }
    };

    template <>
    struct loader<sf::Music> final : ::entt::ResourceLoader<loader<sf::Music>, sf::Music>
    {
        template <typename ... Args>
        std::shared_ptr<sf::Music> load(Args &&...args) const
        {
            auto resource_ptr = std::make_unique<sf::Music>();
            if (!resource_ptr->openFromFile(std::forward<Args>(args)...)) {
                throw std::runtime_error("Impossible to load file");
            }
            return resource_ptr;
        }
    };
}

and for example the registry:

namespace shiva::sfml
{
    class resources_registry
    {
    public:
        template<typename Identifier = const char *, typename ... Args>
        bool load_music(Identifier id, Args&& ...args)
        {
            return musics_.load<loader<sf::Music>>(id, std::forward<Args>(args)...);
        }


    private:
        using textures_cache = ::entt::ResourceCache<sf::Texture>;
        using music_cache = ::entt::ResourceCache<sf::Music>;
        using sound_cache = ::entt::ResourceCache<sf::SoundBuffer>;

        textures_cache textures_{};
        music_cache  musics_{};
        sound_cache  sounds_{};
    };
}

is that a good approach ?

[skypjack]

It's similar to what I did in Face Smash, actually. Not exactly the same, but quite close to it. :+1:

[Milerius]

Thank you for your help in getting a better understanding of EnTT!

[skypjack]

You're welcome @Milerius and thank you for using it!! So far, the best ideas and improvements are the results of requests from users, you know... :-)

[Milerius]

Now i have this:

 class resources_registry
    {
    public:
        resources_registry(shiva::fs::path textures_path = shiva::fs::current_path() /= "assets/textures",
                           shiva::fs::path sounds_path = shiva::fs::current_path() /= "assets/sounds",
                           shiva::fs::path musics_path = shiva::fs::current_path() /= "assets/musics") noexcept :
            textures_path_(std::move(textures_path)),
            sounds_path_(std::move(sounds_path)),
            musics_path_(std::move(musics_path))
        {
        }

        template <typename Identifier = const char *, typename ... Args>
        bool load_music(Identifier id, Args &&...args)
        {
            const auto identifier = musics_cache::resource_type{id};
            return musics_.load<loader<sf::Music>>(identifier, std::forward<Args>(args)...);
        }

        template <typename Identifier = const char *, typename ... Args>
        bool load_texture(Identifier id, Args &&...args)
        {
            const auto identifier = textures_cache::resource_type{id};
            return textures_.load<loader<sf::Texture>>(identifier, std::forward<Args>(args)...);
        }

        template <typename Identifier = const char *, typename ... Args>
        bool load_sound(Identifier id, Args &&...args)
        {
            const auto identifier = sounds_cache::resource_type{id};
            return sounds_.load<loader<sf::SoundBuffer>>(identifier, std::forward<Args>(args)...);
        }

        template <typename LoaderFunctor>
        bool load_resources(shiva::fs::path &current_resource_path,
                            std::string_view resource_type,
                            std::string_view resource_type_singular,
                            LoaderFunctor &&loader_functor,
                            const shiva::fs::path &additional_path) noexcept
        {
            if (additional_path.empty())
                return load_resources(current_resource_path,
                                      resource_type,
                                      resource_type_singular,
                                      loader_functor,
                                      current_resource_path);
            std::string id;
            shiva::fs::path directory_path = current_resource_path;

            if (additional_path != current_resource_path) {
                directory_path = current_resource_path /= additional_path;
                id = additional_path.string() + "/";
            }

            if (!shiva::fs::exists(directory_path)) {
                this->log_->error("trying to load resources from a non existent directory: {0}",
                                  directory_path.string());
                return false;
            }

            log_->info("load {0} from path: {1}", resource_type, directory_path.string());
            bool res = true;
            fs::recursive_directory_iterator endit;
            for (fs::recursive_directory_iterator it(directory_path); it != endit; ++it) {
                if (!fs::is_regular_file(*it))
                    continue;
                try {
                    std::string filename = it->path().filename().string();
                    std::string stem_filename = it->path().filename().stem().string();
                    if (additional_path != current_resource_path) {
                        id += stem_filename;
                    } else {
                        id = filename;
                    }
                    log_->info("loading {0}: [ filename: {1}, id: {2} ]", resource_type_singular, filename, id);
                    res &= loader_functor(id.c_str(), filename);
                }
                catch (const shiva::fs::filesystem_error &error) {
                    this->log_->error("filesystem error occured: {0}", error.what());
                    res = false;
                }
                catch (const std::exception &error) {
                    this->log_->error("error occured: {0}", error.what());
                    res = false;
                }
            }
            return res;
        }

        bool load_textures(const shiva::fs::path &additional_path = "") noexcept
        {
            return load_resources(textures_path_,
                                  "textures",
                                  "texture",
                                  [this](auto &&...params) {
                                      return this->load_texture(std::forward<decltype(params)>(params)...);
                                  },
                                  additional_path);
        }

        bool load_musics(const shiva::fs::path &additional_path = "") noexcept
        {
            return load_resources(musics_path_,
                                  "musics",
                                  "music",
                                  [this](auto &&...params) {
                                      return this->load_music(std::forward<decltype(params)>(params)...);
                                  },
                                  additional_path);
        }

        bool load_sounds(const shiva::fs::path &additional_path = "") noexcept
        {
            return load_resources(sounds_path_,
                                  "sounds",
                                  "sound",
                                  [this](auto &&...params) {
                                      return this->load_sound(std::forward<decltype(params)>(params)...);
                                  },
                                  additional_path);
        }

        bool load_all_resources(const shiva::fs::path &additional_path = "") noexcept
        {
            return load_textures(additional_path) & load_musics(additional_path) & load_sound(additional_path);
        }

    private:
        shiva::logging::logger log_{shiva::log::stdout_color_mt("resources_registry")};
        using textures_cache = ::entt::ResourceCache<sf::Texture>;
        using musics_cache = ::entt::ResourceCache<sf::Music>;
        using sounds_cache = ::entt::ResourceCache<sf::SoundBuffer>;

        textures_cache textures_{};
        musics_cache musics_{};
        sounds_cache sounds_{};
        shiva::fs::path textures_path_;
        shiva::fs::path sounds_path_;
        shiva::fs::path musics_path_;
    };

I will also transform functions for parallel loading with an atomic counter latter !
Look's ok now ?

[skypjack]

Actually it looked ok also two posts ago. :smile:
Btw, it mostly depends on the framework you're using and your final goal. I mean, in Face Smash it was much simpler than your implementation, because it was just enough the way it was.
If you find your engine needs something like this, the implementation looks fine, so give it a try! :wink:

[Milerius]

a last question:

Why there is no overload for non const reference in resource_handle:

const Resource & get() const ENTT_NOEXCEPT {
        assert(static_cast<bool>(resource));
        return *resource;
    }

for a case on a sf::Music, i want to be able to make this:

sf::Music& music = *cache.handle<sf::Music>(id);
music.play();

actual fix:

        sf::Music &get_music(musics_cache::resource_type id)
        {
            return const_cast<sf::Music&>(*musics_.handle(id));
        }

[ArnCarveris]

@Milerius I think that resource should be immutable this way you could avoid various not obvious bugs in future.

Had same problem done exactly same there :/

[skypjack]

@Milerius @ArnCarveris No actual reason but for the fact that (at least in my case) resources are immutable and must remain immutable! I cannot even imagine a shared resource that can be changed. This is all against the flyweight after all.
Btw, in my case a resource like a music doesn't have member functions to _play_. Resources have only member functions to read properties like width, height, path, array of bytes and so on. The play method is put outside of the resources in a service that accepts tracks and just play them the _right_way (that is by arranging channels, discarding conflicting ones or those with fixed timing that cannot access a channel, and so on).

[Milerius]

@skypjack Yes, but in the case of sf::Music -> https://www.sfml-dev.org/documentation/2.5.0/classsf_1_1Music.php

The class have a member function for loading the music, and another to play it, that's why i'm in this situation !

[Milerius]

(you need to make a discord server for EnTT for short discussion like this @skypjack )

[skypjack]

(you need to make a discord server for EnTT for short discussion like this @skypjack )

This would worth its own issue!! :smile:
The risk is that there will be only you and me there... :joy:
Let's see if others like the idea before to spend our time.

[Milerius]

I'm really happy with this solution, it's now asynchrone on the type of resource you are loading:

https://gist.github.com/Milerius/04e23e890618185af841d9d021b9f974

any feedback is appreciable !

screenshot of the console:

i got this warning from open al somewhere in the log:
AL lib: (WW) UpdateDeviceParams: SSE performs best with multiple of 4 update sizes (1114)

if someone know what this means !

[skypjack]

Never seen something like this. Does AL stand for OpenAL? If it's so, you could look through their site.

[skypjack]

@Milerius Can we close this issue? As far as I understand, there is nothing to do in EnTT here. Am I wrong?

[Milerius]

Yeah !