Entt: Call for comments: Job System

[skypjack]

Series _upcoming features_.

More than once I read of users that were to use EnTT along with a job system.
So I thought: why not?

I'd like to design a bunch of template classes well suited for a header only library so as to integrate a tailored job system within EnTT.
Before to start working on it, I'd like to know if it's really a _wanna have feature_.
I think I'll use the job system in my projects, so probably I'll create it anyway. However, knowing that some others are going to use it and can help me testing and improving that part would be great!!

Lighting issue just to collect some feedback. I'll close it in 24/48 hours.
Thank you all in any case!!

[ArnCarveris]

@skypjack There actually process classes that do similar thing, what the difference will be?

[skypjack]

I'd say a single threaded cooperative scheduler vs multi threaded job system. Pretty much orthogonal a tool, they will match with different use cases.

[ArnCarveris]

@skypjack Take look at cpp-taskflow this may help you.

[ArnCarveris]

@skypjack An simple example for Unity.

[Kerndog73]

Cpp-taskflow looks like a brilliant library. Perhaps EnTT could provide a few helpers for using the two libraries together?

[DJuego]

Yes. Yes. It's promising. I like transwarp too.

Recently, i "request a comparison between cpp-taskflow and transwarp here.

However, IMHO, EnTT should avoid dependencies at all costs and be a self-contained project. Any helpers should be just separate extensions with theirs own maintenance.

My two cents.

DJuego

[vblanco20-1]

@ArnCarveris thanks a lot for that taskflow library. I didnt know of it and it seems awesome.
EDIT: Doesnt work on windows (could be fixable) 😢

@skypjack i dont think a built in task system is needed with the ECS. It is something that depends too much on the specific architecture of the engine, i think. It would be like adding a graphics library to EnTT for some reason. Personally, i would prefer that things are kept simple. Let the users implement the task system they want.
What would be useful is some helper functions that make it easier. For example, would it be possible to have a function that compares 2 tuples of component types, and checks if there are dependencies, wich could make the scheduling safer as the users could have them to make the compiler test the possible data races.

[ArnCarveris]

@vblanco20-1 Totally agree with you.
@skypjack Any way, there should be provided necessary API if needed.

[ArnCarveris]

Found another library that could be useful: concurrentqueue

[vblanco20-1]

@ArnCarveris im using that queue in my experiment. I can confirm it works great and it is REALLY fast.

[skypjack]

Actually, I was thinking about something pretty different than a queue from which threads pick up some work to do.
Just to be clear, it won't be _embedded_ within the ECS in any way. It will be part of the library as SigH or Scheduler, that is it will have its own header and users can just ignore it. However, the idea is that I'll extensively use it to schedule my jobs in a multi threaded fashion.
Good to know that there exists a good library I can use for a comparison!! :-)

[ArnCarveris]

@skypjack Notify when will be any API draft .

[danielytics]

I'd like to design a bunch of template classes well suited for a header only library so as to integrate a tailored job system within EnTT.

Can you talk a little about what you are thinking for this?

The reason I’m asking is that the discussion above seems to be discussing adding a job system directly to EnTT. I think this is a bad idea — different people will want different job systems. Some already have one that they want to integrate with. I, personally, want to use Intel’s Threading Building Blocks task system because I’m already familiar with it, but other people won’t want to use that and will want to use something else instead.

So, even if its in a companion library, I’m not sure if the effort is necessarily worth it. However, adding something to make it easier to support plugging in your own job system would be greatly beneficial, I think. I’m not exactly sure what would be needed to make that work. I guess some way to scatter/gather so that component data can be pushed out to tasks and then be pulled back in.

I haevn’t started my own experimentation yet, but I was planning on doing something like this:

Systems that don’t modify component (only read them) can be run in parallel, but sequentially before those that do need to modify components. Then, the ones that write data, they must be run in such a way that only one thread ever writes a component at a time (must be careful that reads are syncrhonised too). I haven’t thought too hard about how this should work yet or how systems should be sequenced so that they read the components at the right time (before antoher sysetm writes? after?).

What I’ve done in the past (not using EnTT) is my state was double buffered: systems always read the previous state and wrote the new state, then at the end of the frame, it gets swapped so the next frame reads the new state. This works well since all reads are synchronised, but means that if the output of one sysetm must be immediately known by another one, the components can’t be used directly but instead messages/events need to be used to pass this data along. Ie written state is never read until it gets swapped at the end of the frame.

I hope that makes sense. Like I said, I haven’t started experimenting yet, so I’m still a bit fuzzy on it.

PS: Unity has a new ECS and tightly integrated Job System. Might be worth checking that out for inspiration: https://github.com/Unity-Technologies/EntityComponentSystemSamples/blob/master/Documentation/index.md

PPS: The above is my opinion, but I’m not a professional gamedev and never have been, so take that into account! Someone who is may have different views that are more relevant, especially in terms of the design. I do think that different people will want to use different job systems though.

[skypjack]

@danielytics The fact is that I need something simple and flexible and I don't want to pull in such a _complex_ external library in my codebase. Therefore, I'm going to design a minimal job system to use along with the registry somehow. However, it won't be integrated with the registry itself, so users can just ignore it as they can ignore the Scheduler class and so on. Moreover, I want to write something that isn't on the same line with TBB and the other mentioned, because so aren't my requirements and those libraries don't match actually.

[vblanco20-1]

@skypjack One thing i found myself needing.

A way to "deep copy" a registry, keeping specific component types, and keeping the entities ID (and versions) correct. Im not sure if the serialization system can be used for this, im still planning things so i havent tested if the serialization system would do the trick for such a "deep copy" of the registry.

The use is for a "pipelined" engine. You have the "main" ECS registry for simulation, and when you are going to render the world, you deep copy the "render" related components into a Render ECS registry, wich then goes into a second thread to run the rendering of the world.

For example in my space simulation thingy. I have components like Position,Rotation, Spaceship,Explosion, RenderCube, RenderTransform, TransformParent. For the render copy, i would need to copy the RenderCube,RenderTransform(After position/rotation is applied to it) , and TransformParent.
TransformParent relies on connection though entity ID, so needs to keep all the indices correct.

This would allow complete "pipelining" of the engine, with a SimulationThread, and a RenderThread, and the rest done through fork/join parallel_for and helpers.

[skypjack]

@vblanco20-1 Put a note in the TODO file so as to think about it.

[danielytics]

@skypjack I’m not asking to pull in TBB or any other library, quite the opposite.

I’m saying that, because different people will have different requirements (or different favourite job systems, or in-house ones or whatever — I only mentioned TBB as an example of what I want to use), I think the most useful thing would be to make sure that the users can themselves easily plug in their favourite job system to work with EnTT. Basically, I’m agreeing with @vblanco20-1

I was thinking that since writing a job system, even a simple one, is a bit of work, that time may be better spent making sure EnTT can be easily integrated with any job system and then letting everyone use whichever one they want, and you could then use an existing simple one for your needs rather than reinventing the wheel. Of course, if none exists that suit your needs, then by all means, write your own, doing so may be a great exercise to figuring out what features might be added to EnTT to better support a job system.

I’ve also used the concurrent queue that @ArnCarveris mentionend. Its worked out great for me and performs extremely well. Its a single-header library too, so its really easy to integrate.

EDIT: fixed typo

[skypjack]

@danielytics I'm sorry. I was in a rush today morning and I replied you while travelling to work. I just realized that I failed to write it correctly, my fault.
What I meant is that I don't want to pull in a third party library within my other project, not within EnTT. This library won't have dependencies any time soon, this is granted. :-)

make sure that the users can themselves easily plug in their favourite job system to work with EnTT [...] that time may be better spent making sure EnTT can be easily integrated with any job system and then letting everyone use whichever one they want

Did you have any issue with this? I mean, as long as I know there are no problems on this topic.
Recent changes made in collaboration with @vblanco20-1 should have helped a lot with this.

Of course, if none exists that suit your needs, then by all means, write your own, doing so may be a great exercise to figuring out what features might be added to EnTT to better support a job system.

Sure, this and also the fact that I'm working on an _EnTT-centered_ application for which I'd like not to have other dependencies.

---

Side not - https://github.com/SergeyMakeev/TaskScheduler
Of course, I'm just using this issue as a memo for an interesting link. :-)

[danielytics]

No, I’ve not had a problem with it, although, as I said in my earlier comment, I haven’t started prototyping it yet. I don’t envision any real problems, although I’ve not yet figured out the best way to manage writes to a component (other than double buffering, but that also has issues — eg how do I swap components without lots of memory copying), but I imagine your design will actually help answer some of these questions.

In any case, I’m looking forward to what you come up with.

[danielytics]

I should have said: other than double buffering or serialising writes. Obviously serialising writes is always an option and, I guess, they can be in parallel as long as they’re different components on the smae entitiy, or the same components on different entities.

[skypjack]

Thank you all for this interesting discussion. Really appreciated.
Time to close this issue.

[skypjack]

@vblanco20-1

A way to "deep copy" a registry, keeping specific component types, and keeping the entities ID (and versions) correct. Im not sure if the serialization system can be used for this, im still planning things so i havent tested if the serialization system would do the trick for such a "deep copy" of the registry.

The use is for a "pipelined" engine. You have the "main" ECS registry for simulation, and when you are going to render the world, you deep copy the "render" related components into a Render ECS registry, wich then goes into a second thread to run the rendering of the world.

I thought about this. Doesn't a continuous loader work in your case?
More than keeping entities' IDs and versions correct, you need a way to keep a correct mapping between entities ID and versions from both the sides. This is exactly the purpose of a continuous loader.
It's a matter of defining an archive that is filled by an instance of a snapshot and writes everything directly to a target registry by means of a continuous loader.

[vblanco20-1]

Some issues ive been runing into while refactoring the space battle benchmark to use taskflow https://github.com/cpp-taskflow/cpp-taskflow

Ive been trying to implement a deep copy for the registry to use in pipelining, but it didnt work. Some kind of way to use the snapshot system to actually copy into a new registry while keeping the entity ID numbers would be great (becouse my Transforms system uses entity ids to handle parenting).

Another issue ive ran into is storing/caching views. both move operator and copy operator on a view are deleted, and the use case for it is a task that "prepares" a system, and then it links into a parallel for type task. The parallel for cant really create the view inside, and it is needed that the task prepares the system, as im using a separate "schedule" step. Due to that separate schedule step, the parallel for cant be scheduled at that time. Im preparing N tasks (where N is the number of threads you have) and having a prepare step that splits the ranges for each task.

The current architecture has worked linearly (one system after another), but im trying to make it much more parallelized. It works by having each system have a "schedule" function that creates one or multiple tasks to schedule the work, and once the tasks tree is built, it is executed. The pipelining with the render systems would work by copying the registry into a "render registry", and that way its safe to keep the game systems doing things while the render systems are busy rendering the last frame.

[skypjack]

@vblanco20-1 I don't think there are problems to make views copy/move constructible/assignable. I can work on it when I'm back home, probably it's a matter of a few changes.

On the other side, to make a _deep copy of the registry_ is quite hard because of the type erased pools. However a deep copy isn't _some kind of way to use the snapshot system to actually copy into a new registry while keeping the entity ID numbers_. The latter is far easier.
Can you argument a bit more about what's the actual requirement?

[vblanco20-1]

@skypjack im trying to improve my spaceship benchmark thing. At the moment, im bottlenecked due to the main thread having sync points. This sync points are due to safely multithreading everything.

To improve it, im moving it to a task or fiber system (testing both). Due to all those clear sync points, i want to pipeline the engine with 2 pipeline stages, the Game stage and the Render stage.

The game stage is as usual, but what i want to do is to "copy" all the relevant components from the registry into a "render only" registry, that then gets used for the rendering portion of the engine (culling + rendering). Im aware that EnTT is not thread safe if you are adding or removing entities, so if i want to have my render code run at the same time as the game thread is doing things, i have no option but to copy it. Once the copy is done, the game thread will start to instantly go execute the next simulation frame, while the render thread is busy rendering to the screen. I expect to get a gain of 30% speed with this.

Another thing that could work is to directly have 2 separate registries (a Game and a Render) and keep them in sync somehow.