Ml-agents: Calculating Good Max Steps

[raeldor]

Hi,

For a real time simulation, I want the simulation to see what the best result it can get over 10 seconds is. Does this mean my max steps should be 60 (fps) * 10, ie 600?

Thanks

[awjuliani]

Hi @raeldor,

Each step actually corresponds to a FixedUpdate rather than an Update, so they don't actually correspond to frames or the framerate. I would recommend using a player brain to time out the episode to get a sense of the length you'd like it to be when run at realtime, since during training "10 seconds" will pass much quicker.

[raeldor]

So the FixedUpdate is on a completely different clock cycle and doesn't relate to frame update at all? Can it vary depend on hardware?

The reason I ask is because my simulation is using a real-time physics engine, so I'm worried that I'm getting different results when increasing the 'time scale' under the training configuration.

[vincentpierre]

The physics of the game are based on FixedUpdates. If all the logic of your game is based on FixedUpdate, the physics will speed up along with your agents and the behavior will be the same regardless of the timeScale. On the other hand, if some of your logic is in Update, it is possible that the Update frame rate will drop if you increase the timescale leading to weird behaviors as the timescale increases. IF you use a different physics engine than the one in Unity, I do not know what will happen and if this physics engine is based on Updates, it might cause problems to speed it up.

[MarcoMeter]

Hi @raeldor

So the FixedUpdate is on a completely different clock cycle and doesn't relate to frame update at all?

It doesn't depend on frame updates.

Can it vary depend on hardware?

No it doesn't. That's why fixed update was introduced. It is limited to a certain amount of calls for each second.

Inside the time project settings, the frequency of Fixed Updates can be set.
https://docs.unity3d.com/Manual/class-TimeManager.html

The above information is completely related to Unity's physics simulation. So it might be completely different in your case using a different engine.

[raeldor]

So, this is interesting. I previously had my max steps to 600 and my time scale to 1. This meant the agent got about 10 seconds of time to try and succeed. As per your instructions, I set the brain type to player and tried with a timescale of 100. I then modified max steps to try and get the 10 seconds again. It resulted in 60000. Which seems to suggest it's literally x100. In which case, doesn't that mean this thing is throttling if you can't go higher than 100? Shouldn't the max time scale relate to the ability of the hardware?

[raeldor]

So, I changed my physics engine to use FixedUpdate() rather than Update(), but I'm still getting different results with TimeScale set to 100 as opposed to 1.

Can I ask a couple of more fundamental questions...

1. Is timescale meant for running the training in faster than realtime?
2. If so, how does the physics engine know to speed up time based on this setting? For example, how does Unity know? Is FixedUpdate() just called more frequently and the elapsed time is calculated based on the setting you mentioned in TimeManager? Does setting the TimeScale in ML-Agents adjust the TimeManager setting? If so, does that mean other physics engines will have a separate similar setting that needs adjusting manually?

[MarcoMeter]

Is timescale meant for running the training in faster than realtime?
A timescale of one is commonly referred to as real-time

About your second question, FixedUpdate() is called more frequently. For machine learning, the goal is to run the simulation as fast as possible (regardless of the actual framerate). As we don't know which physics engine you are using, no one can really attempt to answer what has to be done in your case. I'd suggest that you seek for approaches within your physics engine documentation or ask the publisher of that piece of software.

[vladimiroster]

Thanks for reaching out to us. Hopefully you were able to resolve your issue. We are closing this due to inactivity.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.