Ray: [rllib] In multiagent environment, is timesteps_total the total timesteps per agent or over all agents?

[coreylowman]

In multiagent environment, is timesteps_total the total timesteps per agent or across all agents?

For example, I have 4 policies in my multiagent policy configuration, and after the first training iteration the timesteps_total is 4000.

Is that number per agent or overall? I.e.:

1. Per agent - each agent has run 4000 timesteps, so the total number of timesteps is 16000
2. Overall - each agent has run 1000 timesteps, so the total number of timesteps is 4000

Which one is it?

[ericl]

It's the number of times step has been called on the env (so probably it means each agent has run 4000 timesteps, assuming each agent participates in every step).

[coreylowman]

Thanks, makes sense!

[drozzy]

it means each agent has run 4000 timesteps

Wouldn't his timesteps_total be 16,000 then?