Baselines: Why the batchsize in for PPO for mujoco is so small?

[wjaskowski]

For Atari, we have 256 timesteps per batch and the 4 epochs with batchsize 64. This seems reasonable: 64*4=256, so there is a change that we will take all the observed points into the account. But for mujoco, we have 2048 timesteps/samples per batch and 10 epochs 64 elements each. This means that out of 2048 samples the optimizer will take not more than 640 into account. Isn't it a waste of samples?

[Breakend]

This is a fair point, would also like to know the answer to this.

[AdamStelmaszczyk]

Could you please have a look at this @joschu?

[wjaskowski]

I think this is fine. I misunderstood this parameter. Regardless of the
batch_size the whole dataset is visited (several times).

On 21 September 2017 at 21:13, Adam Stelmaszczyk [e-mail]
wrote:

Could you please have a look at this @joschu https://github.com/joschu?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/openai/baselines/issues/123#issuecomment-331253662,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEuSHdmG14gwOXFlt6z9mdiX42Wuuptnks5skrVFgaJpZM4PIzzO
.

[sharma-arjun]

@wjaskowski I think I have the same confusion that you had. Can you explain how are all of 2048 time steps being used several times if the batch size is 64 and the no. of epochs is 10? Thanks

[Breakend]

hey, noticed this as well. it's kind of hidden in the code, but if you look at the line of code where it iterates over batches, it says "iterate_once" with the batch size passed in. This doesn't mean sample a batch of that size, but iterate over the dataset one time (hence the for batch in iterate once). So the number of epochs is how many times to iterate over the thing with those minibatches. The batch size can still affect learning, but only in the conventional gradient descent minibatch way.

[wjaskowski]

iterate_once iterates over the whole dataset (2048) with batch_size (64).
You have two nested loops (over epochs and over batches):

        for _ in range(optim_epochs):
            losses = []  # list of tuples, each of which gives the loss for
a minibatch
            for batch in d.iterate_once(optim_batchsize):

On 22 September 2017 at 02:04, sharma-arjun [e-mail]
wrote:

@wjaskowski https://github.com/wjaskowski I think I have the same
confusion that you had. Can you explain how are all of 2048 time steps
being used several times if the batch size is 64 and the no. of epochs is
10? Thanks

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openai/baselines/issues/123#issuecomment-331314283,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEuSHXtnm6ITqpjTognDwNhG5JA6ASgHks5skvmhgaJpZM4PIzzO
.