Baselines: ppo2: what's the point of creating 2 network

[scotthuang1989]

In ppo2.py, algorithm create 2 netowrk:

class Model(object):
    def __init__(self, *, policy, ob_space, ac_space, nbatch_act, nbatch_train,
                nsteps, ent_coef, vf_coef, max_grad_norm):
        sess = tf.get_default_session()
        global_step = tf.train.get_or_create_global_step()

        act_model = policy(sess, ob_space, ac_space, nbatch_act, 1, reuse=False)
        train_model = policy(sess, ob_space, ac_space, nbatch_train, nsteps, reuse=True)

act_model: for interaction with environments.
train_model: for training model.

but this 2 network share all variables, it makes them as 2 same network.

update:

I found a difference between act_model and train_model: the act_model will only make a step, but train_step may take multiple steps. Is this the key point?

[brett-daley]

I think this is simply an implementation decision. The network takes different input sizes when acting in the environment (i.e. a single observation per environment instance) versus when training (i.e. a minibatch of observations). It was probably easier to just use the same network twice with different input sizes rather than share a single dynamically-sized placeholder. But there aren't really two networks because reuse=True in the second declaration.

[scotthuang1989]

@brett-daley thanks for the comments. I just worried that I don't understand ppo algorithm correctly, because, DQN have 2 separate network.

[pzhokhov]

@brett-daley correct, the act_model and train_model share all the variables, and differ only in the size of the the input placeholders. We cannot use dynamic-sized placeholders because those won't work when unrolling batch of data into a sequence for recurrent NN-based policies (see LstmPolicy, for instance)

[scotthuang1989]

@pzhokhov thanks for the confirmation.

[denyHell]

If these models are the same, the what's the difference between
self.OLDVPRED, which will be fed act_model.vf, and vpred = train_model.vf ?

[shtse8]

I'm late to the party.
I heard that A2C is on policy algo, while PPO is off policy due to having two networks instead. one network for interacting with env, while another one for training. But as I googled most of PPO codes, there is only one network used. I just wonder if my understanding correctly.

[luizmarao]

@shtse8 , PPO is an on policy method, as TRPO.