Baselines: How to do Single Prediction

[akblissweb]

Hello, I just got everything up and running and am currently working on a project with Deep Q learning. I like how the train_cartpole.py and enjoy_cartpole.py are formatted and it seems easy to tweak parameters and load and save models.

However, what I need to do is train a model and then instead of running it in an interactive game environment, I would like to pass the data for a single observation and get the best prediction of what action I should take given my previously trained model.

Do you have any examples of how I would do this ?? Is it possible without hacking into the baselines core code ??

Thanks.

[Sohojoe]

Are you trying to create a new environment? Ie train / run on something different to cart pole?

[akblissweb]

Yes. Made a new environment class. Got it to train OK. Saved the model. Now I want to make the best action based on a single state.

[Sohojoe]

The code from enjoy_cartpole.py should work for you (with minor modifications to load your enviroment and load your model)

def main():
    env = gym.make("CartPole-v0")
    act = deepq.load("cartpole_model.pkl")

    while True:
        obs, done = env.reset(), False
        episode_rew = 0
        while not done:
            env.render()
            obs, rew, done, _ = env.step(act(obs[None])[0])
            episode_rew += rew
        print("Episode reward", episode_rew)

Breaking it down: these are the key functions:

env.reset() - resets the enviroment and returns the default enviroment state
act(obs[None])[0] - selects the action with the highest Q value for the current enviroment state
obs, rew, done, _ = env.step(...) - is performing the action. It returns the new enviroment state (obs), any reward (rew), if the enviroment is terminated (done) and debug info (_)

Does this make sense?

[akblissweb]

Yes. Thanks I was coming to this conclusion. Just tried the code but need to fiddle with my reset function. Will comment again when I get it working.

[akblissweb]

Ok, so I've got this working. However, I am now running through a set of multiple states and for some reason, each time I run it, the stored model in the pkl file, seems to be giving different actions. I was assuming that if I feed the same data through the trained model, it would come back with the same actions, and therefore results, each time. Is this not the case ?? Is there any randomness built into the trained model ??

[akblissweb]

Ok, I figured it out. In the call to deepq.learn in the training program ...

act = deepq.learn( env, q_func=model, lr=1e-3, max_timesteps=100000, buffer_size=70000, exploration_fraction=0.3, #0.1 exploration_final_eps=0, #0.02 print_freq=5, #10 callback=callback )

The exploration_final_eps parameter must be set to zero in order to store a saved model which has no randomness embedded. If you leave it as in the example of 0.02, you will get randomness in your predictions.

Hope this helps others.