Baselines: Is there a way to access the Q-Values for a given observation?

[dqii]

Specifically, in the deepq/experiments/atari/train.py file?

[DanielTakeshi]

@dqii I've done this before. One way is to change deepq/build_graph.py so that the build_act function returns the q-values in addition to the actual action.

This line has the q-values.

https://github.com/openai/baselines/blob/master/baselines/deepq/build_graph.py#L152

[dqii]

Thanks!

[nflu]

After making the relevant changes to simple.py and build_graph.py as suggested and trying
``python import tensorflow as tf import numpy as np import modified_simple act, q = modified_simple.load("/Users/Neil/cartpole_model.pkl") observation = tf.placeholder(tf.float32, shape=[None, 4]) q.eval(feed_dict={"observation: [np.zeros(4)]}) ```` I get an error InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'deepq/observation' with dtype float and shape [?,4]
[[Node: deepq/observation = Placeholder[dtype=DT_FLOAT, shape=[?,4], _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] `

but it isn't possible to name a variable with a "/" in it. What am I missing?

I suspect that I can save the Q network in the traditional way with checkpoints and then access it from there if this isn't possible but it seems like it should be since act can be used so easily as in enjoy_cartpole.py

[DanielTakeshi]

@nflu change this method

https://github.com/openai/baselines/blob/master/baselines/deepq/build_graph.py#L146

to return q_values in addition to what it already returns. You have access to the q-values when running the policy and it returns the action and then the q-values. I've done this many times.

[nflu]

Yeah I tried that :/ The last line in my modified build_act in modified_build_graph.py is return act, q_values and I modified load in modified_simple.py to

    def load(path):
        with open(path, "rb") as f:
            model_data, act_params = cloudpickle.load(f)
        act, q = modified_build_graph.build_act(**act_params)
        sess = tf.Session()
        sess.__enter__()
        with tempfile.TemporaryDirectory() as td:
            arc_path = os.path.join(td, "packed.zip")
            with open(arc_path, "wb") as f:
                f.write(model_data)

            zipfile.ZipFile(arc_path, 'r', zipfile.ZIP_DEFLATED).extractall(td)
            load_state(os.path.join(td, "model"))

        return ActWrapper(act, act_params), q

I'm wondering how to use the q values once I return them from load. As I showed in my example above I'm not sure how to feed in an observation. If you have done this before perhaps you could make a pull request or fork with your implementation?

[nflu]

For anyone who encounters this issue and would like some more specifics: in build_act you need to change the _act function to evaluate the q_values as well

The last few lines of build_act should be

_act = U.function(inputs=[observations_ph, stochastic_ph, update_eps_ph],
                 outputs=(output_actions, q_values),
                 givens={update_eps_ph: -1.0, stochastic_ph: True},
                 updates=[update_eps_expr])
def act(ob, stochastic=True, update_eps=-1):
    return _act(ob, stochastic, update_eps)
return act

so that you get the values and not the tensor.

Then anywhere where you call act expecting to get an action you will have to go one index deeper. For example in simple.py line 252 action = act(np.array(obs)[None], update_eps=update_eps, **kwargs)[0] should be action = act(np.array(obs)[None], update_eps=update_eps, **kwargs)[0][0]

[AntonioSerrano]

Thanks @nflu! That's exactly what I was looking for. Those Theano-like functions were driving me crazy...

[irotaicnoc]

For anyone who encounters this issue and would like some more specifics: in build_act you need to change the _act function to evaluate the q_values as well

The last few lines of build_act should be

_act = U.function(inputs=[observations_ph, stochastic_ph, update_eps_ph],
               outputs=(output_actions, q_values),
               givens={update_eps_ph: -1.0, stochastic_ph: True},
               updates=[update_eps_expr])
def act(ob, stochastic=True, update_eps=-1):
  return _act(ob, stochastic, update_eps)
return act

so that you get the values and not the tensor.

Then anywhere where you call act expecting to get an action you will have to go one index deeper. For example in simple.py line 252 action = act(np.array(obs)[None], update_eps=update_eps, **kwargs)[0] should be action = act(np.array(obs)[None], update_eps=update_eps, **kwargs)[0][0]

I tried what you suggested, but it dosen't seem to pass any value where I should find q_values. Are there other modifications necessary to make that work?

Thanks

[AntonioSerrano]

Once you have made the modifications that @nflu mentioned, when you call the act function in simple.py (now called deepq.py) in the following line

action = act(np.array(obs)[None], update_eps=update_eps, **kwargs)[0]

you should now get a list with two elements (output_actions and q_values). So if you wanted to print the q_values, for instance, you should do something like this:

print(action[1])

You will probably want to rename the list action to step_output or something similar since that list does not contain only actions anymore.