Gym: Advice on Dict space with Box as key

[ljvmiranda921]

Background

Hi, I'm trying to create an environment that uses an OrderedDict as its observation space.
The idea is like a long sequence of molecules with their coordinates as keys, and their type as spaces.Discrete(2).

For example,

# Example of a chain with the coordinates as keys
# and "molecule type" as spaces.Discrete(2)
my_chain = {array[0,0] : 0, array[-1,0]: 1, array[0,1]: 0, array[0, -3]: 1}

What I've tried

I tried using spaces.Dict in the following manner:

from gym import spaces
observation_space = spaces.Dict({
    spaces.Box(low=-100, high=100, shape=(2,), dtype=int) : spaces.Discrete(2) })

But it gives an error:

TypeError                                 Traceback (most recent call last)
<ipython-input-53-e792eb0a0bc9> in <module>()
----> 1 observation_space = spaces.Dict({
    spaces.Box(low=-100,high=100,shape=(2,), dtype=int): spaces.Discrete(2)})

TypeError: unhashable type: 'Box'

Which shouldn't be surprising because according to the "common usage" in the spaces.Dict docstring, most of the keywords are well, hashable types like strings. I really want to stick to an OrderedDict because it's easier to query the points and retrieve the type of particle that lies on that coordinate.

I would like to know your advice on this matter on a good data structure to implement.

Thank you!

[ljvmiranda921]

Hi, I've found a workaround but defining my observations as the grid itself rather than the chain. I'll be closing this issue now.

[bklebel]

I would not try to set the keys of the spaces.Dict with changing variables, since a network getting this data would probably need to know in advance what it is looking for. Adding an additional dimension to your array, containing the type of molecule should do fine, I guess.

You can look at

https://gist.github.com/bklebel/913d8f155e6ed23f8a35fba989c70140

for a working example of a spaces.Dict environment.

EDIT: gist code now works good

[ljvmiranda921]

Hi @bklebel , thanks for the reply.

I've created this environment and decided on representing the observation space as spaces.Box for the grid/lattice where a molecule will be placed.

Sure, I can add an additional dimension to the array for the type of molecule (0 or 1), but because it is a chain, would my observation_space become a growing list?

At the back of my head, this still feels inefficient, what do you think about it?

[bklebel]

well...I thought of it for a while, sadly I am no expert in protein folding.

However, the observation space should not be a growing list, I think, since you would need a growing network, which is not so good.
I would stick to the grid as observation. however, some information about where the last molecule of the chain sits at a certain moment would be nice to give the agent, though I do not know how best to do this.
For your environment, I think a list would suffice: you have the input sequence of molecule types already, and in the list you can put the corresponding positions. You could also merge them in the end...for printing purposes, right? because, the reward is calculated differently in the end.

Anyways, I think the spaces.Box is good - but someone may correct me, I entered the field of reinforcement learning only recently ;)

[ljvmiranda921]

Hi @bklebel , no worries, thanks a lot for your advice and insight :)

Yes, as for the information regarding the positions of all molecules (including the last), I have made it accessible from the info dictionary being returned by the environment. Thanks a lot! I am also starting out in RL :)