Keras: using shared variables without mucking up the computational graph

[braingineer]

Basically, the situation is that I want to pre-load some shared variables that don't change as I feed in data.

However, they aren't weights or anything that I can just set to a network. They need to be computed on by the other layers, just as the data would. and you can't just arbitrarily put things in. because the computational gets re-ran, if something isn't linked in, it won't get used..

I've tried creating my faux input layer (found here) which creates a Node and gets inserted into the computational graph that way, but then keras expects it to be input at run time.

every variation on that idea (altering parts of Node, altering the variable in various places during runtime) always fails because there's some other part that assumes data can only be passed in at every iteration.

some things I've tried and their errors:

1. the full gist above where input and output tensors are set.

File "/home/cogniton/research/code/keras/keras/engine/topology.py", line 1773, in __init__
    str(layers_with_complete_input))
Exception: Graph disconnected: cannot obtain value for tensor sometensorname at layer layername. The following previous layers were accessed without issue: []

.2. Edited the thing that makes that trigger.

  File "/home/cogniton/research/code/keras/keras/engine/topology.py", line 2131, in run_internal_graph
    assert str(id(x)) in tensor_map, 'Could not compute output ' + str(x)
AssertionError: Could not compute output Reshape{3}.0

.3. maybe if i don't input the input tensor, it won't think it's input

  File "/home/cogniton/research/code/keras/keras/engine/topology.py", line 543, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/home/cogniton/research/code/keras/keras/engine/topology.py", line 141, in create_node
    input_tensors.append(inbound_node.output_tensors[tensor_index])
IndexError: list index out of range

.4. hmm. what if i just skip this whole node thing

  File "/home/cogniton/research/code/keras/keras/engine/topology.py", line 452, in __call__
    '". This layer has no information'
Exception: You tried to call layer "summarize_context". This layer has no information about its expected input shape, and thus cannot be built. You can build it manually via: `layer.build(batch_input_shape)`

[braingineer]

@fchollet can I get your advice on this one? I'm not sure what to try next.

[braingineer]

working so far. not going to prematurely call it. current workaround is below.

The tensors I am using are at the beginning of the computational graph, so the run_internal_graph was never fully getting through due to the line here.

As a 'use case' explanation: I had a high dimensional representation of a bunch of things. Each datapoint has their thing reduced to a vector and then ran through a model. This is expensive and there's a ton of overlapping work. So, I preload the high dim data and run it through the dim reduction stuff on its own. It then gets turned into an embedding matrix for which my input data can index.

the workaround:

in the model creation:

model = Model(input=self.igor.inputs, output=[point_predictions, tree_predictions, EOS], 
                      preloaded_data=self.igor.preloaded_data)

then, in Container.__init__

def __init__(self, input, output, name=None, preloaded_data=None): 

and finally Container.run_internal_graph

        for x in self.preloaded_data:
            tensor_map[str(id(x))] = (x, None)

[fchollet]

Maybe not using the Model abstraction at all would have been simpler? You can define your own Theano/TensorFlow functions (then train the weights with a Keras optimizer) from just input tensors and output tensors.

[braingineer]

Ha.... that does seem way simpler.

I don't think there's anything in particular I'm depending on there. If the current thing doesn't work out / I need to further muck things up, that'll probably be the route I'll take.

Thanks!

[braingineer]

So, thinking about this slightly more. Theano does recommend using shared variables if the input remains the same (at least, for small functions http://deeplearning.net/software/theano/faq.html#faster-small-theano-function)

It might be worth thinking about in the future when making any changes to how Model/Container makes the computation graph. I think all it'd take is changing how run_internal_graph initializes the tensor_map (at least, that's been working for me so far). Currently, it just initializes the tensor_map to the zipped inputs and masks. It could just as easily add any static inputs.

[bcarpenter-pub]

@fchollet could you clarify or point me to documentation on how to use the Keras optimzier/fit/predict function with Theano/Tensorflow Tensors?

I.e. if i created a model like this:

x = tf.placeholder(tf.float32,shape=[None, 1])
W = tf.get_variable("weights", [1, 1],  initializer=tf.random_normal_initializer())
b = tf.get_variable("biases",  1, initializer=tf.constant_initializer(0.0))
fc = tf.matmul(x, W) + b

how would I get this to be optimized via keras?