Tflearn: Support for tensorflow dynamic_rnn

[benbogin]

Currently tflearn doesn't use tensorflow tf.nn.dynamic_rnn and tf.nn.bidirectional_dynamic_rnn for sequences with dynamic lengths.

Although there is a support for dynamic lengths by using dynamic=True with tflearn.layers.recurrent.simple_rnn (or similiar rnns) - as seen in #110 - which allows correct usage for varied-length sequences, under the hood it still uses tensorflow.python.ops.nn.rnn which statically creates the RNN graph.

While this works correctly, it doesn't benefit from the _major_ performance improvement of dynamic_rnn, and this could be a great future for tflearn.

Also, when trying to use tensorflow's dynamic_rnn directly, there is a problem with the control dependency in the trainer.py file, which caused me the error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: The node 'Adam/gradients/RNN/while/LSTMCell/add_grad/BroadcastGradientArgs/StackPush' has inputs from different frames. The input 'Adam/gradients/RNN/while/LSTMCell/add_grad/Shape' is in frame ''. The input 'Adam/gradients/RNN/while/LSTMCell/add_grad/BroadcastGradientArgs/RefEnter' is in frame 'RNN/while/RNN/while/'.

(I made a temporary fix by loosing the following dependency:
with tf.control_dependencies([loss_avg_op, acc_avg_op]): self.grad = tf.gradients(total_loss, self.train_vars)
)

[aymericdamien]

Yes, we should be supporting dynamic_rnn in next TFLearn version, it indeed has great speed improvement.
Also, thanks for reporting that issue, it seems it is link with new TF version 0.12, will work on a fix.

[Pastromhaug]

Any ETA on the fix? I have the same problem: tensorflow.python.framework.errors_impl.InvalidArgumentError: The node 'RNN/DizzyRNNCellOptHacky/Linear/Matrix/Assign' has inputs from different frames. The input 'RNN/while/DizzyRNNCellOptHacky/Linear/random_normal' is in frame 'RNN/while/RNN/while/'. The input 'RNN/DizzyRNNCellOptHacky/Linear/Matrix' is in frame ''.

[y-rok]

I have a same error on TF version 0.12.
I am curious about what is going on now.

[benbogin]

If you're interested, I made my temporary fix available here:
https://github.com/benb969/tflearn/commit/141c0d30b94076c7853ddfa1785faf445c2c32aa
It removes the dependency control that raises the error, however bare in mind it might have some other effects. With this change I was able to use dynamic_rnn

[wsjeon]

@benb969 : I think your solution is the best one until now. I think the problem occurs since tf.while_loop in dynamic_rnn and tf.control_dependencies are used at the same time. Similar problem occurs at 'slim'. (For reference, see https://github.com/tensorflow/tensorflow/issues/6087)