Theano: feature request: support for cudnn v5 rc

Created on 6 Apr 2016 · 20Comments · Source: Theano/Theano

Anyone can help this? https://developer.nvidia.com/rdp/cudnn-download

Source

fayeshine

👍3

Most helpful comment

@nouiz, any plans on also wrapping the RNN implementation?

sherjilozair on 7 Apr 2016

👍12

All 20 comments

We have it here for new convolution:

https://github.com/mgermain/Theano/tree/cudnn5

It will probably be merged this week.

On Tue, Apr 5, 2016 at 8:11 PM, Wenjian Huang [email protected]
wrote:

Anyone can help this? https://developer.nvidia.com/rdp/cudnn-download

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/Theano/Theano/issues/4342

nouiz on 6 Apr 2016

👍5

@nouiz, any plans on also wrapping the RNN implementation?

sherjilozair on 7 Apr 2016

👍12

From the paper the results look very impressive, can't wait for the theano integration to see it live :)

subodhq on 12 Apr 2016

Just an update, we merged that in Theano master.

We also want to wrap the RNN stuff. But we don't know when it will be done.
So if someone want to help and work on it shortly, it would be great.

On Tue, Apr 12, 2016 at 8:30 AM, subodhq [email protected] wrote:

From the paper the results look very impressive, can't wait for the theano
integration to see it live :)

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/Theano/Theano/issues/4342#issuecomment-208880618

nouiz on 13 Apr 2016

@nouiz I just checked the latest version of theano, seems that the RNN stuff haven't been merged yet.

jiumem on 18 Apr 2016

Just to be clear, we merged the change to support cudnn v5 convolution
change.

But we didn't started to work on wrapping the RNN stuff. But we want to do
that.

On Mon, Apr 18, 2016 at 5:29 AM, jiumem [email protected] wrote:

@nouiz https://github.com/nouiz I just checked the latest version of
theano, seems that the RNN stuff haven't been merged yet.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/Theano/Theano/issues/4342#issuecomment-211294106

nouiz on 18 Apr 2016

👍4

@nouiz any progress?

xingdi-eric-yuan on 18 May 2016

👍2

Not to be a nag, but I too was wondering about the status of wrapping the native RNN support in cuDNN v5. Big performance win for theano when you do. While we're on the subject of cuDNN v5, how about a more streamlined relationship between high-level tensor types and the new tensor descriptor in cuDNN v5?

Regards,
Mark

phiber1 on 19 May 2016

👍2

I'll second this request for cuDNN 5 rnn/lstm support. The general release is out now and by many accounts the speedup using their LSTM API is quite substantial. I'm not interested in porting over to torch at this point but am tempted just to see what the speedup is.

zpardos on 21 May 2016

👍4

Any updates on this issue would be great! And if there is a branch where this is worked on, it would be nice to get a pointer.

Froskekongen on 30 May 2016

Yes, anyone working on adding NVIDIA's proposed LSTM speedups? I could help out on the Keras side of things, if needed.

carlthome on 30 May 2016

👍2

We don't know when someone will start to work on that. But we want to wrap it.

Do one of you want to help? This can make it appear faster.

@phiber1 what do you mean by a more streamlined version? What few people probably know is that there first "tensor" was fixed to 4d. Then when they started to make a 5d only tensor, I pushed them to stop that madness and go to nd tensor directly. This is mostly the tensor we have.

Converting in C between both descriptor is pretty fast. This isn't what is taking times.

nouiz on 30 May 2016

👍1

@nouiz Do you know if anyone has started working on this yet?

devrandom2 on 20 Jun 2016

👍1

On the LSTM? no. The Batch normalization is close to be merged.

On Mon, Jun 20, 2016 at 12:21 PM, devrandom2 [email protected]
wrote:

@nouiz https://github.com/nouiz Do you know if anyone has started
working on this yet?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Theano/Theano/issues/4342#issuecomment-227192573, or mute
the thread
https://github.com/notifications/unsubscribe/AALC-7cAnRYrEap2quL_Lk4kVklJNbk8ks5qNr3-gaJpZM4IAsue
.

nouiz on 20 Jun 2016

Status on cuDNN 5 LSTM optimizations?

[x] OPTIMIZATION 1: COMBINING GEMM OPERATIONS
[ ] OPTIMIZATION 2: STREAMING GEMMS
[ ] OPTIMIZATION 3: FUSING POINT-WISE OPERATIONS
[ ] OPTIMIZATION 4: PRE-TRANSPOSING THE WEIGHT MATRIX
[ ] OPTIMIZATION 5: COMBINING INPUT GEMMS

and the weird last optimization ("Step 3: Optimization with Many Layers") of adding more layers that I don't really get because that changes the model (right?).

I added optimization 1 to Keras (https://github.com/fchollet/keras/pull/2523) as that was straightforward thanks to Theano's manual.

Going by Nvidia's proposed speedup table _fused point-wise operations_ should be the priority, as that's the largest performance increase. I have no clue where that should enter into Theano's code base though. @nouiz got some pointers?

carlthome on 7 Jul 2016

👍1

TF just implemented the accelerated RNN/LSTM API in cudnn v5. Not sure if return sequences are supported.
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/cudnn_rnn

zpardos on 4 Sep 2016

https://github.com/fchollet/keras/issues/3692

carlthome on 6 Sep 2016

Just to tell that we are also finising wrapping that function in this PR:

https://github.com/Theano/Theano/pull/4915/

If all goes well, it should be merged this week.

On Tue, Sep 6, 2016 at 6:52 AM, Carl Thomé [email protected] wrote:

fchollet/keras#3692 https://github.com/fchollet/keras/issues/3692

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Theano/Theano/issues/4342#issuecomment-244916032, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALC-_21Gz2X0xdD3_e-nyH-imzms4Djks5qnUYHgaJpZM4IAsue
.