Cntk: 1D convolution gets cuDNN failure 8: CUDNN_STATUS_EXECUTION_FAILED

[vermorel]

With CNTK 2.3, I am trying to perform a 1D convolution in BrainScript with:

MyInput = Input {1000, sparse=true}
W = ParameterTensor{(10:4)}
C = Convolution (W, MyInput, _AsArray(10), mapDims = 4, stride = _AsArray(1), autoPadding = false)

This triggers the failure:

cuDNN failure 8: CUDNN_STATUS_EXECUTION_FAILED ; GPU=0 ; hostname=LOKAD ; expr=workspaceSizeFinder()


[CALL STACK]
    > Microsoft::MSR::CNTK::CudaTimer::  Stop
    - Microsoft::MSR::CNTK::CudaTimer::  Stop (x2)
    - Microsoft::MSR::CNTK::Matrix<float>::  __autoclassinit2 (x12)
    - Microsoft::MSR::CNTK::MatrixBase::  MatrixBase

EXCEPTION occurred: cuDNN failure 8: CUDNN_STATUS_EXECUTION_FAILED ; GPU=0 ; hostname=LOKAD ; expr=workspaceSizeFinder()

Can someone tell me what is going wrong here? How can I get a 1D convolution to work? Thanks!

[KeDengMS]

Sparse convolution is not supported. Can you add an embedding layer?

[vermorel]

Done. Unfortunately, I am getting the same error. Also, changed to Input {1000, sparse=false}.

[KeDengMS]

There might be a bug in BrainScript for padding dimensions in 1D convolution. Can you manually pad it to 2D?

[vermorel]

Thanks! It seems to work when I craft a pseudo 1D convolution through a 2D setup. However, for some reason, it seems that the second dimension should be strictly greater than 1; which complicates further the setup.

[falahati]

Any progress on this?

cuDNN failure 8: CUDNN_STATUS_EXECUTION_FAILED ; GPU=0 ; hostname=DESKTOP-PC ; expr=workspaceSizeFinder()
ApplicationException - Error: System.ApplicationException: cuDNN failure 8: CUDNN_STATUS_EXECUTION_FAILED ; GPU=0 ; hostname=DESKTOP-PC ; expr=workspaceSizeFinder()

[KeDengMS]

@falahati no fix for it yet. Does the workaround above help?

[falahati]

@KeDengMS
Yeah, I managed to get it working with a 2D conv setup on c#. However, I was lucky my data could meaningfully be reshaped into a 2D space.
Considering the fact that this issue is from Dec, we really should solve this tho. Makes the code unnecessarily complicated.

[KeDengMS]

Got it, will track this.

[DiomedesDominguez]

Any news?