Theano: Questions about Theano C code syntax

[notoraptor]

UPDATE:

The question was answered, the rest of the issue is about using params for all ops that would benefit from it. There is one comment that is a big list of ops that need to do that.

As it is planned to reduce variants of C codes generated by Theano, I suggest to use this ticket to centralize all questions about that ! @abergeron @nouiz @lamblin

---

(1) Some C codes use a typedef dtype_%(var)s, but it is not guaranteed to be defined.

[fixed, nothing to do]

For example, here: https://github.com/Theano/Theano/blob/master/theano/tensor/basic.py#L2863

Code assumes there is a type called dtype_%(shp_i)s based on the name of an input shp_i. Indeed, this type is defined via a typedef in TensorType.c_declare(): https://github.com/Theano/Theano/blob/master/theano/tensor/type.py#L385

But the typedef is declared only if c_declare() is called with check_input=True.

I suggest to create this typedef in any case, whatever be the value of check_input.

[abergeron]

You should notice that check_input=True is the default. So most ops will not have a problem with that.

If an op decides to use check_input=False then the defines will not be present but that is part of the option. This will also allow a reduction in the variants of C code generated for ops that do not specifically care about the type of their inputs.

So in all I am against what you propose.

[notoraptor]

@abergeron Ok!

I have another question:

---

(2) Some literal strings are directly compared in some C codes.

[to fix]

Example here, in theano.gpuarray.neighbours.GpuImages2Neibs : https://github.com/Theano/Theano/blob/master/theano/gpuarray/neighbours.py#L316

A literal string is directly compared to another literal string. It should not work, as this does not compare strings, but instead it compares memory addresses. It does not seem to create any error in tests, and I have read that some compilers are smart enough to store a literal string in one address and reuse it everywhere this literal is used, so maybe that's what g++ does here.

Should we be more strict here, to prevent any future problem ? Or isn't it so problematic ?

[abergeron]

Yes, this is bad and should be avoided. This should be replaced with an int or a set of defines.

[notoraptor]

(3) How to manage vectors or arrays props into C code ?

[current fix: when vector length is hardcoded, we use as many attributes as vector elements into a ParamsType, and we group them into an array in C code]

Example with theano.gpuarray.dnn.GpuDnnPoolDesc: https://github.com/Theano/Theano/blob/master/theano/gpuarray/dnn.py#L1217

GpuDnnPoolDesc constructor takes ws, stride and pad as arguments, which are all vectors. To be used in a C code, they are concatenated into a string (str=', '.join(map(str, self.stride))) to create an integer C-array (static const int str[%(nd)d] = {%(str)s};).

So, C code will vary depending on the size (e.g. ws=(1,1) vs ws=(1,1,1)) and the values of these args (e.g. ws=(1,1) vs ws=(1,2)). Isn't it a too big source of variation ?

I have planned to convert them as op params with TensorType, but this will make values harder to access in a C code (as we would have to go through NumPy access functions + static conversions), and real C-arrays is needed by some C functions as arguments.

Another solution could be to create a special Theano Type (e.g. CArrayType) which convert a Python vector (list or tuple) into a C-contiguous array (int array[n];), but then we will have to convert Python vector to C-array into the C code (via c_extract), and, in terms of speed, this step will be a little slower than hardcoding array values as we do currently.

Do you think we can improve something here ?

[notoraptor]

(4) List of ops for which we could add/extend params.

• [ ] theano.compile.ops.Rebroadcast (prop: axis) (careful)
• [x] theano.compile.ops.Shape_i
• [ ] theano.gpuarray.basic_ops.GpuAlloc
• [ ] theano.gpuarray.basic_ops.GpuAllocEmpty
• [ ] theano.gpuarray.basic_ops.GpuEye
• [x] theano.gpuarray.basic_ops.GpuReshape
• [ ] theano.gpuarray.basic_ops.GpuSplit
• [ ] theano.gpuarray.blas.BaseGpuCorrMM and subclasses
• [ ] theano.gpuarray.blas.BaseGpuCorr3dMM and subclasses
• [ ] theano.gpuarray.blas.GpuDot22
• [x] theano.gpuarray.blas.GpuGemm
• [x] theano.gpuarray.blas.GpuGemmBatch
• [x] theano.gpuarray.blas.GpuGemv
• [x] theano.gpuarray.blas.GpuGer
• [ ] theano.gpuarray.blocksparse.GpuSparseBlockGemv
• [ ] theano.gpuarray.blocksparse.GpuSparseBlockOuter
• [x] theano.gpuarray.dnn.GpuDnnBatchNormGrad
• [x] theano.gpuarray.dnn.GpuDnnBatchNormInference
• [x] theano.gpuarray.dnn.GpuDnnConv
• [x] theano.gpuarray.dnn.GpuDnnConvDesc
• [x] theano.gpuarray.dnn.GpuDnnConvGradI
• [x] theano.gpuarray.dnn.GpuDnnConvGradW
• [x] theano.gpuarray.dnn.GpuDnnPool
• [x] theano.gpuarray.dnn.GpuDnnPoolGrad
• [x] theano.gpuarray.dnn.GpuDnnSoftmaxBase and sub-classes
• [x] theano.gpuarray.elemwise.GpuDimShuffle
• [x] theano.gpuarray.extra_ops.GpuCumOp
• [ ] theano.gpuarray.multinomial.GPUAChoiceFromUniform
• [ ] theano.gpuarray.multinomial.GPUAMultinomialFromUniform
• [x] theano.gpuarray.neighbours.GpuImages2Neibs
• [ ] theano.gpuarray.nnet.GpuCrossentropySoftmax1HotWithBiasDx
• [ ] theano.gpuarray.nnet.GpuCrossentropySoftmaxArgmax1HotWithBias
• [ ] theano.gpuarray.nnet.GpuSoftmax
• [ ] theano.gpuarray.nnet.GpuSoftmaxWithBias
• [x] theano.gpuarray.rng_mrg.GPUA_mrg_uniform
• [x] theano.gpuarray.subtensor.GpuAdvancedIncSubtensor1
• [x] theano.gpuarray.subtensor.GpuAdvancedIncSubtensor1_dev20
• [ ] theano.sandbox.multinomial.ChoiceFromUniform
• [ ] theano.sandbox.multinomial.MultinomialFromUniform
• [x] theano.sandbox.rng_mrg.mrg_uniform
• [ ] theano.sparse.opt.AddSD_ccode
• [ ] theano.sparse.opt.UsmmCscDense
• [ ] theano.tensor.basic.Alloc
• [ ] theano.tensor.basic.AllocEmpty
• [ ] theano.tensor.basic.Flatten
• [x] theano.tensor.basic.Reshape
• [ ] theano.tensor.basic.Split
• [x] theano.tensor.blas.Gemm
• [x] theano.tensor.blas_c.CGemv
• [x] theano.tensor.blas_c.CGer
• [x] theano.tensor.elemwise.DimShuffle (hard)
• [x] theano.tensor.nnet.neighbours.Images2Neibs
• [ ] theano.tensor.signal.pool.AveragePoolGrad
• [ ] theano.tensor.signal.pool.MaxPoolGrad
• [ ] theano.tensor.signal.pool.MaxPoolRop
• [ ] theano.tensor.signal.pool.Pool
• [x] theano.tensor.subtensor.AdvancedIncSubtensor1
• [ ] theano.tensor.subtensor.IncSubtensor
• [x] theano.tensor.nnet.corr.BaseCorrMM and sub-classes
• [x] theano.tensor.nnet.corr3d.BaseCorr3dMM and sub-classes
• [ ] theano.typed_list.basic.Reverse

[abergeron]

(3) I think the CArrayType is a good idea. The conversion would happen only once during graph compilation and then be reused for multiple executions so I don't think it would be a big cost to pay.

[notoraptor]

List updated for question 4 (see above).

I will start to work with theano.gpuarray.dnn.GpuDnnSoftmax.

From what I understand about GpuDnnSoftmax, current params (mode and algo) are used as macros (so C code changes every time macros change), and we would prefer to use them as something like an enum instances that will be passed at runtime (so that C code will not need to be updated). Is it right ?

What I think currently is to use Wrapper as follow:

• store enum values as small integers.
• store params as integers, so that a param is equal to the corresponding enum value defined above.
• test params values into C code to choose right macro values.

For example, if a param mode should be chosen in {'instance', 'channel'}, then a wrapper could look like:

• wrapper.INSTANCE = 0
• wrapper.CHANNEL = 1
• wrapper.mode = either wrapper.INSTANCE or wrapper.CHANNEL.

Then, in C code we could do something like it:

if (wrapper->mode == wrapper->INSTANCE) {
    mode = CUDNN_SOFTMAX_MODE_INSTANCE;
} else {
    mode = CUDNN_SOFTMAX_MODE_CHANNEL;
}

and continue working with mode.

Is it what we want, @lamblin @abergeron @nouiz ?

[abergeron]

That looks like a good idea, yes.

I'm less positive about the way you define "constants", but otherwise ok.

[notoraptor]

@abergeron ok. Maybe I could define constants as macros (as they won't change as long as op fields are not modified), and only param value is passed at runtime.

// This would not change from an op instance to another, so no need to recompile.
#define MODE_INSTANCE 0
#define MODE_CHANNEL 1

// ...

if (wrapper->mode == MODE_INSTANCE) { /* ... */ }

I'll look for a way to add macros easily with get_op_params().

Maybe by just adding a new independent abstract field into COp, like enum_type, and then checking it in get_op_params() if it is defined for an op:

enum_type = ('MODE_INSTANCE', 'MODE_CHANNEL').

EDIT: with work to do around, but it's the idea.

[gokul-uf]

@notoraptor I have a PR #5402 that is along these lines. Could you review it?

[lamblin]

Regarding (1), the idea is that sometimes the C code does not depend at all from the dtype of its inputs. This happens for instance when we only call dtype-independent functions of the NumPy C API.
In that case, the exact same C code string gets generated, and we are able to re-use the compiled .so (or .dll) from the cache. Unless check_inputs is true, in which case the input-checking boilerplate code forces generation of different cases for different dtypes.

[notoraptor]

(5) Old back-end class in sandbox/mrg_mrg

[fixed by abergeron]

As we plan to drop the old back-end, should we have to also drop the class theano.sandbox.rng_mrg.GPU_mrg_uniform (or move it into old back-end module) ? It is defined here: https://github.com/Theano/Theano/blob/master/theano/sandbox/rng_mrg.py#L565

[lamblin]

(5) Yes, it will be removed. I think @abergeron is already working on it.

[notoraptor]

(6) Useless macro definitions in COps ?

[fixed - no real problems here - use YourOp.check_input=False when necessary]

While working on GpuDnnSoftmax, I have compared two different mod.cpp files produced for this operation, and I got this diff file:

--- /home/notoraptor/.theano/compiledir_Linux-4.4--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.13-64/tmpxWMCRQ/mod.cpp
+++ /home/notoraptor/.theano/compiledir_Linux-4.4--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.13-64/tmpYidHmR/mod.cpp
@@ -276,7 +276,7 @@


     namespace {
-    struct __struct_compiled_op_cef4e56ee534f343a147a522bacf9aba {
+    struct __struct_compiled_op_928624aa454ff9ec7ce25ef2ba37595d {
         PyObject* __ERROR;

         PyObject* storage_V3;
@@ -287,13 +287,13 @@

         _Params_9422d00378f3a279a00a9672b6a1bd76_579f4ed96a1ad3c07ec0dc451c710996* V5;

-#define DTYPE_INPUT_0 npy_float16
-#define TYPENUM_INPUT_0 23
-#define ITEMSIZE_INPUT_0 2
-#define DTYPE_OUTPUT_0 npy_float16
-#define TYPENUM_OUTPUT_0 23
-#define ITEMSIZE_OUTPUT_0 2
-#define APPLY_SPECIFIC(str) str##_node_cef4e56ee534f343a147a522bacf9aba_0
+#define DTYPE_INPUT_0 npy_float32
+#define TYPENUM_INPUT_0 11
+#define ITEMSIZE_INPUT_0 4
+#define DTYPE_OUTPUT_0 npy_float32
+#define TYPENUM_OUTPUT_0 11
+#define ITEMSIZE_OUTPUT_0 4
+#define APPLY_SPECIFIC(str) str##_node_928624aa454ff9ec7ce25ef2ba37595d_0
 #define PARAMS_TYPE _Params_9422d00378f3a279a00a9672b6a1bd76_579f4ed96a1ad3c07ec0dc451c710996


@@ -386,7 +386,7 @@
 #undef APPLY_SPECIFIC
 #undef PARAMS_TYPE

-        __struct_compiled_op_cef4e56ee534f343a147a522bacf9aba() {
+        __struct_compiled_op_928624aa454ff9ec7ce25ef2ba37595d() {
             // This is only somewhat safe because we:
             //  1) Are not a virtual class
             //  2) Do not use any virtual classes in the members
@@ -398,7 +398,7 @@
             // a junk value.
             memset(this, 0, sizeof(*this));
         }
-        ~__struct_compiled_op_cef4e56ee534f343a147a522bacf9aba(void) {
+        ~__struct_compiled_op_928624aa454ff9ec7ce25ef2ba37595d(void) {
             cleanup();
         }

@@ -461,13 +461,13 @@
         }


-#define DTYPE_INPUT_0 npy_float16
-#define TYPENUM_INPUT_0 23
-#define ITEMSIZE_INPUT_0 2
-#define DTYPE_OUTPUT_0 npy_float16
-#define TYPENUM_OUTPUT_0 23
-#define ITEMSIZE_OUTPUT_0 2
-#define APPLY_SPECIFIC(str) str##_node_cef4e56ee534f343a147a522bacf9aba_0
+#define DTYPE_INPUT_0 npy_float32
+#define TYPENUM_INPUT_0 11
+#define ITEMSIZE_INPUT_0 4
+#define DTYPE_OUTPUT_0 npy_float32
+#define TYPENUM_OUTPUT_0 11
+#define ITEMSIZE_OUTPUT_0 4
+#define APPLY_SPECIFIC(str) str##_node_928624aa454ff9ec7ce25ef2ba37595d_0
 #define PARAMS_TYPE _Params_9422d00378f3a279a00a9672b6a1bd76_579f4ed96a1ad3c07ec0dc451c710996
 #define FAIL { \
         if (!PyErr_Occurred()) { \
@@ -530,13 +530,13 @@
 double __DUMMY_5;
 __label_8:

-#define DTYPE_INPUT_0 npy_float16
-#define TYPENUM_INPUT_0 23
-#define ITEMSIZE_INPUT_0 2
-#define DTYPE_OUTPUT_0 npy_float16
-#define TYPENUM_OUTPUT_0 23
-#define ITEMSIZE_OUTPUT_0 2
-#define APPLY_SPECIFIC(str) str##_node_cef4e56ee534f343a147a522bacf9aba_0
+#define DTYPE_INPUT_0 npy_float32
+#define TYPENUM_INPUT_0 11
+#define ITEMSIZE_INPUT_0 4
+#define DTYPE_OUTPUT_0 npy_float32
+#define TYPENUM_OUTPUT_0 11
+#define ITEMSIZE_OUTPUT_0 4
+#define APPLY_SPECIFIC(str) str##_node_928624aa454ff9ec7ce25ef2ba37595d_0
 #define PARAMS_TYPE _Params_9422d00378f3a279a00a9672b6a1bd76_579f4ed96a1ad3c07ec0dc451c710996


@@ -573,9 +573,47 @@

 {

-    py_V1 = Py_None;
+    py_V1 = PyList_GET_ITEM(storage_V1, 0);
     {Py_XINCREF(py_V1);}
-    V1 = NULL;
+    
+        if (py_V1 == Py_None)
+        {
+            V1 = NULL;
+        }
+        else
+        {
+            
+        V1 = NULL;
+        if (py_V1 == Py_None) {
+            PyErr_SetString(PyExc_ValueError, "expected a GpuArray, not None");
+            {
+        __failure = 2;
+        if (!PyErr_Occurred()) {
+            PyErr_SetString(PyExc_RuntimeError,
+                "Unexpected error in an Op's C code. "
+                "No Python exception was set.");
+            }
+        goto __label_2;}
+        }
+        /* First check if we are the base type exactly (the most common case),
+           then do the full subclass check if needed. */
+        if (py_V1->ob_type != &PyGpuArrayType &&
+            !PyObject_TypeCheck(py_V1, &PyGpuArrayType)) {
+            PyErr_SetString(PyExc_ValueError, "expected a GpuArray");
+            {
+        __failure = 2;
+        if (!PyErr_Occurred()) {
+            PyErr_SetString(PyExc_RuntimeError,
+                "Unexpected error in an Op's C code. "
+                "No Python exception was set.");
+            }
+        goto __label_2;}
+        }
+        V1 = (PyGpuArrayObject *)py_V1;
+        Py_INCREF(V1);
+        
+        }
+        
 {

     py_V3 = PyList_GET_ITEM(storage_V3, 0);
@@ -615,7 +653,7 @@
 {
 // Op class GpuDnnSoftmax

-                #define APPLY_SPECIFIC(str) str##_node_cef4e56ee534f343a147a522bacf9aba_0
+                #define APPLY_SPECIFIC(str) str##_node_928624aa454ff9ec7ce25ef2ba37595d_0
 #define PARAMS_TYPE _Params_9422d00378f3a279a00a9672b6a1bd76_579f4ed96a1ad3c07ec0dc451c710996
                 {
                   if (APPLY_SPECIFIC(softmax)(V3, &V1, V5) != 0) {
@@ -703,12 +741,12 @@
     }


-        static int __struct_compiled_op_cef4e56ee534f343a147a522bacf9aba_executor(__struct_compiled_op_cef4e56ee534f343a147a522bacf9aba* self) {
+        static int __struct_compiled_op_928624aa454ff9ec7ce25ef2ba37595d_executor(__struct_compiled_op_928624aa454ff9ec7ce25ef2ba37595d* self) {
             return self->run();
         }

-        static void __struct_compiled_op_cef4e56ee534f343a147a522bacf9aba_destructor(void* executor, void* self) {
-            delete ((__struct_compiled_op_cef4e56ee534f343a147a522bacf9aba*)self);
+        static void __struct_compiled_op_928624aa454ff9ec7ce25ef2ba37595d_destructor(void* executor, void* self) {
+            delete ((__struct_compiled_op_928624aa454ff9ec7ce25ef2ba37595d*)self);
         }

 //////////////////////
@@ -720,12 +758,12 @@
      PyErr_Format(PyExc_TypeError, "Wrong number of arguments, expected 4, got %i", (int)PyTuple_Size(argtuple));
      return NULL;
   }
-  __struct_compiled_op_cef4e56ee534f343a147a522bacf9aba* struct_ptr = new __struct_compiled_op_cef4e56ee534f343a147a522bacf9aba();
+  __struct_compiled_op_928624aa454ff9ec7ce25ef2ba37595d* struct_ptr = new __struct_compiled_op_928624aa454ff9ec7ce25ef2ba37595d();
   if (struct_ptr->init( PyTuple_GET_ITEM(argtuple, 0),PyTuple_GET_ITEM(argtuple, 1),PyTuple_GET_ITEM(argtuple, 2),PyTuple_GET_ITEM(argtuple, 3) ) != 0) {
     delete struct_ptr;
     return NULL;
   }
-  PyObject* thunk = PyCObject_FromVoidPtrAndDesc((void*)(&__struct_compiled_op_cef4e56ee534f343a147a522bacf9aba_executor), struct_ptr, __struct_compiled_op_cef4e56ee534f343a147a522bacf9aba_destructor);
+  PyObject* thunk = PyCObject_FromVoidPtrAndDesc((void*)(&__struct_compiled_op_928624aa454ff9ec7ce25ef2ba37595d_executor), struct_ptr, __struct_compiled_op_928624aa454ff9ec7ce25ef2ba37595d_destructor);
   return thunk; }

 //////////////////////
@@ -735,11 +773,11 @@
    {"instantiate", instantiate, METH_VARARGS, "undocumented"} ,
    {NULL, NULL, 0, NULL}
 };
-PyMODINIT_FUNC initcef4e56ee534f343a147a522bacf9aba(void){
+PyMODINIT_FUNC init928624aa454ff9ec7ce25ef2ba37595d(void){
    import_pygpu__gpuarray();


 setup_ext_cuda();

-   (void) Py_InitModule("cef4e56ee534f343a147a522bacf9aba", MyMethods);
-}
+   (void) Py_InitModule("928624aa454ff9ec7ce25ef2ba37595d", MyMethods);
+}

Although code is extremely similar, there are differences which seem to come from 2 sources:

• the macro definitions for DTYPE_(INPUT|OUTPUT)_*, TYPENUM_(INPUT|OUTPUT)_*, and ITEMSIZE_(INPUT|OUTPUT)_*. I remark than DTYPE_* macros are not used in main codes for this op.
• The extraction code for output storage (variable V1). It seems that in one case, there is no such code, probably because storage is set to None, and in the other case, the extraction code is added because output storage contains something.

These are minor changes, but because of them, we could have 2 or more compilations even when main codes does not vary at all, and it is a general problem (not only for GpuDnnSoftmax) when using COps. So, I suggest some changes:

• completely remove TYPENUM_* and ITEMSIZE_* macros, as I think these values could be retrieved at runtime, for example with PyArray_TYPE() and PyArray_ITEMSIZE() for PyArrayObjects, and it should not be so hard to get them for GpuArray objects also.
• DTYPE_* macros are used in some codes and cannot be retrieved at runtime as they are type names. But maybe we can check (e.g. with a regex) if a code use them before creating them ? With such checking, we could avoid add DTYPE_* macros when they are not used at all in C code (as in this case).
• Always add code extraction for output storages even when there are no storage, and then always call c_extract() with check_input==True when we want to get C extraction code for an output object.

@abergeron @nouiz @lamblin

[abergeron]

We could skip generating the macros for ops that set check_input = False. Or we can add a new parameter to COp.__init__ that disables those macros.

[notoraptor]

@abergeron indeed ! In fact, method COp.get_c_macros() will already skip generating the macros if COp does have an attribute check_input set to False. So I will update GpuDnnSoftmaxBase op.

Now the last difference seems always about the extraction code for output storage:

--- /home/notoraptor/.theano/compiledir_Linux-4.4--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.13-64/tmpHAnF5H/mod.cpp
+++ /home/notoraptor/.theano/compiledir_Linux-4.4--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.13-64/tmpIKAzom/mod.cpp
@@ -276,7 +276,7 @@


     namespace {
-    struct __struct_compiled_op_8d09d3f9714a2bcb6117db6464e6b8d2 {
+    struct __struct_compiled_op_9875e5964f54592ba0ede19d881a1bc2 {
         PyObject* __ERROR;

         PyObject* storage_V3;
@@ -287,7 +287,7 @@

         _Params_9422d00378f3a279a00a9672b6a1bd76_579f4ed96a1ad3c07ec0dc451c710996* V5;

-#define APPLY_SPECIFIC(str) str##_node_8d09d3f9714a2bcb6117db6464e6b8d2_0
+#define APPLY_SPECIFIC(str) str##_node_9875e5964f54592ba0ede19d881a1bc2_0
 #define PARAMS_TYPE _Params_9422d00378f3a279a00a9672b6a1bd76_579f4ed96a1ad3c07ec0dc451c710996


@@ -374,7 +374,7 @@
 #undef APPLY_SPECIFIC
 #undef PARAMS_TYPE

-        __struct_compiled_op_8d09d3f9714a2bcb6117db6464e6b8d2() {
+        __struct_compiled_op_9875e5964f54592ba0ede19d881a1bc2() {
             // This is only somewhat safe because we:
             //  1) Are not a virtual class
             //  2) Do not use any virtual classes in the members
@@ -386,7 +386,7 @@
             // a junk value.
             memset(this, 0, sizeof(*this));
         }
-        ~__struct_compiled_op_8d09d3f9714a2bcb6117db6464e6b8d2(void) {
+        ~__struct_compiled_op_9875e5964f54592ba0ede19d881a1bc2(void) {
             cleanup();
         }

@@ -449,7 +449,7 @@
         }


-#define APPLY_SPECIFIC(str) str##_node_8d09d3f9714a2bcb6117db6464e6b8d2_0
+#define APPLY_SPECIFIC(str) str##_node_9875e5964f54592ba0ede19d881a1bc2_0
 #define PARAMS_TYPE _Params_9422d00378f3a279a00a9672b6a1bd76_579f4ed96a1ad3c07ec0dc451c710996
 #define FAIL { \
         if (!PyErr_Occurred()) { \
@@ -506,7 +506,7 @@
 double __DUMMY_5;
 __label_8:

-#define APPLY_SPECIFIC(str) str##_node_8d09d3f9714a2bcb6117db6464e6b8d2_0
+#define APPLY_SPECIFIC(str) str##_node_9875e5964f54592ba0ede19d881a1bc2_0
 #define PARAMS_TYPE _Params_9422d00378f3a279a00a9672b6a1bd76_579f4ed96a1ad3c07ec0dc451c710996


@@ -537,9 +537,47 @@

 {

-    py_V1 = Py_None;
+    py_V1 = PyList_GET_ITEM(storage_V1, 0);
     {Py_XINCREF(py_V1);}
-    V1 = NULL;
+    
+        if (py_V1 == Py_None)
+        {
+            V1 = NULL;
+        }
+        else
+        {
+            
+        V1 = NULL;
+        if (py_V1 == Py_None) {
+            PyErr_SetString(PyExc_ValueError, "expected a GpuArray, not None");
+            {
+        __failure = 2;
+        if (!PyErr_Occurred()) {
+            PyErr_SetString(PyExc_RuntimeError,
+                "Unexpected error in an Op's C code. "
+                "No Python exception was set.");
+            }
+        goto __label_2;}
+        }
+        /* First check if we are the base type exactly (the most common case),
+           then do the full subclass check if needed. */
+        if (py_V1->ob_type != &PyGpuArrayType &&
+            !PyObject_TypeCheck(py_V1, &PyGpuArrayType)) {
+            PyErr_SetString(PyExc_ValueError, "expected a GpuArray");
+            {
+        __failure = 2;
+        if (!PyErr_Occurred()) {
+            PyErr_SetString(PyExc_RuntimeError,
+                "Unexpected error in an Op's C code. "
+                "No Python exception was set.");
+            }
+        goto __label_2;}
+        }
+        V1 = (PyGpuArrayObject *)py_V1;
+        Py_INCREF(V1);
+        
+        }
+        
 {

     py_V3 = PyList_GET_ITEM(storage_V3, 0);
@@ -579,7 +617,7 @@
 {
 // Op class GpuDnnSoftmax

-                #define APPLY_SPECIFIC(str) str##_node_8d09d3f9714a2bcb6117db6464e6b8d2_0
+                #define APPLY_SPECIFIC(str) str##_node_9875e5964f54592ba0ede19d881a1bc2_0
 #define PARAMS_TYPE _Params_9422d00378f3a279a00a9672b6a1bd76_579f4ed96a1ad3c07ec0dc451c710996
                 {
                   if (APPLY_SPECIFIC(softmax)(V3, &V1, V5) != 0) {
@@ -667,12 +705,12 @@
     }


-        static int __struct_compiled_op_8d09d3f9714a2bcb6117db6464e6b8d2_executor(__struct_compiled_op_8d09d3f9714a2bcb6117db6464e6b8d2* self) {
+        static int __struct_compiled_op_9875e5964f54592ba0ede19d881a1bc2_executor(__struct_compiled_op_9875e5964f54592ba0ede19d881a1bc2* self) {
             return self->run();
         }

-        static void __struct_compiled_op_8d09d3f9714a2bcb6117db6464e6b8d2_destructor(void* executor, void* self) {
-            delete ((__struct_compiled_op_8d09d3f9714a2bcb6117db6464e6b8d2*)self);
+        static void __struct_compiled_op_9875e5964f54592ba0ede19d881a1bc2_destructor(void* executor, void* self) {
+            delete ((__struct_compiled_op_9875e5964f54592ba0ede19d881a1bc2*)self);
         }

 //////////////////////
@@ -684,12 +722,12 @@
      PyErr_Format(PyExc_TypeError, "Wrong number of arguments, expected 4, got %i", (int)PyTuple_Size(argtuple));
      return NULL;
   }
-  __struct_compiled_op_8d09d3f9714a2bcb6117db6464e6b8d2* struct_ptr = new __struct_compiled_op_8d09d3f9714a2bcb6117db6464e6b8d2();
+  __struct_compiled_op_9875e5964f54592ba0ede19d881a1bc2* struct_ptr = new __struct_compiled_op_9875e5964f54592ba0ede19d881a1bc2();
   if (struct_ptr->init( PyTuple_GET_ITEM(argtuple, 0),PyTuple_GET_ITEM(argtuple, 1),PyTuple_GET_ITEM(argtuple, 2),PyTuple_GET_ITEM(argtuple, 3) ) != 0) {
     delete struct_ptr;
     return NULL;
   }
-  PyObject* thunk = PyCObject_FromVoidPtrAndDesc((void*)(&__struct_compiled_op_8d09d3f9714a2bcb6117db6464e6b8d2_executor), struct_ptr, __struct_compiled_op_8d09d3f9714a2bcb6117db6464e6b8d2_destructor);
+  PyObject* thunk = PyCObject_FromVoidPtrAndDesc((void*)(&__struct_compiled_op_9875e5964f54592ba0ede19d881a1bc2_executor), struct_ptr, __struct_compiled_op_9875e5964f54592ba0ede19d881a1bc2_destructor);
   return thunk; }

 //////////////////////
@@ -699,11 +737,11 @@
    {"instantiate", instantiate, METH_VARARGS, "undocumented"} ,
    {NULL, NULL, 0, NULL}
 };
-PyMODINIT_FUNC init8d09d3f9714a2bcb6117db6464e6b8d2(void){
+PyMODINIT_FUNC init9875e5964f54592ba0ede19d881a1bc2(void){
    import_pygpu__gpuarray();


 setup_ext_cuda();

-   (void) Py_InitModule("8d09d3f9714a2bcb6117db6464e6b8d2", MyMethods);
-}
+   (void) Py_InitModule("9875e5964f54592ba0ede19d881a1bc2", MyMethods);
+}

[abergeron]

The difference in the extraction code is probably about recycle or inplace.

[notoraptor]

Yes, it seems it comes from here: https://github.com/Theano/Theano/blob/master/theano/gof/cc.py#L704

When I just set

                    policy = [[get_nothing, get_nothing, get_nothing],
                              [get_c_declare, get_c_extract_out,
                                  (get_c_sync, get_c_cleanup)]]

in any case as long as variable is in self.outputs, then I solve the problem (only 1 compilation for GpuDnnSoftmax when running nosetests -xvs theano/gpuarray/tests/test_dnn.py:test_SoftMax !)

[abergeron]

I wouldn't change this just to make the tests compile less. It will affect every single node that is compiled in Theano, in any situation, and might break some cases.

So I think we should just accept that there will be two compiles in this case.

[notoraptor]

Ok !

[notoraptor]

Hello ! Is there a way to know the device in which we work (cuda, or opencl) where we use the new back-end in C code ? In Python we can use context.kind, but it seems it is more complicated in C code.

If not, I think we could add an option to know it (e.g. a new flag for context). This may help to use some devices capabilities only when we are in such device (e.g. calling gpuarray_get_extension() only when we know we are on expected device). @abergeron @nouiz @lamblin ?

[nouiz]

It can be done, but I forgot how. @abergeron knows.

[abergeron]

In python we only know what kind it is by remembering what was passed in the constructor.

The only way to do it is to remember what type it is or to trust the caller.

[notoraptor]

NB: Sorry, I was asking it to help use CUDA streams for Baidu CTC integration only when we are on a CUDA device library ( #5949 ). But ultimately, Baidu CTC library calls some CUDA routines and will then fail if we are on an OPENCL device.

So it is useless, at the end.

[nouiz]

Should this issue be closed? What about the big list of ops?

[notoraptor]

I will update the list. Do we want to add op params to remaining ops for the next release ?

Anyway, we can close this issue, the list will remain available.