Onednn: Problems about zero dimension, zeroes in dimension, and etc.

[CaoZhongZ]

Er, hello guys, I'm here again.

After update to latest master, I found there was a major change in "memory_desc_sanity_check", it did not allow zeros in dimension anymore. Before this, there was a problem of whether letting zero-in-dimension tensor participate in computation, no conclusion yet.

We integrated MKL-DNN tightly into Chainer. Tightly, I mean we use MKL-DNN C API and memory primitive as tensor itself. There is few bridge between tensor in Chainer(NumPy) and MKL-DNN memory. All the information is stored inside memory primitive backend and getter is synonym of queries. We wanted no glue code, saved some memory, and avoided sync information between glue layer and backend (memory primitive). Our design is also achieved completely compatible to NumPy ND-Array (Directly operable by NumPy API).

So the major problem now is the test, we no-longer allowed to create tensors that had zeros inside dimensions (NumPy allowed). We no-longer allowed corner case tests (Zero dimension calculations). They were all work before, with only a minor glitch that batch size can not be zero. There was also a real case scenario that zero-in-dimension tensor emerged from normal calculation (Other framework). I think it might happen in Chainer soon if we run the same model.

So, I'm a bit of prefer old-time. Things were a little relaxed. You know, before RNN interface we can't even create a memory with nullptr, so I have to stuff a 42 into it and now we can use nullptr explicitly, that saves a lot mental burden. I'm saying is it possible that we support it instead of blocking it?

[emfomenk]

Hi @4pao,

What is the semantics of a tensor that has one of its dimensions is zero, while the others are not, say {0, 2, 3}? For MKL-DNN that currently makes no sense, that's why we prohibited such tensors. In fact, from time to time people try concatenating several tensors where few of them are those strange guys. For instance:

concat(axis=1, {4, 6, 16}, {4, 3, 16}, {4, 0, 16}, {4, 9, 16})

Before the recent patch MKL-DNN just crashed.

Please note that zero_md is valid memory descriptor which can be used to express the fact that this memory is not used (e.g. in convolution backward by weights with bias_md == zero_md):

zero_md = {ndims=0, dims{0, ..., 0}, data_type=*, format=*}

[CaoZhongZ]

@emfomenk We are following NumPy semantics and I think since most of frameworks are compatible with NumPy, its semantics should be versatile. For example:

In [1]: import numpy as np

In [2]: a = np.ndarray([0, 2, 3])

In [3]: a
Out[3]: array([], shape=(0, 2, 3), dtype=float64)

In [4]:

In [18]: np.concatenate((a, b))
Out[18]: array([], shape=(0, 2, 3), dtype=float64)

In [19]: np.concatenate((a, b), 1)
Out[19]: array([], shape=(0, 4, 3), dtype=float64)

In [20]: np.concatenate((a, b), 2)
Out[20]: array([], shape=(0, 2, 6), dtype=float64)

The operators guard against zero-in-dimension tensor is not a good idea and it affects us in a minor way. But if MKL-DNN does not allow to create zero-in-dimension tensor, we can't create MKL-DNN backed ND-Array in this form, that's will crash a lot. And zero_md is not enough.

[CaoZhongZ]

For semantics of concatenation, there are two-fold. One is for dimension restructuring and one is for real buffer manipulation. If there is zero in dimensions, then according to math, buffer manipulation can be avoided. Only the dimension manipulation. This kind of case is a lot in Chainer and Caffe2 too.

If MKL-DNN concat crash in this scenario, we can look into it deeply and fix it elegantly. Just blocking zero-in-dimension tensor made integration painful.

I myself did a tests like this, I was allowing MKL-DNN to accept zero-in-dimension, and fed three cases to it.

-    if (i_mdw.nelems() == 0 || o_mdw.nelems() == 0)
-        return invalid_arguments;
+//    if (i_mdw.nelems() == 0 || o_mdw.nelems() == 0)
+//        return invalid_arguments;

  concat_test_params{engine::kind::cpu, 1,
  {memory::format::nchw, memory::format::nchw}, memory::format::nchw,
  {{2, 8, 0, 1}, {2, 16, 0, 1}}, {2, 24, 0, 1}},
  concat_test_params{engine::kind::cpu, 1,
  {memory::format::nchw, memory::format::nchw}, memory::format::nchw,
  {{2, 0, 1, 1}, {2, 0, 1, 1}}, {2, 0, 1, 1}},
  concat_test_params{engine::kind::cpu, 1,
  {memory::format::nchw, memory::format::nchw}, memory::format::nchw,
  {{0, 8, 1, 1}, {0, 16, 1, 1}}, {0, 24, 1, 1}}



md5-0829c58b64961932fc012726bbcb9e52



  thread #8, stop reason = EXC_ARITHMETIC (code=EXC_I386_DIV, subcode=0x0)
    frame #0: 0x00000001003eed00 libmkldnn.0.dylib`unsigned long mkldnn::impl::utils::nd_iterator_init<unsigned long, unsigned long, int, unsigned long&, unsigned long const&>(start=0, x=0x000070000c2aeac8, X=0x00007ffeefbfde5c, tuple=0x000070000c2aeac0, tuple=0x00007ffeefbfde40) at utils.hpp:180
   177  template<typename T, typename U, typename W, typename... Args>
   178  inline T nd_iterator_init(T start, U &x, const W &X, Args &&... tuple) {
   179      start = nd_iterator_init(start, utils::forward<Args>(tuple)...);
-> 180      x = start % X;
   181      return start / X;
   182  }

It boils down to this place. How about give a definition when X is zero, clearly when X is zero this funciton's formula is un-defined. Since there is no work to do, should we avoid calling this function when there is no work to do and just return??

[vpirogov]

@4pao, it's not about this particular function. The larger problem is that MKL-DNN is not designed to handle zero in dimensions and we do not have this behavior defined. Could you please define what should happen when you pass tensor with a zero dimension to MKL-DNN?

[CaoZhongZ]

@vpirogov @emfomenk That's the thing we can do! We are collecting 0-in-dimension tensor behavior from at least 3 framework, including all their test case, and will provide a complete definition of 0-in-dimension behavior.

[CaoZhongZ]

Well, after collected requirements and did a research of this issue, I could class requirements in following two categories:

1. The allowing of existing of 0-in-dimension memory
2. Which primitive should accept 0-in-dimension memory as operand.

From all test cases collected, including a real case found in Yolo (or SSD?), the existence of 0-in-dimension memory make things a lot easier. Two primitives are required to support 0-in-dimention, concat and reorder, which boils down to reorder. There is no requirement of others, yet.

Definitions needed are list below:

• [ ] Indexing scheme. From implementation perspective, helpers like off_v, off_l, off, ..., etc, inside memory_desc_wrapper. Definition can be quite simple, all indexing attempt should be out of range. Implementing out of range might be a little challenge, however, all can be avoid if there is no one actually indexing the 0-in-dimension memory (in reorder).
• [x] Format of 0-in-dimention memory. Format criteria of current MKL-DNN applies to 0-in-dimension.
• [ ] Workload balancing of 0-in-dimension reorder. Nothing should be done, so no workload balancing should performed. Currently, nd_iterator_init might abort if accepted 0 batch size. Should it be categories as indexing scheme or balancing problem?
• [ ] Reorder a 0-in-dimension memory. It changes its format if necessary, no memory access should performed.
• [ ] Concat 0-in-dimension memories. It only involves dimension arithmetic, no memory access should performed (guaranteed in reorder)
• [ ] All other primitives remain to throw exception or return error if encounter 0-in-dimension memory.

[vpirogov]

@4pao, the statements you collected are related to existing implementation of some primitives in MKL-DNN. What @emfomenk and I are trying to understand is semantics of using 0 in the dimension. To define semantics you need to articulate what result you expect MKL-DNN to produce for given input and how this result will be used in the application.

Maybe an example of how zeroes in dimension are used in Yolo implementation would help.

[CaoZhongZ]

@emfomenk @vpirogov Let me get more detail.

[CaoZhongZ]

@emfomenk @vpirogov

Sorry for a delay. I'm reviewing the Mask-RCNN case that will trigger MKL-DNN concat exception. In Mask-RCNN process, there is a concept called RoI(Region of interest). Put the details aside, RoI might be empty for some images. Specifically in implementation in tensorflow, they use tf.image.crop_and_resize to generate new images from original images for given parameters(coordinates). It very much like view primitive concept in MKL-DNN. It generates tensors of new images by carving them out from original images and resize them, represented as N, C, H, W. As we observed, it generated N=0 cases when the set of RoI is out of images area.

I'm assuming the semantics means physical meaning of zeros-in-dimension, then there was two fold of semantics. Since what we called 4-d tensor is actually 3-d tensor with a batch number. It is hard to understand zero dimension in 3d physical object but zero batch size definitely makes sense. Since R-CNN region proposal step might actually propose no bounding box for a completely blank image, and this proposal will be resize to 224x224x3(ch) which is a 0, 3, 224, 224 tensor.

There is another discussion of same concept in PyTorch:

This is particularly useful for code using a variable number of items, e.g. boxes in object detection. Otherwise it's necessary to have all these special case checks for the empty tensor!

https://github.com/pytorch/pytorch/issues/5014

[CaoZhongZ]

This process remind me that currently all DNN computation essentially are 2-d on multiple channels, with a batch extension. From MKL-DNN code, most of them are batch agonistic in JIT. So 0 batch size is easier to define than other cases, and this is what we need at the moment. And its scope is limited in tensor representation and shape manipulation (concat, reorder). That's what I mean in a post 3 floor above.

[vpirogov]

@4pao, let me try to play back what you are saying (and the discussion in Pytorch) to make sure that I understand everything correctly. Numpy ndarray implementation allows using 0 in array dimension sizes to accomodate certain functionality related to array arithmetics and in some cases has implicit rules related to broadcasting values. The request here is to allow creating memory objects with a zero-sized dimension. Primitives created with memory that has 0 sized dimensions should not perform any memory access and return success immediately when executed.

Does this sound correct?

[CaoZhongZ]

@vpirogov Yes. The request here is to allow creating memory objects with zero-sized dimension. Currently only concat and reorder are required to behave as described, while others can wait for further analysis. In this condition, the behavior will consistent with NumPy without exception.

[CaoZhongZ]

We actually run into an error in Caffe2 fast-RCNN with zero-sized dimension tensor. BoxWithNMSLimit Op in caffe2 generate [0, 4] shaped tensor, very common.

[jgong5]

@vpirogov Is the requirement clear to you or you are still waiting for more clarification?