Onednn: What is the philosophy behind 'view' primitive

[CaoZhongZ]

How to use it? A little baffling.

Why don't let it inherit from cpu_memory_t, and act like memory primitive. Seems a lot of case there would be zero-copy.

Can view primitive be used in like convolution_forward? does MKL-DNN automatically adjust offset according to view's descriptor?

[emfomenk]

Hi @4pao,

Sorry for delay with response.
Yeah, view is not the simplest thing in MKL-DNN. One of the reason for that is we developed it with desire to have extendability at a price of loosing simplicity.

Let me try to describe how view works now (though that might be adjusted in future because it is definitely awful now).

View allows you only describes a part of memory (represented by memory descriptor and memory primitive descriptor). The only way to use it is to create view_pd and then either use it directly where memory primitive descriptor is required (e.g. to create a reorder, see example [1]) or query for corresponding memory descriptor (and then use it to create a primitive, see example [2]).

Example [1] based on intelcaffe.
Let's say you want to copy a part of memory into another buffer (and probably change the format).
Then your steps are:

1. create memory primitive descriptor in_mem_pd and memory primitive in_mem_p for the entire source data
2. create view primitive descriptor in_submem_pd based on in_mem_pd, initial offsets, and sub-sizes
3. create memory primitive descriptor out_mem_pd and memory primitive out_mem_p for the output (the logical sizes should much sub-sizes used in step 2, but the format might be arbitrary)
4. create reorder primitive descriptor reorder_pd based on in_submem_pd and out_mem_pd
5. create reorder primitive itself based on reorder_pd, in_mem_p, and out_mem_p

Please notice that there is no view primitive. There is only view primitive descriptor. And the reorder uses source memory as input but traverses it according to a view in_submem_pd.

Example [2] based on a hope that this works :).
Let's say you want to perform a convolution on a part of memory you need.
Then your steps are:

1. create memory primitive descriptor in_mem_pd and memory primitive in_mem_p for the entire source data
2. create view primitive descriptor in_submem_pd based on in_mem_pd, initial offsets, and sub-sizes
3. query in_submem_pd for memory descriptor in_submem_d (see mkldnn_query_memory_d in include/mkldnn_types.h)
4. create convolution descriptor conv_d based on in_submem_d and other parameters
   ...
   n. create a convolution itself based on convolution primitive descriptor (which is based on conv_d), and in_mem_p

Again you pass initial memory, but convolution has to traverse only the part that is described by the view.

Of course there is some redundancy here. In particular view primitive descriptor is very close to a memory primitive descriptor. In fact to some extend:

memory_primitive_desc(view_pd->query(memory_d)) == view_pd

We might want get rid of view primitive descriptor and have clearer API for describing submemory in memory descriptor... In fact one can do that right now if he fully understands how memory descriptor works. Again, view is just a tool that helps to fill memory descriptor correctly.

[CaoZhongZ]

Thanks for the details, the second case is very enlighten. Use view_pd to create primitive instead use view as an input. And according to code it had something to do with memory's blocking access. Is it non-continues memory supported or in-progress?

We had a case, the Googlenet concat three memory into one and during backward it broke into three. Currently we use concat to copy into single chunk of memory and use view_pd/reorder to copy out three pieces. I wonder if there is any zero-copy chance.

[emfomenk]

Hi @4pao,

Is it non-continues memory supported or in-progress?

not sure I got the question... memory structure does support non-continuous memory thanks to strides. E.g. if sizes = {16, 6} and strides = {8, 1}, there would be an extra space of 2 elements between the rows. The same is applied to blocked memory: the strides between elements and blocks are controlled by strides[2][ndims] field in memory_desc_t.

I wonder if there is any zero-copy chance.

For forward pass the concat is required. MKL-DNN has a special primitive for that, so you don't need to do it yourself. There is no chance for zero-copy here.
The view might help to avoid explicit split operation, where you need only a part of channels. In theory that should work. I've never seen that on practice though..
IntelCaffe uses view+reorder for explicit extraction (most likely for simplicity).

[CaoZhongZ]

Hi @emfomenk, thanks for the reply.

Yes, you get the question good. And we seeking zero-copy opportunity is a wrong direction. After carefully analyzed our performance data, we found the problem might related to other issues, so please see the mail I sent to you and the pull request I recently created. Thank you.

[pinzhenx]

Hi @emfomenk, do all primitives guarantee to work on non-contiguous (but plain) memory? Is there any primitive make assumptions on the continuity of any dim?

Update: It seems that this doesn't even hold for a simple binary primitive given these two inputs:

memory src0({{2, 4}, dt::f32, {4, 1}}, eng, src0_buf);
memory src1({{2, 4}, dt::f32, {16, 4}}, eng, src1_buf);
memory dst({{2, 4}, dt::f32, {4, 1}}, eng, dst_buf);

So the rule of thumb is never to use a noncontiguous memory?

[emfomenk]

Hi @pinzhenx,

I would expect that all primitives should at least be able to dispatch the reference implementation if strided inputs/outputs are used.
Currently, the following primitives are somewhat _optimized_ for non-contiguous memory on CPU:

• reorder: generic jit supports arbitrary strides and blocking
• concat and sum: since there is an implementation that is based on reorders
• matmul: since it uses inner product underneath that has native support for LD (leading dimension). note, that at least one stride should be 1 though

As I said, other primitives should at least use reference implementation. IMO, binary, as an auxiliary and quite popular primitive, should be optimized for strided input and outputs too. My suggestion will be to open an issue with relevant reproducer and comments on why you need this (for us to prioritize).