One: Clarifying odd(?) shapes

[wateret]

While I was working on #4693 , I got some questions about tensor shape. There are some shapes that are not clear to me. I would appreciate if someone can clarify.

I'm going to use [?, ?, ?] notation for shapes.

Shape | Description | What I know | Size in onert
-- | -- | -- | --
[] | Zero-dimension shape | Scalar type | 1
[0] | 1D 0-sized | Literally 0-sized array(no data) | 0
[1, 0, 3, 4] | 0 for some particular dims | ? | 0
[1] |   | Could be a dummy shape for unspecified tensor? | 1

Questions

• Is it correct [] is size of 1, while [0] is 0 ?
• Is it there actual difference [] between [1]?
• [0] tensor vs. optional tensor - it looks like [0] tensor can be replaced by an optional tensor should we allow both?
• AFAIK unspecified(dynamic) tensor is represented as [1]. As long as its shape signature(tflite) is -1, does shape value here still matter?
• What does [1, 0, 3, 4] mean? is it useful in any way?
• Are these semantics identical for tensorflow/tflite(circle)/NNAPI?

cc: @Samsung/nnfw @samsung/nncc

[hseok-oh]

Are these semantics identical for tensorflow/tflite(circle)/NNAPI?

NNAPI define zero dimension when dimension value is unspecified.

 * A tensor operand type of specified rank but some number of
 * unspecified dimensions is represented by setting dimensionCount to
 * the rank and each unspecified dimension to 0.
 *
 * Available since API level 27.
 *
 * Starting at API level 29, a tensor operand type of unspecified rank is
 * represented by setting dimensionCount to 0 and dimensions to NULL (just as if
 * it were a scalar operand type).

[seanshpark]

What I assume and not sure...

Is it correct [] is size of 1, while [0] is 0 ?

Yes

Is it there actual difference [] between [1]?

[] is scalar. but [1] is size 1 vector. I think memory structure is the same.

[0] tensor vs. optional tensor - it looks like [0] tensor can be replaced by an optional tensor should we allow both?

Don't know either...

AFAIK unspecified(dynamic) tensor is represented as [1].
Does shape value here still matter? if its shape signature(tflite) is -1

I think this is for BATCH. So user may give another value for batch jobs. signature -1 is for that this is unknown, to distinghish real [1] or unknown [?]. I think when TFlite didn't handle unknown batch size, toco just set this to 1 so that interpreter could run. Just my guess.

What does [1, 0, 3, 4] mean? is it useful in any way?

I have also question for this. I've met this on with our in-house model. it's strange. I think 0 is like -1 that is unknown.
So why 0 not -1... maybe someone used uint value for the dimension...

Are these semantics identical for tensorflow/tflite(circle)/NNAPI?

Well, we have to figure this out... My guess is it is not identical.

[llFreetimell]

Following answers are as of my experience.
If there are something wrong, let me know.

---

Is it correct [] is size of 1, while [0] is 0 ?

Yes.

---

Is it there actual difference [] between [1]?

[] is scalar and [1] is non-scalar.
In cpp, following example shows similar situation.

int a;    // Scalar
int b[1]; // non-scalar

int c = a + b    // Not possible : Scalar + Non-scalar
int d = a + b[0] // Possible : Scalar + Scalar

Therefore, for TensorFlow, we should fit the rank.

a = tf.zeros([])
b = tf.zeros([1])

c = tf.concat([a,b], axis=0) # Not possible
d = tf.add(tf.reshape(a,[1]), b) # Possible

---

[0] tensor vs. optional tensor - it looks like [0] tensor can be replaced by an optional tensor should we allow both?

Mostly Not. When we represent optional tensor, it means that there is no tensor for calculating.
However, [0] tensor is alive so when tensor access to the [0] tensor as optional tensor, the operation think that they have a tensor for optional tensor.
Let's think about following cpp codes.

// Return a*b+c. (c is optional)
// If c is nullptr, just return a*b.
int mul_add(int *a, int *b, int *c)
{
  int ans = (*a) * (*b);
  if (c != nullptr)
    ans += (*c);
  return ans;
}

// Return a*b*c. (c is optional)
// If c is nullptr, just return a*b.
int mul_mul(int *a, int *b, int *c)
{
  int ans = (*a) * (*b);
  if (c != nullptr)
    ans *= (*c);
  return ans;
}

If an operation is like mul_add, it may be replaced by [0] tensor.
However, if an operation is like mul_mul, [0] tensor should not be used as optional tensor.

---

AFAIK unspecified(dynamic) tensor is represented as [1]. Does shape value here still matter? if its shape signature(tflite) is -1

In theory, -1 in shape_signature means that the dimension can be any value.
However, TFLite converter always set those dimensions as 1 as of now.

---

What does [1, 0, 3, 4] mean? is it useful in any way?

In TensorFlow, zero sized tensor is valid and very useful.
For example, let's suppose that we want to make a list from 0 to 9.
Then, most of us will think of following codes in python

result = []
for i in range(10):
  result.append(i)

Then how can we represent this in TensorFlow?
There are two possible answers as following.

result_tensor = tf.zeros([1,0])
for i in range(10):
  result_tensor = tf.concat([result_tensor, tf.constant(i)], axis=1)

result_tensor = tf.zeros([1,10])
for i in range(10):
  result_tensor = (tf.concat([result_tensor, tf.constant(i)], axis=1))[:,1:]

Which one do you prefer? It will gives you the answer :)

Note However, TFLite converter usually removes all the zero sized tensor during optimizations. Therefore, there is not much cases that zero sized tensor is alive in TFLite model.

---

Are these semantics identical for tensorflow/tflite(circle)/NNAPI?

Between TensorFlow, TFLite, circle, identical AFAIK.
However, I don't know about NNAPI at all :(

[wateret]

@llFreetimell Thank you for your response.

Note However, TFLite converter usually removes all the zero sized tensor during optimizations. Therefore, there is not much cases that zero sized tensor is alive in TFLite model.

It sounds like the zeros are useful during model building, but not the result model file. And I still wonder it would be useful in the result model files.

[hyunsik-yoon]

AFAIK unspecified(dynamic) tensor is represented as [1]. Does shape value here still matter? if its shape signature(tflite) is -1

Yes. tflite interpreter considers Input tensor with shape [1] and shape signature [-1]) to Input tensor with shape [1] if something like resize is not called. Therefore, if resize is not called, we must consider the input shape [1].

Note However, TFLite converter usually removes all the zero sized tensor during optimizations. Therefore, there is not much cases that zero sized tensor is alive in TFLite model.

It sounds like the zeros are useful during model building, but not the result model file. And I still wonder it would be useful in the result model files.

I guess dim == 0 is still useful if the model file contains any shape of which dim == 0.
I guess , in the code @llFreetimell mentioned, my guess is that the initial shape of result_tensor in model file would be [1,0] and runtime should handle shape with dim == 0. (didn't really try thought :-0)

result_tensor = tf.zeros([1,0])
for i in range(10):
  result_tensor = tf.concat([result_tensor, tf.constant(i)], axis=1)

[wateret]

Yes. tflite interpreter considers Input tensor with shape [1] and shape signature [-1]) to Input tensor with shape [1] if something like resize is not called. Therefore, if resize is not called, we must consider the input shape [1].

@hyunsik-yoon Thank you for your response. Does this mean that a tensor can be shape of [999] (any number but 1) but its signature is [-1] ?

I guess , in the code @llFreetimell mentioned, my guess is that the initial shape of result_tensor in model file would be [1,0] and runtime should handle shape with dim == 0. (didn't really try thought :-0)

Yes, and the code looks like compiler-side code(that's what I meant by "model-building"), but I still wonder it would be useful in the result CIRCLE files. As a user(onert) of generated CIRCLE file, what would that mean?

[wateret]

Yes, and the code looks like compiler-side code(that's what I meant by "model-building"), but I still wonder it would be useful in the result CIRCLE files. As a user(onert) of generated CIRCLE file, what would that mean?

I talked to @hyunsik-yoon , and it turns out that there are some models that do something like the python code above. Before I misunderstood that it was just for compiler tools. Thanks!

[hyunsik-yoon]

Yes. tflite interpreter considers Input tensor with shape [1] and shape signature [-1]) to Input tensor with shape [1] if something like resize is not called. Therefore, if resize is not called, we must consider the input shape [1].

@hyunsik-yoon Thank you for your response. Does this mean that a tensor can be shape of [999] (any number but 1) but its signature is [-1] ?

AFAIK, no for input tensor. if a shape signature for a dim in an input tensor is -1, its matching dim in shape is always 1.
for tensors other than input tensor, I am not sure. @llFreetimell, have you seen any tensor of which dim != 1 and shape signature is 1 ?

As a user(onert) of generated CIRCLE file, what would that mean?

Some model that converts input seq to some output sequence uses its output seq of [0] as its initial value. As the model converts an input word to output word, it concats the converted word to its output seq. :-)

[wateret]

• Are these semantics identical for tensorflow/tflite(circle)/NNAPI?

One thing that was confirmed by @hseok-oh - [0] is used to say it is a dynamic tensor from NN API (whilst tflite has shape of [1] with shape signature of [-1]). And also note that NN API cannot have negative values in its dims.