Coremltools: Strange behaviour when attempting to evaluate batches of sequences

[vellamike]

I have the following model defined in keras which I convert to CoreML:

SEQUENCE_LENGTH = 8000
NUM_CATS = 1025

inputs = Input(shape=(SEQUENCE_LENGTH,))
L0 = Reshape((SEQUENCE_LENGTH,1))(inputs)
L1 = Conv1D(filters=4,
             kernel_size=4,
             strides=4, 
             padding='same', 
             activation='relu')(L0)
L2  = Dense(32, activation='relu')(L1)
L3 = Dense(NUM_CATS, activation='softmax')(L2)

model = Model(inputs=inputs, outputs=L3)

If I run a prediction with one sequence it appears to work correctly:

inputx = np.array(X[0])
print(inputx.shape)
t0 = time.time()
print("CoreML model output shape: ", coreml_model.predict({'input1': inputx})['output1'].shape)
print("Time taken: ", time.time()-t0)

(8000,)
CoreML model output shape:  (2000, 1025)
Time taken:  0.049160003662109375

However if I run it on multiple sequences the output shape is incorrect:

inputx = np.array([X[0:20]])
print(inputx.shape)
t0 = time.time()
print("CoreML model output shape: ", coreml_model.predict({'input1': inputx})['output1'].shape)
print("Time taken: ", time.time()-t0)

(1, 20, 8000)
CoreML model output shape:  (2000, 1025)
Time taken:  0.5843930244445801

The output shape should be (20, 2000, 1025) not (2000, 1025). The time taken to process has increased by 20x so something is happening. As things stand without the ability to process in batches evaluating this particular model on CPU is faster than GPU.

[aseemw]

This is related to #70

[vellamike]

@aseemw I'm not entirely sure this is a duplicate of #70 ? In #70 the issue is that the correct input shape is not being inferred from the Keras model, whereas here the issue appears to be that the output shape is incorrect or that batching is not being correctly implemented.

[aseemw]

Input to the CoreML for the predict call have to be either 1D, 3D or 5D, depending on the input type in the mlmodel file.
Each has a specific meaning in terms of how the shapes are interpreted:

• in mlmodel if input type is 1D: (C), then predict call expects input to be 1D or 3D (Seq, B, C)
• in mlodel if input type is 3D: (C,H,W), then predict call expects input to be 3D or 5D (Seq, Batch, C, H, W)

However, currently in CoreML both B and Seq. cannot be more than 1. That is, the modes supported are either a sequence of batch size 1 or a batch of sequence length 1.

Seq and Batch are special dimensions, in that, they are not hardcoded to the mlmodel file. That enables to provide variable shapes for seq and batch at runtime.

In the case of your Keras model, because the input shape is 1D, the converter maps that to the channel dimension for CoreML input. That is why you see the size of the input as (8000) in .mlmodel Xcode description. Now to pass the batch you should really be reshaping the input to be (1,B,8000).

Reshape operations can be slow on the GPU because of the way values are packed in a metal texture. I would suggest you modify your Keras model in the following manner to get rid of reshapes. Essentially, you can use a 2D convolution with a unit height dimension and sequence mapped on the width dimension. (In fact, that is how CoreML evaluates a 1D Conv, i.e. it maps it to a 2D conv where the seq. dimension is transferred to the width dimension).

SEQUENCE_LENGTH = 8000
NUM_CATS = 1025

inputs = Input(shape=(1,SEQUENCE_LENGTH,1)) #(H,W,C)
L0 = Conv2D(filters=4,
             kernel_size=(1,4),
             strides=(4,4), 
             padding='same', 
             activation='relu')(inputs)
L1 = Flatten()(L0)             
L2  = Dense(32, activation='relu')(L1)
L3 = Dense(NUM_CATS, activation='softmax')(L2)

model = Model(inputs=inputs, outputs=L3)
print(model.summary())

[vellamike]

Thanks @aseemw for the very detailed answer. There is however something I still don't understand which seems like a bug.

When I run a prediction with an input array of dimensions (1, 20, 8000) the output is an array of dimensions (2000, 1025). If a 3D array is interpreted as CHW and the input shape to my model should be a 1D array:

input {
  name: "input1"
  type {
    multiArrayType {
      shape: 8000
      dataType: DOUBLE
    }
  }
}
output {
  name: "output1"
  type {
    multiArrayType {
      shape: 1025
      dataType: DOUBLE
    }
  }
}

Then why doesn't the predict method raise an exception when I use a 3D array as input? Indeed what is even going on with that (2000, 1025) output? What part of the input is being used to produce those numbers?

In fact if I reshape my input (e.g to (1,40,4000) I get the following exception when running a prediction:

RuntimeError: {
    NSLocalizedDescription = "Input feature input1 was presented as a batch  (length 40) of sequences (length 1) of vectors of length 4000, but the model expects vectors of length 8000.";
}

Which is particularly confusing if as you say a 3D input is needed for a batch.

Unless of course the 3D input to a model which accepts a 1D input is being interpreted as (Seq, Batch, C) ? But then it still doesn't explain why the output dimension is (2000, 1025)?

[vellamike]

I don't think the suggestion to remap the problem to a 2D convolution works. The summary for my model is as follows:

SEQUENCE_LENGTH = 8000
NUM_CATS = 1025

inputs = Input(shape=(SEQUENCE_LENGTH,))
L0 = Reshape((SEQUENCE_LENGTH,1))(inputs)
L1 = Conv1D(filters=4,
             kernel_size=4,
             strides=4, 
             padding='same', 
             activation='relu')(L0)
L2  = Dense(32, activation='relu')(L1)
L3 = Dense(NUM_CATS, activation='softmax')(L2)

model = Model(inputs=inputs, outputs=L3)

print(model.summary())

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_17 (InputLayer)        (None, 8000)              0         
_________________________________________________________________
reshape_7 (Reshape)          (None, 8000, 1)           0         
_________________________________________________________________
conv1d_10 (Conv1D)           (None, 2000, 4)           20        
_________________________________________________________________
dense_28 (Dense)             (None, 2000, 32)          160       
_________________________________________________________________
dense_29 (Dense)             (None, 2000, 1025)        33825     
=================================================================

However the model you propose has the following input/output summary:

SEQUENCE_LENGTH = 8000
NUM_CATS = 1025

inputs = Input(shape=(1,SEQUENCE_LENGTH,1)) #(H,W,C)
L0 = Conv2D(filters=4,
            kernel_size=(1,4),
            strides=(4,4), 
            padding='same', 
            activation='relu')(inputs)
L1 = Flatten()(L0)             
L2  = Dense(32, activation='relu')(L1)
L3 = Dense(NUM_CATS, activation='softmax')(L2)
print(model.summary())
model = Model(inputs=inputs, outputs=L3)

Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         (None, 1, 8000, 1)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 1, 2000, 4)        20        
_________________________________________________________________
flatten_2 (Flatten)          (None, 8000)              0         
_________________________________________________________________
dense_3 (Dense)              (None, 32)                256032    
_________________________________________________________________
dense_4 (Dense)              (None, 1025)              33825     
=================================================================

I haven't yet understood exactly how the topologies of the two models are different but they are clearly very different from one another.

If I remove the flattening layer the 2D convolution appears to work as expected:

SEQUENCE_LENGTH = 8000
NUM_CATS = 1025

inputs = Input(shape=(1,SEQUENCE_LENGTH,1)) #(H,W,C)
L0 = Conv2D(filters=4,
            kernel_size=(1,4),
            strides=(4,4), 
            padding='same', 
            activation='relu')(inputs2)
L2  = Dense(32, activation='relu')(L0)
L3 = Dense(NUM_CATS, activation='softmax')(L2)
model2D = Model(inputs=inputs2, outputs=L3)
print(model2.summary())

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_4 (InputLayer)         (None, 1, 8000, 1)        0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 1, 2000, 4)        20        
_________________________________________________________________
dense_7 (Dense)              (None, 1, 2000, 32)       160       
_________________________________________________________________
dense_8 (Dense)              (None, 1, 2000, 1025)     33825     
=================================================================
Total params: 34,005
Trainable params: 34,005
Non-trainable params: 0

````
There's an extra dimension in the output but that's not an issue. However if I try and convert this from Keras to coreML I get the following error:

----> 1 coreml_model = coremltools.converters.keras.convert(MODEL_2D_NAME)

~/dev/coreml_experiments/.venv/lib/python3.6/site-packages/coremltools/converters/keras/_keras_converter.py in convert(model, input_names, output_names, image_input_names, is_bgr, red_bias, green_bias, blue_bias, gray_bias, image_scale, class_labels, predicted_feature_name, model_precision, predicted_probabilities_output, add_custom_layers, custom_conversion_functions)
745 custom_conversion_functions=custom_conversion_functions)
746
--> 747 return _MLModel(spec)
748
749

~/dev/coreml_experiments/.venv/lib/python3.6/site-packages/coremltools/models/model.py in __init__(self, model)
151 filename = _tempfile.mktemp(suffix = '.mlmodel')
152 _save_spec(model, filename)
--> 153 self.__proxy__ = _get_proxy_from_spec(filename)
154 else:
155 raise TypeError("Expected model to be a .mlmodel file or a Model_pb2 object")

~/dev/coreml_experiments/.venv/lib/python3.6/site-packages/coremltools/models/model.py in _get_proxy_from_spec(filename)
75 return None
76
---> 77 return _MLModelProxy(filename)
78 else:
79 return None

RuntimeError: Error compiling model: "compiler error: Inner product layer: 'dense_11' : Product of input blob dimensions C,H,W (4,1,2000) must be equal to the parameter 'inputChannels' (4)
".
```

Which is surprising because this is quite an ordinary 2D convolutional model.

[aseemw]

I erred in my description of the input shape required at predict time. I have corrected my comment above, please see.

Yes I agree, the model I had suggested is different from your original model because it flattens the sequence dimension along with the channel dimension. I guess what you really need is to feed in a batch of sequences of arrays. I slightly misunderstood your model and the use case. My bad again.
Actually currently CoreML does not support feeding in a batch of sequences.
So, you should just feed in the batches inside a for loop.

Actually you do not need the reshape layer in the beginning. Instead of this:
inputs = Input(shape=(SEQUENCE_LENGTH,)) L0 = Reshape((SEQUENCE_LENGTH,1))(inputs)

use:
inputs = Input(shape=(SEQUENCE_LENGTH,1))

This will result in an mlmodel without the reshape layer in the beginning. The resulting mlmodel will have input as a multi array of size 1. During predict call, you can pass in (Seq,1,1) as the input.

Btw, the error you get above is because the inner product layer in CoreML, similar to (but not exactly the same) Dense layer in Keras requires flat inputs i.e. both W,H dimensions should be 1. However, if we use the width dimension for 8000, this condition is no longer met and we get the compiler error. Ideally, the converter should catch this error and raise an exception during conversion. This is a converter bug and should be fixed.

[vellamike]

Thanks @aseemw, I'm starting to get the idea of how CoreML works now. I may take a look at that converter bug myself.

When you say that both Batch and Seq cannot currently be more than 1, do you mean only for 1D data or for 2D data also? If the Batch size cannot be >1 am I right in thinking this can have quite a bad impact on inference performance?

[aseemw]

That is true for images as well.
Batch size can be more than 1 when seq is 1.

Inference performance depends on the type of model and size of the batch/sequences. A very high batch size need not be efficient due to memory constraints. Generally there is a sweet spot after which the performance would go down or would be same as looping.
So, if you already have a long sequence having a batch size may or may not be efficient depending on the model.

[vellamike]

@aseemw I tried your suggestion of removing the reshape layer and replacing the input layer with shape 8000,1 and feeding in a layer of dimensions [SEQUENCE_LENGTH,1,1] but encountered the following issue at inference time:

Thread 1: Fatal error: 'try!' expression unexpectedly raised an error: Error Domain=com.apple.CoreML Code=0 "Cannot evaluate a sequence of length 8000, which is longer than maximum of 400." UserInfo={NSLocalizedDescription=Cannot evaluate a sequence of length 8000, which is longer than maximum of 400.}

Is this a known limitation?

This isn't the only issue either, at inference a sequence of length 200 works but if the length is 300 (so under the 400 limit) the following exception is raised:

Thread 1: signal SIGABRT

2018-03-05 14:11:23.679832+0000 BaseCaller[3382:525445] [MC] Lazy loading NSBundle MobileCoreServices.framework
2018-03-05 14:11:23.680282+0000 BaseCaller[3382:525445] [MC] Loaded MobileCoreServices.framework
2018-03-05 14:11:25.027432+0000 BaseCaller[3382:525445] validateTextureDimensions, line 768: error 'MTLTextureDescriptor has arrayLength (2400) greater than the maximum allowed size of 2048.'
validateTextureDimensions:768: failed assertion `MTLTextureDescriptor has arrayLength (2400) greater than the maximum allowed size of 2048.'
(lldb) 

[aseemw]

Yes the sequence upper bound is a known limitation and will be addressed in future.

Indeed, a dense operation with size of batch/sequence in the hundreds is very big and unable to be performed on a GPU because of the memory constrains of a metal texture. You should use the forceCPU Flag to run it in the CPU or use loops to break the sequence length.

[vellamike]

Thanks @aseemw for all your help. I am now getting good performance for my model on ios.

[FrozenGene]

@aseemw I google the sequence of CoreML and come here. I am also meet the problem of Reshape / Permute. Could you explain the concept of Sequence in CoreML? Others framework we almost care about N H W C. But CoreML has one Sequence concept, which seems very import in Reshape / Permute. If you can compare it with the Batch, it is nicer.