Coremltools: Unsupported option input_shape

[eaigner]

I'm trying to convert my YOLO keras model to CoreML. Those are the last 2 layers

model.add(Activation('linear'))
model.add(Reshape((GRID_H, GRID_W, BOX, 4 + 1 + CLASS)))

Now when I want to convert this model with coremltools it complains

...
31 : conv2d_8, <keras.layers.convolutional.Conv2D object at 0x1149f8a50>
32 : activation_1, <keras.layers.core.Activation object at 0x114b87b90>
33 : reshape_1, <keras.layers.core.Reshape object at 0x114be7f10>
Traceback (most recent call last):
  File "./tinyyolo.py", line 160, in <module>
    coreml_model = coremltools.converters.keras.convert(model, image_input_names='yolo_in', output_names='reshape_1')
  File "/Users/era/git/coremltools/coremltools/converters/keras/_keras_converter.py", line 505, in convert
    predicted_probabilities_output = predicted_probabilities_output)
  File "/Users/era/git/coremltools/coremltools/converters/keras/_keras2_converter.py", line 278, in _convert
    converter_func(builder, layer, input_names, output_names, keras_layer)
  File "/Users/era/git/coremltools/coremltools/converters/keras/_layers2.py", line 858, in convert_reshape
    _utils.raise_error_unsupported_categorical_option('input_shape', str(input_shape), 'reshape', layer)
  File "/Users/era/git/coremltools/coremltools/converters/keras/_utils.py", line 12, in raise_error_unsupported_categorical_option
    layer_type, layer_name))
RuntimeError: Unsupported option input_shape=(None, 13, 13, 160) in layer reshape(reshape_1)

Installed versions:

Keras (2.1.1)
coremltools (0.6.3)

Complete layer stack

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
yolo_in (Conv2D)             (None, 416, 416, 16)      432       
_________________________________________________________________
batch_normalization_1 (Batch (None, 416, 416, 16)      64        
_________________________________________________________________
leaky_re_lu_1 (LeakyReLU)    (None, 416, 416, 16)      0         
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 208, 208, 16)      0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 208, 208, 32)      4608      
_________________________________________________________________
batch_normalization_2 (Batch (None, 208, 208, 32)      128       
_________________________________________________________________
leaky_re_lu_2 (LeakyReLU)    (None, 208, 208, 32)      0         
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 104, 104, 32)      0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 104, 104, 64)      18432     
_________________________________________________________________
batch_normalization_3 (Batch (None, 104, 104, 64)      256       
_________________________________________________________________
leaky_re_lu_3 (LeakyReLU)    (None, 104, 104, 64)      0         
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 52, 52, 64)        0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 52, 52, 128)       73728     
_________________________________________________________________
batch_normalization_4 (Batch (None, 52, 52, 128)       512       
_________________________________________________________________
leaky_re_lu_4 (LeakyReLU)    (None, 52, 52, 128)       0         
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 26, 26, 128)       0         
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 26, 26, 256)       294912    
_________________________________________________________________
batch_normalization_5 (Batch (None, 26, 26, 256)       1024      
_________________________________________________________________
leaky_re_lu_5 (LeakyReLU)    (None, 26, 26, 256)       0         
_________________________________________________________________
max_pooling2d_5 (MaxPooling2 (None, 13, 13, 256)       0         
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 13, 13, 512)       1179648   
_________________________________________________________________
batch_normalization_6 (Batch (None, 13, 13, 512)       2048      
_________________________________________________________________
leaky_re_lu_6 (LeakyReLU)    (None, 13, 13, 512)       0         
_________________________________________________________________
max_pooling2d_6 (MaxPooling2 (None, 13, 13, 512)       0         
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 13, 13, 1024)      4718592   
_________________________________________________________________
batch_normalization_7 (Batch (None, 13, 13, 1024)      4096      
_________________________________________________________________
leaky_re_lu_7 (LeakyReLU)    (None, 13, 13, 1024)      0         
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 13, 13, 1024)      9437184   
_________________________________________________________________
batch_normalization_8 (Batch (None, 13, 13, 1024)      4096      
_________________________________________________________________
leaky_re_lu_8 (LeakyReLU)    (None, 13, 13, 1024)      0         
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 13, 13, 160)       164000    
_________________________________________________________________
activation_1 (Activation)    (None, 13, 13, 160)       0         
_________________________________________________________________
reshape_1 (Reshape)          (None, 13, 13, 5, 32)     0         
=================================================================
Total params: 15,903,760
Trainable params: 15,897,648
Non-trainable params: 6,112

[G-mel]

https://github.com/apple/coremltools/blob/master/coremltools/converters/keras/_layers2.py#L834

Seems like 4 dimensional reshape is not currently supported.

[eaigner]

Is there a way to „emulate“ this behavior in CoreML? Perhaps a 2 step approach??

[G-mel]

I would suggest:
1) Taking the last reshape layer out of your keras model (e.g. remove model.add(Reshape((GRID_H, GRID_W, BOX, 4 + 1 + CLASS))))
2) Converting your Keras model to a CoreML model
3) Using something like MultiArray in https://github.com/hollance/CoreMLHelpers, you can convert the MLMultiArray output from your CoreML model to a MultiArray
4) You can then reshape the MultiArray to your desired 4D shape (13, 13, 5, 32) using MultiArray.reshaped([13, 13, 5, 32]) and then use your this output as desired in your application.

[mercy0387]

@G-mel 's suggestion looks good when the last layer is Reshape .

However, I met the same problem and I want to put Reshape (to 4 dimension) layer between other layers.
Is there a way to work around as @eaigner says? or do I have no choice but giving up high-dimension layer?

[G-mel]

@mercy0387 Unfortunately, it seems like higher dimensional reshapes aren't allowed currently in the Neural Network Builder (builder.add_reshape), which is used in the keras converter for reshape layers to create a coreML model.
https://github.com/apple/coremltools/blob/master/coremltools/models/neural_network.py#L2092

You may have to experiment with the reduced dimensionality in your model until a higher dimension reshape is supported.

[mercy0387]

I got it. I'll consider the model without this problem :(
Thank you for your explanation in detail.

[mercy0387]

@G-mel As far as I see, this function accept 4 dimension target.
I want to add layer that is time series image, shaped (Seq, H, W, Channel), and this is written in here.
https://github.com/apple/coremltools/blob/master/coremltools/models/neural_network.py#L2057

I'm sorry if I don't understand correctly, why does not keras converter accept 4 dimension?

[G-mel]

@mercy0387 I experimented with creating a coreML model with adding a 4 dimensional reshape (for example: builder.add_reshape(..., target_shape=(1, 512,6,8))) and it seemed to consider the first dimension as the batch or sequence dimension in my model, so it actually treated it like a 3-Dimensional reshape, not 4. I'm assuming your keras model has a batch dimension in it as well?

what @eaigner was trying to reshape to was actually 5 dimensional if you consider his batch shape none

[mercy0387]

@G-mel I understand. Thanks!