Coremltools: Coreml Pipeline with preprocessing example

[kezakool]

Hi,

i'm using coremltools beta release 3.0b5, trying to convert from keras to coreml3.
the conversion of my model went well but I would like to use the SoundAnalysisPreprocessing model. (I already have implemented the same preprocessing step in my python project)

I can't find an example of building a pipeline with a coreml preprocessing model, could you provide us any exmple?

for now, i'm stuck into this :

save_model_path = os.path.join(config.freeze_path, "keras_cnnlstm_greedy_perf_82.mlmodel")
mlmodel = coremltools.models.MLModel(save_model_path)

mlmodel = coremltools.models.MLModel('path/to/the/model.mlmodel')

input_audio = [('audio', coremltools.models.datatypes.Array((1,15600)))]
output_feature = [('output', coremltools.models.datatypes.Array((1,2)))]
pipeline = coremltools.models.pipeline.Pipeline(input_audio , output_feature)

sound_model = soundAnalysisPreprocessing.SoundAnalysisPreprocessing()

pipeline.add_model(sound_model)

giving me this error :
TypeError: Parameter to CopyFrom() must be instance of same class: expected CoreML.Specification.Model got CoreML.Specification.CoreMLModels.SoundAnalysisPreprocessing.

Many thanks

[anilkatti]

@kezakool you are on the right track however, I think you should add soundAnalysisPreprocessing rather than soundAnalysisPreprocessing.SoundAnalysisPreprocessing() as the sub-model in your pipeline. See if that works.

Here is some documentation regarding add_model. It. expects the spec (Model proto message). https://apple.github.io/coremltools/generated/coremltools.models.pipeline.html#coremltools.models.pipeline.PipelineRegressor.add_model

I don't have an example that uses sound analysis preprocessing inside a pipeline to share with you at the moment but, I can point you to another example for how pipelines are built: https://github.com/apple/coremltools/blob/master/examples/updatable_models/updatable_tiny_drawing_classifier.ipynb

Let me know if you are still blocked. Good luck!

[kezakool]

@anilkatti thanks for helping!
i tried your solution, get a new error because soundAnalysisPreprocessing is a module, so i'm still blocked :)

i'm not sure but, i feel that the real problem is that i can't get the protobuf spec from SoundAnalysisPreprocessing.

the notebook example is a bit different because you can "reach" the model KNearestNeighborsClassifierBuilder by the models module, and in my case there only is the proto module to reach SoundAnalysisPreprocessing, so i'm not sure it can help me...

Are there examples of .mlmodel generated from .proto file?

thank you for your time

[anilkatti]

i tried your solution, get a new error because soundAnalysisPreprocessing is a module

Could you paste your code for me a take a look? Specifically, how you get soundAnalysisPreprocessing?

and in my case there only is the proto module to reach SoundAnalysisPreprocessing, so i'm not sure it can help me...

You'd do something like this:

soundAnalysisPreprocessing_path = '/path/to/the/mlmodel'
soundAnalysisPreprocessing_model = MLModel(soundAnalysisPreprocessing_path)
soundAnalysisPreprocessing_spec = soundAnalysisPreprocessing_model.get_spec()
pipeline.add_model(soundAnalysisPreprocessing_spec)

[kezakool]

@anilkatti, here is how i get soundAnalysisPreprocessing
from coremltools.proto import SoundAnalysisPreprocessing_pb2 as soundAnalysisPreprocessing

i know it's the proto file but for now, i can't find a way to get the .mlmodel or to convert the proto to mlmodel (@aseemw do you know how can we explain the different behavior between a model like KNearestNeighborsClassifierBuilder and SoundAnalysisPreprocessing?)

thanks

[anilkatti]

(@aseemw do you know how can we explain the different behavior between a model like KNearestNeighborsClassifierBuilder and SoundAnalysisPreprocessing?)

Aseem could clarify, but in my opinion, KNearestNeighborsClassifierBuilder is a utility that helps you build a nearest neighbor classifier model spec (translates to Model_pb2 instance) whereas, SoundAnalysisPreprocessing_pb2 is just a python wrapper to read / access SoundAnalysisPreprocessing proto message. It is not a model builder.

add_model method needs either a spec (Model_pb2 instance) or a builder (MLModel instance) that is holding on to a spec internally (like KNearestNeighborsClassifierBuilder or NeuralNetworkBuilder).

 def add_model(self, spec):
        Add a protobuf spec or :py:class:`models.MLModel` instance to the pipeline. 
        All input features of this model must either match the input_features 
        of the pipeline, or match the outputs of a previous model. 
        Parameters
        ----------
        spec: [MLModel, Model_pb2]
            A protobuf spec or MLModel instance containing a model.

You need to build a model (using Model_pb2 module) around SoundAnalysisPreprocessing with proper input and ouput in model description & preprocessing parameters, if you want to add one as a sub-model for pipeline.

I've attached a sample model that I just built that you could load and add as a sub-model in your pipeline.

SoundAnalysisPreprocessing.mlmodel.zip

[kezakool]

@anilkatti big thanks it work with your mlmodel!
but just to understand, how did you build the model, with help of the book, i tried implement it with this :

`sap_spec = coremltools.proto.Model_pb2.Model()
sap_spec.specificationVersion = 4

new_input = sap_spec.description.input.add()
new_input.name = "audioSamples"
new_input.type.multiArrayType.shape.extend([15600])

new_output = sap_spec.description.output.add()
new_output.name = "preprocessedAudioSamples"
new_output.type.multiArrayType.shape.extend([1,96,64])

sap_model = coremltools.models.MLModel(sap_spec)`

but the model seems corrupted with an incorrect description when i try to verify it with netron...

[anilkatti]

I think you need to set type.multiArrayType.dataType inside new_input and new_output. I can share the actual script later today.

[kezakool]

@anilkatti reminder :) thanks.

[anilkatti]

Sorry about the delay.. here's the script that will generate the CoreML model I attached earlier.

import coremltools
from coremltools.proto import FeatureTypes_pb2
from coremltools.proto import SoundAnalysisPreprocessing_pb2

sap_spec = coremltools.proto.Model_pb2.Model()
sap_spec.specificationVersion = 4
sap_spec.description.predictedFeatureName = "preprocessedAudioSamples"
sap_spec.description.metadata.shortDescription = "Example SoundAnalysisPreprocessing Model"
sap_spec.soundAnalysisPreprocessing.vggish.CopyFrom(SoundAnalysisPreprocessing_pb2.SoundAnalysisPreprocessing.Vggish())

new_input = sap_spec.description.input.add()
new_input.name = "audioSamples"
new_input.type.multiArrayType.shape.extend([15600])
new_input.type.multiArrayType.dataType = FeatureTypes_pb2.ArrayFeatureType.FLOAT32

new_output = sap_spec.description.output.add()
new_output.name = "preprocessedAudioSamples"
new_output.type.multiArrayType.shape.extend([1,96,64])
new_output.type.multiArrayType.dataType = FeatureTypes_pb2.ArrayFeatureType.FLOAT32

sap_model = coremltools.models.MLModel(sap_spec)
sap_model.save('SoundAnalysisPreprocessing.mlmodel')

Let me know if you have any questions.

[Mstronach]

We believe this issue has been resolved, feel free to re-open if you are still seeing the issue. Thanks!