Coremltools: Create MLModel from Spec gives segmentation fault

[ardamavi]

I have a CoreML Model (Neural Network Classifier) that works well. I want to create updatable model from that one. For this, I wrote the Python script below. But I get zsh: segmentation fault python script.py error when I run script.

import coremltools
from coremltools.models.neural_network.update_optimizer_utils import AdamParams

# File paths:
old_coreml_file_path = 'AModel.mlmodel'
updatable_coreml_file_path = 'updatableAModel.mlmodel'

# Get old CoreML Model:
old_model = coremltools.models.MLModel(old_coreml_file_path)
spec = old_model.get_spec()
builder = coremltools.models.neural_network.NeuralNetworkBuilder(spec=spec)

# CoreML Model to Updatable CoreML Model:
builder.make_updatable(['dense_1', 'dense_2'])

# Set update params:
builder.set_categorical_cross_entropy_loss(name='lossLayer', input='class')

builder.set_adam_optimizer(AdamParams(lr=0.01, batch=8))

builder.set_epochs(allowed_set=[1,2,4,8,16])

# Save Updatable Model:
mlmodel_updatable = coremltools.models.MLModel(spec)
mlmodel_updatable.save(updatable_coreml_file_path)

The error comes in the mlmodel_updatable = coremltools.models.MLModel(spec) line.
If I give incorrect input and get a warning in builder.set_epochs section like builder.set_epochs(8, [1,2,4,10,16]), the model is saved without any error, but in my Xcode project, it gives the error about determining the epoch number for the ML Model.

I tried another way to save:

When I try to save with
coremltools.utils.save_spec(spec, updatable_coreml_file_path)
instead of

mlmodel_updatable = coremltools.models.MLModel(spec)
mlmodel_updatable.save(updatable_coreml_file_path)

at the end of the script, the script runs smoothly, when I drag the saved model to Xcode, MLModel looks good in Xcode and the class is created automatically, but when I build the project, I get the Build Error: Command CoreMLModelCompile failed with a nonzero exit code error.
I tried to clean the build folder and rebuild it, but it did't work.

System environment:

• coremltools version 3.2 (Version 3.0b6 gives same problems.)
• macOS 10.15.3
• XCode 11.3.1
• Python 3.6.8 (anaconda)

[mrfarhadi]

@ardamavi, do you see any error when you open your model with XCode? something that starts withvalidator error: ?

One thing I can see from your code is that builder.set_epochs(8, [1,2,4,10,16]) is incorrect, as 8 must also be included in allowed_set. You need to change it to builder.set_epochs(8, [1,2,4,8,10,16]). However, the builder.set_epochs(allowed_set=[1,2,4,8,16]) looks OK.

[ardamavi]

@mrfarhadi No, There is no validator error in the Xcode. Even, the model class is created correctly with coremltools.utils.save_spec(spec, updatable_coreml_file_path), but I get Build Error: Command CoreMLModelCompile failed with a nonzero exit code in build.

I don't use builder.set_epochs(8, [1,2,4,10,16]), I just give extra information. I mean I don't get segmentation fault error with this incorrect line while using mlmodel_updatable = coremltools.models.MLModel(spec) line but I get build error in Xcode.
Maybe it was unnecessary information.

[mrfarhadi]

@ardamavi I see. Can you send me your app with the model so that I can test it on my end?

[ardamavi]

@mrfarhadi Unfortunately, I can't send it because it is a private work.

[mrfarhadi]

That is OK. Would you share if XCode shows anything else in the build logs that might shed some light to the issue?

[ardamavi]

@mrfarhadi I get just one error message and no further information.

[ardamavi]

@mrfarhadi Full message of build error:

CoreMLModelCompile /Users/my_user/Library/Developer/Xcode/DerivedData/AModel-bchtxlhinhhzkjffzfmwgrdpnorj/Build/Products/Debug-iphoneos/Project.app/ /Users/path_to_project/AModel.mlmodel (in target 'Project' from project 'Project')
    cd /Users/path_to_project/Project
    /Applications/Xcode.app/Contents/Developer/usr/bin/coremlc compile /Users/path_to_project/updatableAModel.mlmodel /Users/my_user/Library/Developer/Xcode/DerivedData/Project-bchtxlhinhhzkjffzfmwgrdpnorj/Build/Products/Debug-iphoneos/Project.app/ --output-partial-info-plist /Users/my_user/Library/Developer/Xcode/DerivedData/Project-bchtxlhinhhzkjffzfmwgrdpnorj/Build/Intermediates.noindex/Project.build/Debug-iphoneos/Project.build/updatableAModel-CoreMLPartialInfo.plist

[ardamavi]

I try to run my python script with my CoreML model without apply Quantize. Script and Xcode build worked without any problems.
And I noticed that, when I put the model I got the build error into my Xcode files, a question mark appears next to it. When I wait above the question mark with the mouse, Unversioned message appeared. But I couldn't understand what that mean.
I can't make Quantized CoreML model updatable.

[mrfarhadi]

Quantized models are not supported for update unfortunately. You might want to de-quantize your model before making it updatable.

[aseemw]

Only the layers marked as updatable need to have un-quantized weights.

You can have a model with, say 10 conv/dense layers (layers with weights), such that the first 9 layers are quantized and not marked as updatable and the last layer is marked as updatable and unquantized. This is totally fine and should work.

[ardamavi]

Thank you for your interest. @mrfarhadi @aseemw

[ardamavi]

@aseemw Is there any function to de-quantize the selected layers that I will make updatable?

[mario11c]

A similar phenomenon has occurred for me. Is this problem now solved?
If it has already been resolved, I want to know how it was resolved.

[mrfarhadi]

@mario11c if the layer is quantized, it cannot be updatable. How is your model structured?

[mario11c]

The model I tried is a half-precision model.
The single-precision model is converted to a half-precision model, and the half-precision model is converted to an updatable model.
When building with Xcode, the following error message is displayed.

Command CoreMLModelCompile failed with a nonzero exit code
Command CompileSwift failed with a nonzero exit code

I tried with a quantized model, but the result was the same as you.

For single-precision models, the build with Xcode has been successful.
However, when trying to update the model, the app crashes due to memory issues.

For this reason, I thought that the model should be lightweight, and tried using a half-precision model and a quantization model, and this problem occurred.

[mrfarhadi]

I tried with a quantized model, but the result was the same as you.

For single-precision models, the build with Xcode has been successful.
However, when trying to update the model, the app crashes due to memory issues.

For this reason, I thought that the model should be lightweight, and tried using a half-precision model and a quantization model, and this problem occurred.

@mario11c when you make the model half precision, all the layers will be quantized and quantized layers are not supported to be marked 'updatable'.
The single precision model is supported to be updatable. What is the memory issue you are getting on that?

[mario11c]

when you make the model half precision, all the layers will be quantized and quantized layers are not supported to be marked 'updatable'.

The half-precision model and the quantization model(low-precision model) are different.

Conversion to a half-precision model is performed by coremltools.utils.convert_neural_network_spec_weights_to_fp16.

Conversion to a quantization model (low-precision model) is performed by quantize_weights.

Reference URL
https://developer.apple.com/documentation/coreml/reducing_the_size_of_your_core_ml_app
https://apple.github.io/coremltools/generated/coremltools.models.neural_network.quantization_utils.html
　
　
　

The single precision model is supported to be updatable. What is the memory issue you are getting on that?

The single-precision model I describe is, for example, a model generated from a SFrame file created by Turi Create into a machine learning model for CoreML using coremltools.

This single-precision model itself cannot be updated with CoreML in iOS applications.
It must be changed to be updatable by make_updatable.

Reference URL
https://machinethink.net/blog/coreml-training-part1/

Is my perception wrong?
　
　
　
The memory problem that occurs when updating a single-precision model that has been made updatable is similar to the phenomenon shown at the following URL.

Reference URL
https://github.com/apple/coremltools/issues/428#issuecomment-524885795

When the MLUpdateTask is executed, the following message is displayed in the console log and the application crashes.
A large amount of memory is used and not released, and it seems to be running out of memory and crashing.

Device:
iPhone 11 Pro
iOS13.4

[mrfarhadi]

The half-precision model and the quantization model(low-precision model) are different.

Conversion to a half-precision model is performed by coremltools.utils.convert_neural_network_spec_weights_to_fp16.

Conversion to a quantization model (low-precision model) is performed by quantize_weights.

Reference URL
https://developer.apple.com/documentation/coreml/reducing_the_size_of_your_core_ml_app
https://apple.github.io/coremltools/generated/coremltools.models.neural_network.quantization_utils.html

Half precision model is actually a quantize model with float 16 weights.

The single-precision model I describe is, for example, a model generated from a SFrame file created by Turi Create into a machine learning model for CoreML using coremltools.

This single-precision model itself cannot be updated with CoreML in iOS applications.
It must be changed to be updatable by make_updatable.

Reference URL
https://machinethink.net/blog/coreml-training-part1/

Is my perception wrong?

You are right, you need to call make_updatable method to mark some layers updatable. However if you call this method on a quantized model ( or half-precision model), it mark those quantized layers updatable which is not a valid case. That is why you are seeing the crash.

The memory problem that occurs when updating a single-precision model that has been made updatable is similar to the phenomenon shown at the following URL.

Reference URL
https://github.com/apple/coremltools/issues/428#issuecomment-524885795

When the MLUpdateTask is executed, the following message is displayed in the console log and the application crashes.
A large amount of memory is used and not released, and it seems to be running out of memory and crashing.

This is a known issue unfortunately. This issue will be gone on newer versions of iOS.