Coremltools: PyTorch to CoreML _MLModelProxy Error compiling model: "The file couldn’t be saved.". See this? Any solution?

[leovinus2001]

I have this funny issue that sometimes when calling convert() to go from PyTorch to CoreML model, I get a warning

Library/Python/3.7/lib/python/site-packages/coremltools/models/model.py:119: RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "The file couldn’t be saved.".

convert() DOES return a model but when you use it, but then I see again something like this on predict()

mlmodel.predict( { inputName[0]: dummy_input.detach().numpy() }, useCPUOnly=True)
File "Library/Python/3.7/lib/python/site-packages/coremltools/models/model.py", line 367, in predict
raise self._framework_error
File "Library/Python/3.7/lib/python/site-packages/coremltools/models/model.py", line 113, in _get_proxy_and_spec
return (_MLModelProxy(filename, use_cpu_only), specification, None)
RuntimeError: Error compiling model: "The file couldn’t be saved.".

It is about saving a temporary model file. First, the issue was intermittent, now it is persistent.
As I cannot look into _MLModelProxy I cannot see what the issue is. The filename has been checked and the directory is writable.

Question - Any thoughts on why this happens? And how to solve it? Or how to debug the _MLModelProxy further?

I tried a lot already

• The Mac with Catalina latest has several other accounts, one admin, two user. Two other accounts (1 admin, 1 user) do NOT have this warning and issue which means it is some kind of setup "per account" thingy. But what?
• Same version coremltools4.0b3 and torch1.6 (but also happens with 1.5.1 and 4.0b2)
• All pip installed software up to date
• Verified via DYLD_PRINT_LIBRARIES that the same libraries are loaded in working and not-working account. All good.
• Verified PATH, all good.
• Verified same python3.7 everywhere, yep.
• Now I am running out of options. If feels like I am missing something obvious.
  Does anyone else see this? Any solutions?

[aseemw]

Not sure, need to investigate. I have seen this issue as well on Mac OS < macOS Big Sur seed, intermittently

[leovinus2001]

No, I am on Catalina. 10.15.6 ? I believe. Help is appreciated. At the moment, trying a reboot as a last ditch attempt. Anyway, good to know that you have seen it as well. Let me know how I can help.

[srikris]

@leovinus2001 I am able to reproduce the error to. This is fixed in MacOS 11. It's an old bug that surfaces up because of how we were handling temporary directories (we weren't cleaning up correctly). We are trying to find a work around for this for MacOS 10.15 and below. It will show up if you have loaded ~1000 models. The "fix" is to reboot or log-off/log-on and resume but's really painful when developing as it can make it hard to run all unit tests in the same python session without it hitting this issue.

The issue is the in-ability of the file system to open a temporary file when we are running nosetests or pretty much any thing repeatedly without restarts. The trace comes down to use of this function

+ (NSURL *)compileModelAtURL:(NSURL *)modelURL
                       error:(NSError * __autoreleasing *)error {

The above block seems to have some sort of limitation of # of temporary directories we can open and we seem to be hitting it. It has been fixed in MacOS 11 but sadly this still exists in 10.15 so we have to figure out a workaround so we don't create these directories or clean up.

[leovinus2001]

@srikris @aseemw Thanks for info. Just to confirm, last night's reboot did fix my issue magically confirming that some kind of 'state' is involved besides the temporary filename. Based on @srikris comment, a wild guess would be a corrupted plist. Hope you find a workaround!

[dileep223]

Hi @aseemw, @srikris , @leovinus2001

While converting the ONNX model to the Core Ml model I'm facing the same issue, as per the above comments https://github.com/apple/coremltools/issues/882#issuecomment-682599968 I tried with rebooting the MAC laptop but still, I can see the same issue.
And upgrade the Mac OS with the latest version Catalina 10.15.7 still, the issue existed.

System Info:
Mac OS - Catalina 10.15.7
Xcode - 11.7
Python - 3.8
coremltools - 4.0b1

For any workaround on the above issue, please let me know if any suggestion and solution for the above issue?

Thanks,

[dileep223]

hi @srikris

@leovinus2001 I am able to reproduce the error to. This is fixed in MacOS 11. It's an old bug that surfaces up because of how we were handling temporary directories (we weren't cleaning up correctly). We are trying to find a work around for this for MacOS 10.15 and below. It will show up if you have loaded ~1000 models. The "fix" is to reboot or log-off/log-on and resume but's really painful when developing as it can make it hard to run all unit tests in the same python session without it hitting this issue.

The issue is the in-ability of the file system to open a temporary file when we are running nosetests or pretty much any thing repeatedly without restarts. The trace comes down to use of this function

+ (NSURL *)compileModelAtURL:(NSURL *)modelURL
                       error:(NSError * __autoreleasing *)error {

The above block seems to have some sort of limitation of # of temporary directories we can open and we seem to be hitting it. It has been fixed in MacOS 11 but sadly this still exists in 10.15 so we have to figure out a workaround so we don't create these directories or clean up.

as per you comment , i'm upgraded to MAC OS 11 (Big-Sur public beta version). while converting the coreMl model i got the same runtime warning, it was showing not able run predict() on this core ml model ,but this time compiler error was different encountered with unexpected error while compiling a neural network model. please find the below logs

/Users/opt/anaconda3/envs/coreml/lib/python3.8/site-packages/coremltools/models/model.py:115: RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error: Encountered an unexpected error while compiling a neural network model.".
_warnings.warn(

@aseemw , @srikris
What might be the reason for causing of the above issue ? Any thoughts on why this happens? And how to solve it?
System info : mac OS 11(Big-Sur, public beta version), Xcode -11.7, coremltools - 4.0b1, onnx -1.7.0

Thanks