Hub: Could not load local module, with RuntimeError: Missing implementation that supports: loader(*('/tmp/tfhub_modules/mobilenet_module',), **{})

[htjacky]

Hi,
I tried this tutorial with a little bit change: https://www.tensorflow.org/hub/tutorials/image_retraining
I download the mobilenet module(mobilenet_v2_1.4_224.tgz) and decompressed under the folder /tmp/tfhub_modules/mobilenet_module.
Run the retrain.py with "python retrain.py --image_dir flower_photos --tfhub_module /tmp/tfhub_modules/mobilenet_module", but got following error:
Traceback (most recent call last):
File "retrain.py", line 1318, in
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "retrain.py", line 1001, in main
module_spec = hub.Module(FLAGS.tfhub_module)
File "/usr/local/lib/python3.5/dist-packages/tensorflow_hub/module.py", line 144, in __init__
self._spec = as_module_spec(spec)
File "/usr/local/lib/python3.5/dist-packages/tensorflow_hub/module.py", line 33, in as_module_spec
return load_module_spec(spec)
File "/usr/local/lib/python3.5/dist-packages/tensorflow_hub/module.py", line 58, in load_module_spec
return registry.loader(path)
File "/usr/local/lib/python3.5/dist-packages/tensorflow_hub/registry.py", line 45, in __call__
self._name, args, kwargs))
RuntimeError: Missing implementation that supports: loader('/tmp/tfhub_modules/mobilenet_module',), *{})
How can I fix it? Thanks!
BR,
Jacky

[samhavens]

I'm getting this problem locally (macOS 10.14.1) when trying to load Universal Sentence Encoder with Tensorflow version 1.12.0, Hub version 0.2.0. However, on Kaggle, it's using Tensorflow version 1.11.0-rc1 (also version 0.2.0 of Hub) and loading USE works.

I can't check if using those versions fixes things, because Tensorflow version 1.11.0-rc1 isn't a valid version for macOS. One potential problem is that I am using Pipenv locally. @htjacky, is this happening in a virtualenv?

[samhavens]

I just tried it in a Docker container and it worked, so it might be a macOS specific issue?

[GlassyWing]

You should change all directory access permissions under tfhub cache directory to rwxr-xr-x.

[alberduris]

I walked through the same error and this is how I solved it;

My error was:

RuntimeError: Missing implementation that supports: loader(*('C:\\Users\\Alber\\AppData\\Local\\Temp\\tfhub_modules\\a7fe827a4e68369aab0fa6a65479cd37c499e0f4',), **{})

So the problem was with the following path:

C:/Users/Alber/AppData/Local/Temp/tfhub_modules/a7fe827a4e68369aab0fa6a65479cd37c499e0f4

Just with the explorer I checked the path and found that the a7fe827a4e68369aab0fa6a65479cd37c499e0f4 folder was empty. I don't know why but that shouldn't happen.

Then I just deleted the a7fe827a4e68369aab0fa6a65479cd37c499e0f4 folder and even the tf_hub folder (because I didn't have any other thing but I think it's not necessary to remove the tf_hub folder).

After that I run the script and it downloaded again the required modules normally

INFO:tensorflow:Using C:\Users\Alber\AppData\Local\Temp\tfhub_modules to cache modules.
INFO:tensorflow:Downloading TF-Hub Module 'https://tfhub.dev/google/nnlm-es-dim128-with-normalization/1'.
INFO:tensorflow:Downloading https://tfhub.dev/google/nnlm-es-dim128-with-normalization/1: 38.58MB
...

There is the same issue posted also on Stackoverflow: https://stackoverflow.com/questions/54029556/how-to-fix-runtimeerror-missing-implementation-that-supports-loader-when-cal

[arnoegw]

Hi htjacky, samhavens,

thanks for reporting your issue(s). I've been trying to reproduce but couldn't, or only under unintentional circumstances.

@htjacky: What are the contents of /tmp/tfhub_modules/mobilenet_module just before you run retrain.py? If this directory contains the contents of a tarball downloaded from tfhub.dev, it should look something like

$ ls
assets/  saved_model.pb  tfhub_module.pb  variables/

The error message RuntimeError: Missing implementation ... comes up when using a directory that exists but does not contain a tfhub_module.pb file with a known module format. In particular, you can get this message by trying to load a hub.Module from a directory that contains something else. The particular filename in your example makes me wonder whether you maybe tried using mobilenet_v2_1.4_224.tgz as linked from https://github.com/tensorflow/models/blob/master/research/slim/README.md which is a not a Hub module but a tarball with a bare TensorFlow checkpoint. -- Could that be it?

Side note: While it is technically possible to manually download and uncompress modules from tfhub.dev, and then use the absolute directory name on the local filesystem, we recommend to use the canonical module names https://tfhub.dev/... in code and set the environment variable TFHUB_CACHE_DIR to a convenient location.

@samhavens, what's in the module directory in your case? Could it be the same problem? (I tried htjacky's repro instructions with your PIP package versions to no avail.) At first sight, it seems unlikely that the version of tensorflow itself should have any impact here.

[arnoegw]

Hi @alberduris, good catch: an empty module directory in the cache also triggers this error, and deleting it should remove the problem as you described. But how did it get there in the first place? resolver.py:407 is meant to move it into its final location only after downloading and unpacking has completed successfully.

@samhavens, are you using a https://tfhub.dev/... module and seeing the error for an empty cache dir as well?

[samhavens]

Yeah, @alberduris' solution worked for me as well. I got the error

RuntimeError: Missing implementation that supports: loader(*('/var/folders/l3/2_gkqqs57rl2l1vk0zg978jw0000gn/T/tfhub_modules/1fb57c3ffe1a38479233ee9853ddd7a8ac8a8c47',), **{})

so running rm -rf /var/folders/l3/2_gkqqs57rl2l1vk0zg978jw0000gn/T/tfhub_modules/1fb57c3ffe1a38479233ee9853ddd7a8ac8a8c47/ got it to download the module from tfhub.dev

[arnoegw]

@samhavens, was the deleted directory empty before? Any idea what might have tripped up the original download attempt?

[alberduris]

Hi @alberduris, good catch: an empty module directory in the cache also triggers this error, and deleting it should remove the problem as you described. But how did it get there in the first place? resolver.py:407 is meant to move it into its final location only after downloading and unpacking has completed successfully.

Hi @arnoegw,

I was running the script that uses the TF_Hub module and suddenly the error appeared.

So in one run it went well and for the following the directory was empty, I do not have idea why or how. If it happens to me again, I'll post it here with as much detail as I can.

Also if some type of log or something could be useful, please let me know. Btw, I'm on W10.

[samhavens]

@arnoegw, it wasn’t empty, but it didn’t have the 1GB model in it. Sorry, by the time I went to look I’d lost that console window. It is totally conceivable that the download was interrupted by unreliable internet.

[arnoegw]

To summarize, the failure mode appears to be this: $TFHUB_CACHE_DIR ends up with a subdirectory that looks valid (i.e., its name is the hex hash of a module name, without .uuid.tmp suffix) but has incomplete contents, in particular, not a valid tfhub_module.pb file, which triggers the RuntimeError: Missing implementation that supports: loader(...). (NB: loader, not resolver.)

The key question is: how does the bad directory come about?

• Can download problems trick the code into moving a .tmp directory into its permanent location
  even if downloading or unpacking had problems? That's not meant to happen.
  I compared the code between 0.1.1 and 0.2.0, but no change in that behavior caught my eye.

• Does something else mutate those cache directories? @alberduris said above:
  "in one run it went well and for the following the directory was empty".

Please let us know if you have any useful observations about that.

[samhavens]

@arnoegw definitely if it happens again I will debug more thoroughly and pass my notes on.

[Harshini-Gadige]

@htjacky Any updates please ?

[Harshini-Gadige]

Closing as it is in "awaiting response" status for more than 3 days. Feel free to add comments(if any) and we will reopen. Thanks!

[bobpearson]

I noticed that /google/tf2-preview/mobilenet_v2/feature_vector/2 is missing tfhub_module.pb
$ ls
assets/ saved_model.pb variables/

[arnoegw]

It's expected that the tf2-preview modules no longer contain a tfhub_module.pb file: for TF2, Hub modules will become more like standard SavedModels.

[hw1]

I am using tf version 2.0.0-dev20190504 (by printing the version in my code) and hub version 0.4.0. As suggested above, deleted all the files from the cache directory:
C:\Users\XX\AppData\Local\Temp\tfhub_modules

When I tried to run the following:
module = hub.Module("https://tfhub.dev/google/tf2-preview/mobilenet_v2/classification/2")
print(module.get_signature_names())

I received the same error as above:
RuntimeError: Missing implementation that supports: loader(('C:\Users\XX\AppData\Local\Temp\tfhub_modules\6f74ba811a9cabfa24e78922ad6ca96a0ddfe094',), *{})

BTW, after running the above code, the directory "6f74ba811a9cabfa24e78922ad6ca96a0ddfe094" was created under "tfhub_modules" and it does have the saved_model.pb file (there are two more files: assets and variables). There is also a 6f74ba811a9cabfa24e78922ad6ca96a0ddfe094.descriptor.txt file at the same level as the "6f74ba811a9cabfa24e78922ad6ca96a0ddfe094" directory.
Any suggestions for a fix please?

[alberduris]

Hey @hw1,

You could try this: https://github.com/tensorflow/hub/issues/212#issuecomment-451923229

[hw1]

@alberduris: Thanks for your reply. I have seen your remedy/post on stackoverflow as well. As I mentioned above, I had deleted the directory per your suggestion above, but that did not help in my case (mine is windows CPU version). IMO, the problem is that code might be expecting a tfhub_module.pb  file in the directory which I believe is no longer provided. Instead, a saved_model.pb is made available (see the post by @arnoegw above). However, the code may not be compatible with this change.

[Nishant-Ingle]

I'm getting this problem locally (macOS 10.14.1) when trying to load Universal Sentence Encoder with Tensorflow version 1.12.0, Hub version 0.2.0. However, on Kaggle, it's using Tensorflow version 1.11.0-rc1 (also version 0.2.0 of Hub) and loading USE works.

I can't check if using those versions fixes things, because Tensorflow version 1.11.0-rc1 isn't a valid version for macOS. One potential problem is that I am using Pipenv locally. @htjacky, is this happening in a virtualenv?

The problem is that you need to have USE model extracted in that directory ('/var/folders/l3/2_gkqqs57rl2l1vk0zg978jw0000gn/T/tfhub_modules/1fb57c3ffe1a38479233ee9853ddd7a8ac8a8c47').

If it is not then use :
# For Colab !mkdir -p [folder_name] !cd [folder_location] !curl -L "https://tfhub.dev/google/universal-sentence-encoder-large/3?tf-hub-format=compressed" | tar -zxvC [folder_location]

[bharat0tarahb]

hey @hw1
with tf version
pip install tensorflow==1.13.1
pip install tensorflow-hub

script
https://github.com/tensorflow/hub/blob/master/examples/image_retraining/retrain.py

Try
for "float-point version"
--tfhub_module https://tfhub.dev/google/imagenet/mobilenet_v1_100_224/feature_vector/3
"instrumented for quantization version"
--tfhub_module https://tfhub.dev/google/imagenet/mobilenet_v1_100_224/quantops/feature_vector/3

[hanneshapke]

I got the same error message for a different reason. However, the cause for my error was different.
I tried to run an older TF script using tf-hub's hub.Module call (TensorFlow V1). Once replaced with hub.load(...) (TensorFlow V2, TF 1.15 or >TF2.0), everything works ok.

[iamxpy]

It's expected that the tf2-preview modules no longer contain a tfhub_module.pb file: for TF2, Hub modules will become more like standard SavedModels.

@hw1 @arnoegw Hi guys, I am facing the same problem and want some help. I am trying to get the immediate output of Bert, and according to the solution in this issue, it seems like I have to use tensorflow 1.0 right? But when I change from tf2.0 to tf 1.0, as you pointed out, I got the error because I do not have the file tfhub_module.pb which is expected by hub.Module, so what can I do? Any suggestions will be highly appreciated!

[arnoegw]

The two different file formats are explained here:
https://www.tensorflow.org/hub/tf1_hub_module
https://www.tensorflow.org/hub/tf2_saved_model

[Remorax]

Hi,
None of the above-mentioned solutions worked for me. So I just replaced hub.Module with hub.KerasLayer and it worked. Just so:

Change
embed = hub.Module(<your_model_path>)
to
embed = hub.KerasLayer(<your_model_path>)

[hanneshapke]

@Remorax That is correct. The initial issue was developed under TF 1.x which required the model to be loaded as a graph with hub.Module. Since then, tf.hub evolved and supports now TF2.x which allows eager execution with the hub.KerasLayer. Thank you for the correction!