Tfjs: urllib.error.HTTPError: HTTP Error 404: Not Found

[loretoparisi]

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js):
  NO
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

ProductName:    Mac OS X
ProductVersion: 10.14.5
BuildVersion:   18F132

• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
  Not applicable
• TensorFlow.js installed from (npm or script link):
  Not applicable
• TensorFlow.js version (use command below):
  Not applicable
• Browser version:
  Not applicable
• Tensorflow.js Converter Version:

$ tensorflowjs_converter --version
tensorflowjs 2.4.0

Dependency versions:
  keras 2.4.0
  tensorflow 2.3.1

Describe the current behavior
If I run the command

tensorflowjs_converter --input_format=tf_hub 'https://tfhub.dev/tensorflow/tfjs-model/toxicity/1/default/1' model/

Describe the expected behavior
to download the model from 'https://tfhub.dev/tensorflow/tfjs-model/toxicity/1/default/1'

Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/CodePen/any notebook.
To run the command

tensorflowjs_converter --input_format=tf_hub 'https://tfhub.dev/tensorflow/tfjs-model/toxicity/1/default/1' model/

Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

Traceback (most recent call last):
  File "/usr/local/bin/tensorflowjs_converter", line 8, in <module>
    sys.exit(pip_main())
  File "/usr/local/lib/python3.7/site-packages/tensorflowjs/converters/converter.py", line 757, in pip_main
    main([' '.join(sys.argv[1:])])
  File "/usr/local/lib/python3.7/site-packages/tensorflowjs/converters/converter.py", line 761, in main
    convert(argv[0].split(' '))
  File "/usr/local/lib/python3.7/site-packages/tensorflowjs/converters/converter.py", line 711, in convert
    experiments=args.experiments)
  File "/usr/local/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 701, in convert_tf_hub_module
    module_path = hub.resolve(module_handle)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_hub/module_v2.py", line 52, in resolve
    return registry.resolver(handle)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_hub/registry.py", line 42, in __call__
    return impl(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_hub/compressed_module_resolver.py", line 88, in __call__
    self._lock_file_timeout_sec())
  File "/usr/local/lib/python3.7/site-packages/tensorflow_hub/resolver.py", line 402, in atomic_download
    download_fn(handle, tmp_dir)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_hub/compressed_module_resolver.py", line 83, in download
    response = self._call_urlopen(request)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_hub/compressed_module_resolver.py", line 96, in _call_urlopen
    return url.urlopen(request)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 563, in error
    result = self._call_chain(*args)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 755, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

[tafsiri]

Can confirm that this is not working, both for this model and pretty much from any url I copy from tfhub.

@pyu10055 the url listed in our docs 'https://tfhub.dev/google/imagenet/mobilenet_v1_100_224/classification/1' does still work. I wonder if there is an issue with the newer urls that have 'tfjs_model' in them, or if there is a redirect that needs to be resolved that urllib is not handling. Could you take a look.

[pyu10055]

@loretoparisi The issue is that, toxicity model is an JS model, and the converter can only work with TFHub modules which this model does not have.
are you trying to download the model artifacts to use in an offline use case?

[loretoparisi]

@pyu10055 yes I would like to use the TFHub model artifacts in the offline case, so I was trying the converter to load this model. In fact I realized this model.json is different than the regular ones (it is missing some keys see here).
Any way to convert this model and/or use it offline?

Thank you

[fawazahmed0]

The toxicty model seems to be a graph model, so even if you download it and try converting it with --input_format=tfjs_layers_model flag it will fail

My question:
Is there any way to get the sequential variant of these tfhub js models? So we can convert it

Update: I read this , so my question is somewhat irrelavant to what OP is trying to achieve, but I still would like to ask that question

[loretoparisi]

I would add that, as someone on SF, pointed out, it should be possible to locally host the models folder and serve the model artifacts (via model.json) using maybe a simple nodejs http server.
My point here is: why is not possible to convert the GraphModel to a SavedModel o standard layered model and make it accessible locally via the
tfjs.loadLayersModel api?

Thanks

[tafsiri]

Thanks for the clarification @pyu10055 and @loretoparisi. So it seems in this case you shouldn't need to to do any conversion as it is already a tfjs graphmodel?

You could download the model artifacts here https://tfhub.dev/tensorflow/tfjs-model/toxicity/1/default/1 and serve them from your own storage/locally. Your stack overflow posts suggests you are partway there.

The missing piece that I can see would be updating the model wrapper to support loading from a custom url (and doing the same for loading the tokenizer/vocabulary). This would be needed unless you plan on writing your own pre and post processing code for the model at which point you wouldn't need the wrapper code from the hosted toxicity model.

Could you tell us what you would overall like to achieve so that we can capture what the appropriate feature requests might be?

[loretoparisi]

@tafsiri thank you. I think as you say the wrapper should support custom urls. More specifically, as the layered model loading api, it could support the file://
protocol, so that it would be easy to load models from local serving (via http) or from the file system.

Thanks a lot for your help!

[tensorflow-butler[bot]]

Are you satisfied with the resolution of your issue?
Yes
No