Serving: tensorflow.python.framework.errors_impl.NotFoundError

[SeanYanxml]

Hi, i install the serving by docker.

• First, when i run bazel-bin/tensorflow_serving/example/mnist_saved_model /tmp/mnist_model.
  It can not find the backports.

root@cf72d64a4bb5:/serving# bazel-bin/tensorflow_serving/example/mnist_saved_model /tmp/mnist_model
Traceback (most recent call last):
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/tf_serving/tensorflow_serving/example/mnist_saved_model.py", line 32, in <module>
    import tensorflow as tf
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import *
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/__init__.py", line 64, in <module>
    from tensorflow.python.framework.framework_lib import *
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/framework/framework_lib.py", line 100, in <module>
    from tensorflow.python.framework.subscribe import subscribe
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/framework/subscribe.py", line 26, in <module>
    from tensorflow.python.ops import variables
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/ops/variables.py", line 26, in <module>
    from tensorflow.python.ops import control_flow_ops
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/ops/control_flow_ops.py", line 70, in <module>
    from tensorflow.python.ops import tensor_array_ops
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/ops/tensor_array_ops.py", line 33, in <module>
    from tensorflow.python.util import tf_should_use
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/util/tf_should_use.py", line 28, in <module>
    from backports import weakref  # pylint: disable=g-bad-import-order
ImportError: No module named backports

I fix it by pip install backports.weakref

• Secondly, when i run bazel-bin/tensorflow_serving/example/mnist_saved_model /tmp/mnist_model. It can not work like this.

Traceback (most recent call last):
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/tf_serving/tensorflow_serving/example/mnist_saved_model.py", line 142, in <module>
    tf.app.run()
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/tf_serving/tensorflow_serving/example/mnist_saved_model.py", line 71, in main
    table = tf.contrib.lookup.index_to_string_table_from_tensor(
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/util/lazy_loader.py", line 53, in __getattr__
    module = self._load()
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/util/lazy_loader.py", line 42, in _load
    module = importlib.import_module(self.__name__)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/__init__.py", line 35, in <module>
    from tensorflow.contrib import image
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/image/__init__.py", line 40, in <module>
    from tensorflow.contrib.image.python.ops.single_image_random_dot_stereograms import single_image_random_dot_stereograms
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/image/python/ops/single_image_random_dot_stereograms.py", line 26, in <module>
    "_single_image_random_dot_stereograms.so"))
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/util/loader.py", line 55, in load_op_library
    ret = load_library.load_op_library(path)
  File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/framework/load_library.py", line 64, in load_op_library
    None, None, error_msg, error_code)
tensorflow.python.framework.errors_impl.NotFoundError: /serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/image/python/ops/_single_image_random_dot_stereograms.so: undefined symbol: _ZN6google8protobuf8internal10LogMessageC1ENS0_8LogLevelEPKci

I can not fix it. Any advise?

[SeanYanxml]

I found it is like the 421

[shenshaoyong]

hi, @SeanYanxml , I installed the by pip install backports.weakref, however, I still got the error:
File "/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/util/tf_should_use.py", line 28, in
from backports import weakref # pylint: disable=g-bad-import-order
ImportError: No module named backports

I am running tensorflow1.2.0 ,python3.6 on macSierra.
Thank you

[SeanYanxml]

@shenshaoyong The python3's order maybe pip3 install backports.weakref.
You can check your installed python modules by pip list or pip3 list. If you install them by pip.
I suggest you to use docker to install it. More to read https://tensorflow.github.io/serving/docker .
Also it is not support mac OS X to install it by compiling the file. For more to read Support building TensorFlow Serving on OS X #1

[shenshaoyong]

hi, @SeanYanxml , Thank you for your reply. Now, I get the answer.
I changed the PYTHON_BINARY in the file mnist_saved_model.
From PYTHON_BINARY = '/usr/bin/python'
to
PYTHON_BINARY ='/Users/ssy/anaconda2/bin/python' # '/usr/bin/python'
If I changed the value to '/Users/ssy/anaconda3/bin/python', namely python3. The error will occur as before.
The value '/usr/bin/python' is the MacOS's python2.

Thank you again.

[SeanYanxml]

@shenshaoyong I think you should config the tensorlfow first by the following orders:

cd tensorflow
./configure
# to config your python path and the python repo path.

The python2 of the Mac OS is protected. So we can not use it .
Best wishes.

[slothkong]

@SeanYanxml
Hi, I also encounter an ImportError related to backports when I run:

bazel-bin/tensorflow_serving/example/inception_export --checkpoint_dir=inception-v3 --export_dir=inception-export

I'm following the _tensorflow serving_ tutorial in Siraj Raval's github. I got something like:

<module> from tensorflow.python.ops import tensor_array_ops File "/serving/bazel-bin/tensorflow_serving/example/inception_export.runfiles/org_tensorflow/tensorflow/python/ops/tensor_array_ops.py", line 33, in <module> from tensorflow.python.util import tf_should_use File "/serving/bazel-bin/tensorflow_serving/example/inception_export.runfiles/org_tensorflow/tensorflow/python/util/tf_should_use.py", line 28, in <module> from backports import weakref # pylint: disable=g-bad-import-order ImportError: No module named backports root@ea5c40be5da4:/serving#

As you can see in the last line, I'm executing the container in bash (root@ea5c40be5da4:/serving#) My question is: If I try using pip install backports, this command will the pip installation that is inside the container to modify the container files, correct? It will not install anything in my OS? What do you mean when you say "I suggest you to use docker to install it" ?
I hope I'm not asking too much. I'm a complete beginner to this.

[gbin]

I think something is really broken in the python detection, it should never try to import backport under python3 but checkout what I have on Linux Arch:

sal ➜  ~  python3
Python 3.6.1 (default, Mar 27 2017, 00:27:06) 
[GCC 6.3.1 20170306] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import *
  File "/usr/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 63, in <module>
    from tensorflow.python.framework.framework_lib import *
  File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/framework_lib.py", line 100, in <module>
    from tensorflow.python.framework.subscribe import subscribe
  File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/subscribe.py", line 26, in <module>
    from tensorflow.python.ops import variables
  File "/usr/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 26, in <module>
    from tensorflow.python.ops import control_flow_ops
  File "/usr/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 70, in <module>
    from tensorflow.python.ops import tensor_array_ops
  File "/usr/lib/python3.6/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 33, in <module>
    from tensorflow.python.util import tf_should_use
  File "/usr/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 28, in <module>
    from backports import weakref  # pylint: disable=g-bad-import-order
ModuleNotFoundError: No module named 'backports'
>>> 

My installed package is:

community/python-tensorflow-cuda 1.2.0-1 [installed]
    Library for computation using data flow graphs for scalable machine learning

[slothkong]

@gbin , I managed to get through that issue with pip install backports.weakref. To be hones I don't know why I needed that. I though the image had all what I needed to set the tensorflow_serving application.
Anyways, I'm past that now, but whenever I try to use bazel-bin to input the image of a panda, I get an error for the rpc call. I found an issue thread on the "issues" section of tensorflow_serving git, and I'm trying to solve it. Let me know if you get to this point and experience something similar.

[gbin]

I just changed the line to "import weakref" and it works perfectly on python3.

[owulveryck]

@gbin thank you, that did the trick!

[jfmlima]

@gbin in which file?

[owulveryck]

@Jflima92 tensorflow/python/util/tf_should_use.py

[jfmlima]

@owulveryck Thanks mate

[kashefy]

Made the same change as @gbin on python 2.7 and it worked also.
pip install backports.weakref didn't get rid of the error.
It seems that weakref needs to be imported as a any other package and not a subpackage of backports. Might have to do with the way it was installed (pip install backports.weakref instead of just pip install backports, which doesn't work anyway)

[Prarthana1116]

I run this code:
bazel-bin/tensorflow_serving/example/inception_saved_model --checkpoint_dir=inception-v3 --output_dir=\tmp/inception-export
from "https://www.tensorflow.org/serving/serving_inception" this tutorial and got the following error.Can anyone figure out where did I make the mistake?

root@73eff5dd7407:/serving# bazel-bin/tensorflow_serving/example/inception_saved_model --checkpoint_dir=inception-v3 --output_dir=\tmp/inception-export
Traceback (most recent call last):
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/tf_serving/tensorflow_serving/example/inception_saved_model.py", line 202, in
tf.app.run()
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/python/platform/app.py", line 124, in run
_sys.exit(main(argv))
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/tf_serving/tensorflow_serving/example/inception_saved_model.py", line 198, in main
export()
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/tf_serving/tensorflow_serving/example/inception_saved_model.py", line 91, in export
table = tf.contrib.lookup.index_to_string_table_from_tensor(class_tensor)
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/python/util/lazy_loader.py", line 53, in __getattr__
module = self._load()
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/python/util/lazy_loader.py", line 42, in _load
module = importlib.import_module(self.__name__)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/contrib/__init__.py", line 83, in
from tensorflow.contrib.eager.python import tfe as eager
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/contrib/eager/python/tfe.py", line 78, in
from tensorflow.contrib.eager.python.datasets import Iterator
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/contrib/eager/python/datasets.py", line 23, in
from tensorflow.contrib.data.python.ops import prefetching_ops
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/contrib/data/python/ops/prefetching_ops.py", line 25, in
resource_loader.get_path_to_datafile("../../_prefetching_ops.so"))
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/contrib/util/loader.py", line 55, in load_op_library
ret = load_library.load_op_library(path)
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/python/framework/load_library.py", line 56, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename, status)
File "/serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: /serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/contrib/data/python/ops/../../_prefetching_ops.so: undefined symbol: _ZN6google8protobuf8internal9ArenaImpl10AddCleanupEPvPFvS3_E

[nileshmalpekar]

I'm getting same error as @Pokharaa

tensorflow.python.framework.errors_impl.NotFoundError: /serving/bazel-bin/tensorflow_serving/example/inception_saved_model.runfiles/org_tensorflow/tensorflow/contrib/data/python/ops/../../_prefetching_ops.so: undefined symbol: _ZN6google8protobuf8internal9ArenaImpl10AddCleanupEPvPFvS3_E

[SeanYanxml]

You can found the solution at the issus #421 . This issus will be closed. @nileshmalpekar @Pokharaa

[WanZhang1]

tensorflow.python.framework.errors_impl.NotFoundError: /opt/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/data/python/ops/../../_prefetching_ops.so: undefined symbol: _ZN6google8protobuf8internal9ArenaImpl15AllocateAlignedEm

how to solve ?

[Diamond0728]

I got the familiar trouble. I just removed the other folders which had no relationship to my project. It worked.

[fennudehaogua]

have you solve this problem ?
why I comment out the code

from tensorflow.contrib.image.python.ops.single_image_random_dot_stereograms import single_image_random_dot_stereograms

in bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/image/init.py
didn't work for me .

[Prarthana1116]

you can refer this video..hope it might help to solve those problems:
https://www.youtube.com/watch?v=SsaJJsRvwIs&lc=z23evh2p1xiacv4fvacdp434cpgd0yx25vdlosz0g4xw03c010c

[BUPTAlanMa]

@Pokharaa @nileshmalpekar I meet the same error ".../../../_prefetching_ops.so: undefined symbol: _ZN6google8protobuf8internal9ArenaImpl15AllocateAlignedEm". I tried comment "#from tensorflow.contrib.image.python.ops.single_image_random_dot_stereograms import". It didi not work.
Did you finnally Solve this problem "../_prefetching_ops.so". Your video link does work? Help, Confuse me long time

[maybeluo]

I have similar errors as @BUPTAlanMa when I use lazy_adam as optimizer form contrib.opt.

[jianyinglang]

I am getting similar error message:
tensorflow.python.framework.errors_impl.NotFoundError: ~/.tf_15/lib/python3.5/site-packages/tensorflow/contrib/data/python/ops/../../_prefetching_ops.so: undefined symbol: _ZN6google8protobuf8internal26fixed_address_empty_stringB5cxx11E

[gautamvasudevan]

Closing - please see the latest Docker examples for bringing up a build environment that should also work for this example.

[symbolnotdefined]

Still the issue is not solved what to do now??