Serving: NotFoundError in mnist_export example

[moskomule]

Hello, I'm trying tensorflow_serving according to the doc . The build and test have passed.

While

bazel build //tensorflow_serving/example:mnist_export

succeeds,

bazel-bin/tensorflow_serving/example/mnist_export /tmp/mnist_model

fails and shows the error message as below.

Traceback (most recent call last):
  File "/home/usr/serving/bazel-bin/tensorflow_serving/example/mnist_export.runfiles/tf_serving/tensorflow_serving/example/mnist_export.py", line 36, in <module>
    from tensorflow.contrib.session_bundle import exporter
  File "/home/usr/serving/bazel-bin/tensorflow_serving/example/mnist_export.runfiles/org_tensorflow/tensorflow/contrib/__init__.py", line 34, in <module>
    from tensorflow.contrib import image
  File "/home/usr/serving/bazel-bin/tensorflow_serving/example/mnist_export.runfiles/org_tensorflow/tensorflow/contrib/image/__init__.py", line 39, in <module>
    from tensorflow.contrib.image.python.ops.single_image_random_dot_stereograms import single_image_random_dot_stereograms
  File "/home/usr/serving/bazel-bin/tensorflow_serving/example/mnist_export.runfiles/org_tensorflow/tensorflow/contrib/image/python/ops/single_image_random_dot_stereograms.py", line 26, in <module>
    "_single_image_random_dot_stereograms.so"))
  File "/home/usr/serving/bazel-bin/tensorflow_serving/example/mnist_export.runfiles/org_tensorflow/tensorflow/contrib/util/loader.py", line 55, in load_op_library
    ret = load_library.load_op_library(path)
  File "/home/usr/serving/bazel-bin/tensorflow_serving/example/mnist_export.runfiles/org_tensorflow/tensorflow/python/framework/load_library.py", line 64, in load_op_library
    None, None, error_msg, error_code)
tensorflow.python.framework.errors_impl.NotFoundError: /home/usr/serving/bazel-bin/tensorflow_serving/example/mnist_export.runfiles/org_tensorflow/tensorflow/contrib/image/python/ops/_single_image_random_dot_stereograms.so: undefined symbol: _ZN6google8protobuf8internal10LogMessageC1ENS0_8LogLevelEPKci

I use VirtualBox and the guest OS is Ubuntu 16.04 and the host OS is the latest macOS.

Thank you.

[berli]

same error
erli@berli-VirtualBox:~/serving$ bazel-bin/tensorflow_serving/example/mnist_saved_model /tmp/mnist_model
Training model...
Extracting /tmp/train-images-idx3-ubyte.gz
Extracting /tmp/train-labels-idx1-ubyte.gz
Extracting /tmp/t10k-images-idx3-ubyte.gz
Extracting /tmp/t10k-labels-idx1-ubyte.gz
2017-04-27 14:40:07.078667: W external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-04-27 14:40:07.078819: W external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-04-27 14:40:07.078877: W external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
WARNING:tensorflow:From /home/berli/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/tf_serving/tensorflow_serving/example/mnist_saved_model.py:72: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use tf.global_variables_initializer instead.
Traceback (most recent call last):
File "/home/berli/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/tf_serving/tensorflow_serving/example/mnist_saved_model.py", line 140, in
tf.app.run()
File "/home/berli/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/home/berli/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/tf_serving/tensorflow_serving/example/mnist_saved_model.py", line 77, in main
prediction_classes = tf.contrib.lookup.index_to_string(
File "/home/berli/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/util/lazy_loader.py", line 53, in __getattr__
module = self._load()
File "/home/berli/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/util/lazy_loader.py", line 42, in _load
module = importlib.import_module(self.__name__)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/berli/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/__init__.py", line 34, in
from tensorflow.contrib import image
File "/home/berli/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/image/__init__.py", line 39, in
from tensorflow.contrib.image.python.ops.single_image_random_dot_stereograms import single_image_random_dot_stereograms
File "/home/berli/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/image/python/ops/single_image_random_dot_stereograms.py", line 26, in
"_single_image_random_dot_stereograms.so"))
File "/home/berli/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/util/loader.py", line 55, in load_op_library
ret = load_library.load_op_library(path)
File "/home/berli/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/python/framework/load_library.py", line 64, in load_op_library
None, None, error_msg, error_code)
tensorflow.python.framework.errors_impl.NotFoundError: /home/berli/serving/bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/image/python/ops/_single_image_random_dot_stereograms.so: undefined symbol: _ZN6google8protobuf8internal10LogMessageC1ENS0_8LogLevelEPKci

[moskomule]

I got the same error on the official Docker on macOS

[ashern]

Joining the choir -

tensorflow.python.framework.errors_impl.NotFoundError: /home/asher/src/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/org_tensorflow/tensorflow/contrib/image/python/ops/_single_image_random_dot_stereograms.so: undefined symbol: _ZN6google8protobuf8internal10LogMessageC1ENS0_8LogLevelEPKci

Tensorflow & Serving built from head vs both Python 2.7 & 3.5 get the same error.

[ashern]

FWIW - if this is a blocking issue for anyone, it can be temporarily worked around by commenting out the single_image_random_dot_stereograms import from the images/__init__.py in the headers used to compile serving. Not ideal - but it gets it working for the moment.

[Shoshin23]

Can you tell me where this file is?

EDIT: I found it. tensorflow/tensorflow/contrib/image/__init__.py. I ran bazel clean after this and tried to build tensorflow serving again but it wouldnt work. It still shows the same error. Can anyone guide me as to how to fix this is?

[Shoshin23]

Also, I'm facing the exact same issue when i try to deploy an inception model with docker and kubernetes as described in the tensorflow serving docs.

[MtDersvan]

@Shoshin23 I actually managed to find it in runfiles that were build by bazel
For example here:

~console
bazel-bin/tensorflow_serving/example/INSERT_NAME.runfiles/org_tensorflow/tensorflow/contrib/image/__init__.py
~

Open it and comment the line as @ashern suggested.
Btw, thanks @ashern, that works for me!

[dongjk]

same problem in Ubuntu 16.04
and comment out

#from tensorflow.contrib.image.python.ops.single_image_random_dot_stereograms import single_image_random_dot_stereograms

in
bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/image/__init__.py
as @ashern mentioned, it works.

[zack6514]

has solved!
comment out

from tensorflow.contrib.image.python.ops.single_image_random_dot_stereograms import single_image_random_dot_stereograms

in bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/image/__init__.py

[stephencwelch]

Commenting out

from tensorflow.contrib.image.python.ops.single_image_random_dot_stereograms import single_image_random_dot_stereograms

in

bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/image/init.py

worked for me as well. Thanks!

[abrykt]

Same issue here.

[nkconnor]

:+1:

[mahnunchik]

Any news?

[ankitachandak]

Same issue here. I've installed tensorflow from source: CPU version with XLA enabled.

[errord]

the problem reason is single_image_random_dot_stereograms.cc<<kernels>> be dependent on TensorShapeProto
it define on protobuf, use class LogMessage. but no deps protobuf on BUILD rule python/ops/_single_image_random_dot_stereograms.so .
so Symbol not found: __ZN6google8protobuf8internal10LogMessageC1ENS0_8LogLevelEPKci(google::protobuf::internal::LogMessage::LogMessage)

[hufangjian]

hello，I got the same erro，my tf serving document is "/home/hufangjian2017/serving". I comment out
"serving/tensorflow/tensorflow/contrib/image/__init__py" "#from tensorflow.contrib.image.python.ops.single_image_random_dot_stereograms import "
what should i do next?

[fennudehaogua]

anyone face the same error after you comment out the code.
why I comment out the code

from tensorflow.contrib.image.python.ops.single_image_random_dot_stereograms import single_image_random_dot_stereograms

in bazel-bin/tensorflow_serving/example/mnist_saved_model.runfiles/org_tensorflow/tensorflow/contrib/image/init.py
didn't work for me .

[MtDersvan]

@fennudehaogua try an earlier version e.g. r1.4:
~
git clone --recurse-submodules https://github.com/tensorflow/serving && \
cd serving && \
git checkout r1.4 && \
cd tensorflow && \
git checkout r1.4 && \
./configure
~
I think they are investigating why this is not working in the newer releases rn.

[fennudehaogua]

@MtDersvan thank you for your reply
my tensorflow version is r1.4 and finally I found the reason is my bazel version is 0.9.0 result in this problem.
bazel 0.8.1 worked for me！

[Anirudh1313]

I am using bazel version 0.9.0 its not working for me either after commenting the line you mentioned above. For using bazel 0.8.1 do we have to repeat the whole build process again?

[fennudehaogua]

@Anirudh1313 I am not sure about this question,but I had executed the code
bazel clean
after change my bazel version.then repeat the whole build process，
the code had been successfully executed.

[soumya-ranjan-sahoo]

Hi. I have been following https://www.tensorflow.org/serving/serving_advanced tutorial for my model deployment. And while invoking the endpoint by the client I end up getting the following error - tensorflow.python.framework.errors_impl.NotFoundError: /home/loonycorn/serving/bazel-bin/tensorflow_serving/example/mnist_client.runfiles/org_tensorflow/tensorflow/contrib/data/python/ops/../../_prefetching_ops.so: undefined symbol: ZN6google8protobuf8internal9ArenaImpl10AddCleanupEPvPFvS3_E

Could you please help me understand this. Or if possible can you help me with a suitable client implementation for mnist_model.

Thanks in advance.

[abrykt]

Hi!

I don't know why that error is there but feel free to take a look at this
simple rest-api that processes requests using tensorflow-serving and a
mnist model. This is my first attempt using tensorflow-serving. Comments
and suggestions are welcome.

https://bitbucket.org/abrykt/mnist_web_service/

Cheers,

Andreas

2018-03-06 6:35 GMT+01:00 soumya-ranjan-sahoo [e-mail]:

Hi. I have been following https://www.tensorflow.org/
serving/serving_advanced tutorial for my model deployment. And while
invoking the endpoint by the client I end up getting the following error -
tensorflow.python.framework.errors_impl.NotFoundError:
/home/loonycorn/serving/bazel-bin/tensorflow_serving/
example/mnist_client.runfiles/org_tensorflow/tensorflow/
contrib/data/python/ops/../../_prefetching_ops.so: undefined symbol:
ZN6google8protobuf8internal9ArenaImpl10AddCleanupEPvPFvS3_E

Could you please help me understand this. Or if possible can you help me
with a suitable client implementation for mnist_model.

Thanks in advance.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/serving/issues/421#issuecomment-370669184,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtxPo661EFiwBHvcZJz2_X4Apk-kUWhks5tbiArgaJpZM4NIoCq
.

--

---

Andreas Brykt
Kocksgatan 36
S-11629 Stockholm
Sweden

[phone]
[email protected]

[gautamvasudevan]

Closing issue - please give the development docker image a shot via:
docker pull tensorflow/serving:1.8.0-devel

That environment has been vetted for building TF Serving + any examples.

[15940260868]

@Anirudh1313 Hi, have you solved the problem? I have the same error even though I have modified the init.py. Could you help me please?