Serving: Example mnist_client in the tutorial failed

Created on 1 Jul 2016 · 10Comments · Source: tensorflow/serving

I just started to learn to setup TensorFlow Serving. I followed the TensorFlow Serving Basics. But failed in the last client step. Any help will be appreciated.

I have run the server side mnist_inference successfully.

$bazel-bin/tensorflow_serving/example/mnist_inference --port=9000 /tmp/mnist_model/00000001/
I tensorflow_serving/session_bundle/session_bundle.cc:130] Attempting to load a SessionBundle from: /tmp/mnist_model/00000001/
I tensorflow_serving/session_bundle/session_bundle.cc:107] Running restore op for SessionBundle
I tensorflow_serving/session_bundle/session_bundle.cc:184] Done loading SessionBundle
I tensorflow_serving/servables/tensorflow/session_bundle_factory.cc:132] Wrapping SessionBundle session to perform batch processing
I tensorflow_serving/example/mnist_inference.cc:166] Running...

But I failed when runing the client.

$bazel-bin/tensorflow_serving/example/mnist_client --num_test=1000 --server=localhost:9000
D0701 17:42:35.348354769   17914 ev_posix.c:101]             Using polling engine: poll
('Extracting', '/tmp/train-images-idx3-ubyte.gz')
('Extracting', '/tmp/train-labels-idx1-ubyte.gz')
('Extracting', '/tmp/t10k-images-idx3-ubyte.gz')
('Extracting', '/tmp/t10k-labels-idx1-ubyte.gz')
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/work/.jumbo/lib/python2.7/site-packages/grpc/_channel.py", line 658, in _call_spin
    completed_call = event.tag(event)
  File "/home/work/.jumbo/lib/python2.7/site-packages/grpc/_channel.py", line 174, in handle_event
    callback()
  File "/home/work/.jumbo/lib/python2.7/site-packages/grpc/_channel.py", line 294, in <lambda>
    self._state.callbacks.append(lambda: fn(self))
  File "/home/work/.jumbo/lib/python2.7/site-packages/grpc/beta/_client_adaptations.py", line 135, in <lambda>
    self._future.add_done_callback(lambda ignored_callback: fn(self))
  File "/home/work/huohuarong/serving/bazel-bin/tensorflow_serving/example/mnist_client.runfiles/tf_serving/tensorflow_serving/example/mnist_client.py", line 98, in <lambda>
    lambda result_future, l=label[0]: done(result_future, l))  # pylint: disable=cell-var-from-loop
  File "/home/work/huohuarong/serving/bazel-bin/tensorflow_serving/example/mnist_client.runfiles/tf_serving/tensorflow_serving/example/mnist_client.py", line 73, in done
    exception = result_future.exception()
  File "/home/work/.jumbo/lib/python2.7/site-packages/grpc/beta/_client_adaptations.py", line 120, in exception
    return _abortion_error(rpc_error_call)
  File "/home/work/.jumbo/lib/python2.7/site-packages/grpc/beta/_client_adaptations.py", line 74, in _abortion_error
    code = rpc_error_call.code()
AttributeError: 'NoneType' object has no attribute 'code'

Source

Huarong

👍1

Most helpful comment

Problem solved with deleting
curl -fSsL -o /bazel/LICENSE.txt https://raw.githubusercontent.com/bazelbuild/bazel/master/LICENSE.txt

WeilongWu-Reflektion on 15 Apr 2017

🎉1 👍1

All 10 comments

Hi @Huarong, thanks for reporting. I wasn't able to reproduce the error (the client works for me); I do notice an extra log message in your stack trace: "ev_posix.c:101] Using polling engine: poll". From the stack-trace, this might be a gRPC issue. @fangweili, any ideas?

sukritiramesh on 1 Jul 2016

File "/home/work/.jumbo/lib/python2.7/site-packages/grpc/beta/_client_adaptations.py",
line 120, in exception
return _abortion_error(rpc_error_call)
File "/home/work/.jumbo/lib/python2.7/site-packages/grpc/beta/_client_adaptations.py",
line 74, in _abortion_error
code = rpc_error_call.code()
AttributeError: 'NoneType' object has no attribute 'code'

The log indicates a bug of gRPC Future implementation.

On Fri, Jul 1, 2016 at 10:37 AM Sukriti Ramesh [email protected]
wrote:

Hi @Huarong https://github.com/Huarong, thanks for reporting. I wasn't
able to reproduce the error (the client works for me); I do notice an extra
log message in your stack trace: "ev_posix.c:101] Using polling engine:
poll". From the stack-trace, this might be a gRPC issue. @fangweili
https://github.com/fangweili, any ideas?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/tensorflow/serving/issues/114#issuecomment-230005672,
or mute the thread
https://github.com/notifications/unsubscribe/ABjH53UeLCRcZjQjbwupWZQFmCTrxkqgks5qRVBHgaJpZM4JC_xH
.

fangweili on 1 Jul 2016

Yes, and this looks similar to the issue described here: https://github.com/grpc/grpc/issues/7133. Will follow up there too to check if there is a recommended way to fix this.

sukritiramesh on 1 Jul 2016

The following Gist is a Dockerfile bug repro of this issue specifically for TensorFlow Serving.
https://gist.github.com/xeb/c831009eec6c850d98383f96ec8e303e

The fix is to install gRPC from master, see:
https://github.com/xeb/tensorflow_serving_examples/blob/master/Dockerfile.BUILD#L27
(which is a repo I started for getting tutorial examples working via Docker. If anyone thinks these should be in main repo, I'll be happy to submit a PR)

xeb on 10 Jul 2016

👍1

Followed up with the gRPC team and the fix is in, but it's not in the latest grpcio python package yet. It will be in the next release.

For now I updated the client code to catch the exception as was recommended on the gRPC thread. Please update to the latest TensorFlow Serving and give that a try to see if the issue is resolved.

kirilg on 13 Jul 2016

@kirilg I ran into the same issue few minutes ago. I updated the tensorflow and the issue is resolved with the latest update. Thanks a lot.

pradyumnanpk on 13 Jul 2016

Great! Thanks for confirming @pradyumnanpk. Closing this bug, please reopen if you encounter this again after syncing to the latest TensorFlow Serving.

kirilg on 13 Jul 2016

@xeb I have the same problem, and got
curl: (22) The requested URL returned error: 404 Not Found
The command '/bin/sh -c mkdir /bazel && cd /bazel && curl -fSsL -O https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh && curl -fSsL -o /bazel/LICENSE.txt https://raw.githubusercontent.com/bazelbuild/bazel/master/LICENSE.txt && chmod +x bazel-*.sh && ./bazel-$BAZEL_VERSION-installer-linux-x86_64.sh && cd / && rm -f /bazel/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh' returned a non-zero code: 22

when building your dockerfile.
I am on Mac OS 10.12.3

Any help would be appreciated!

WeilongWu-Reflektion on 14 Apr 2017

👍1

Problem solved with deleting
curl -fSsL -o /bazel/LICENSE.txt https://raw.githubusercontent.com/bazelbuild/bazel/master/LICENSE.txt

WeilongWu-Reflektion on 15 Apr 2017

🎉1 👍1

This is still an issue:
https://github.com/tensorflow/serving/blob/master/tensorflow_serving/tools/docker/Dockerfile.devel#L64:
curl -fSsL -o /bazel/LICENSE.txt https://raw.githubusercontent.com/bazelbuild/bazel/master/LICENSE.txt

This issue should be reopened. I don't know which other License should be used instead, thereby I can't open a pull request with a fix.