Bert-as-service: fail to optimize the graph!

[flyzaway]

Traceback (most recent call last):
File "/home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: Read less bytes than requested
[[{{node checkpoint_initializer_166}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/admin/.local/lib/python3.6/site-packages/bert_serving/server/graph.py", line 134, in optimize_graph
sess.run(tf.global_variables_initializer())
File "/home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: Read less bytes than requested
[[node checkpoint_initializer_166 (defined at home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'checkpoint_initializer_166':
File "home/admin/.local/bin/bert-serving-start", line 8, in
sys.exit(main())
File "home/admin/.local/lib/python3.6/site-packages/bert_serving/server/cli/__init__.py", line 4, in main
with BertServer(get_run_args()) as server:
File "home/admin/.local/lib/python3.6/site-packages/bert_serving/server/__init__.py", line 68, in __init__
with Pool(processes=1) as pool:
File "usr/lib/python3.6/multiprocessing/pool.py", line 174, in __init__
self._repopulate_pool()
File "usr/lib/python3.6/multiprocessing/pool.py", line 239, in _repopulate_pool
w.start()
File "usr/lib/python3.6/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "usr/lib/python3.6/multiprocessing/context.py", line 277, in _Popen
return Popen(process_obj)
File "usr/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "usr/lib/python3.6/multiprocessing/popen_fork.py", line 73, in _launch
code = process_obj._bootstrap()
File "usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(self._args, *self._kwargs)
File "usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(args, *kwds))
File "home/admin/.local/lib/python3.6/site-packages/bert_serving/server/graph.py", line 85, in optimize_graph
tf.train.init_from_checkpoint(init_checkpoint, assignment_map)
File "home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/training/checkpoint_utils.py", line 291, in init_from_checkpoint
init_from_checkpoint_fn)
File "home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1940, in merge_call
return self._merge_call(merge_fn, args, kwargs)
File "home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1947, in _merge_call
return merge_fn(self._strategy, args, *kwargs)
File "home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/training/checkpoint_utils.py", line 286, in
ckpt_dir_or_file, assignment_map)
File "home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/training/checkpoint_utils.py", line 334, in _init_from_checkpoint
_set_variable_or_list_initializer(var, ckpt_file, tensor_name_in_ckpt)
File "home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/training/checkpoint_utils.py", line 458, in _set_variable_or_list_initializer
_set_checkpoint_initializer(variable_or_list, ckpt_file, tensor_name, "")
File "home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/training/checkpoint_utils.py", line 412, in _set_checkpoint_initializer
ckpt_file, [tensor_name], [slice_spec], [base_type], name=name)[0]
File "home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_io_ops.py", line 1696, in restore_v2
name=name)
File "home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(args, *kwargs)
File "home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "home/admin/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
self._traceback = tf_stack.extract_stack()

Traceback (most recent call last):
File "/home/admin/.local/bin/bert-serving-start", line 8, in
sys.exit(main())
File "/home/admin/.local/lib/python3.6/site-packages/bert_serving/server/cli/__init__.py", line 4, in main
with BertServer(get_run_args()) as server:
File "/home/admin/.local/lib/python3.6/site-packages/bert_serving/server/__init__.py", line 71, in __init__
self.graph_path, self.bert_config = pool.apply(optimize_graph, (self.args,))
TypeError: 'NoneType' object is not iterable

[kenniy]

I had similar issue... if you have TensorFlow 2 installed, try uninstall that and install the last version of TensorFlow 1.. i.e. pip install tensorflow==1.15.2. that fixed it.. till this gets updated to work with v2

[flyzaway]

my tensorflow is v1, but this problem still exists

[yolochai]

check if those cpkt file name, config file name of your files is same as your starting server. I got the same problem when starting a server for Biobert. then I realized i should set cpkt_name="biobert_model.cpkt" because my cpkt name is not "bert_model.cpkt" as default.