Hub: DataLossError: Checksum does not match:

[tokestermw]

Hi I'm having some problems running TensorFlow Hub models. I am getting DataLossError when running sess.run. I am using Ubuntu 16.04, Python3.6 with tensorflow==1.8.0 and tensorflow_hub=0.1.0.

However, when I run it on a macOS machine, the script below works without problems.

Thanks

In [1]: import tensorflow as tf
   ...: import tensorflow_hub as hub
   ...: 

In [2]: from tensorflow.python.client import device_lib
   ...: print(device_lib.list_local_devices())
   ...: 
2018-04-30 13:53:30.576282: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFl
ow binary was not compiled to use: AVX2 FMA
[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 9094741255132568752
]

In [3]: print("TensorFlow version: {}".format(tf.VERSION))
   ...: print("Eager execution: {}".format(tf.executing_eagerly()))
   ...: 
TensorFlow version: 1.8.0
Eager execution: False

In [4]: ENGLISH_WORD2VEC = 'https://tfhub.dev/google/nnlm-en-dim128/1'
   ...: 

In [5]: # switch here
   ...: embedding_module = ENGLISH_WORD2VEC
   ...: embed = hub.Module(embedding_module, trainable=False)
   ...: 
INFO:tensorflow:Using /tmp/tfhub_modules to cache modules.
INFO:tensorflow:Initialize variable module/embeddings/part_0:0 from checkpoint b'/tmp/tfhub_modules/32f2b2259e1cc8ca58c87692174836
1283e73997/variables/variables' with embeddings

In [6]: with tf.Session() as sess:
   ...:     sess.run(tf.global_variables_initializer())
   ...:     sess.run(tf.local_variables_initializer())
   ...:     sess.run(tf.tables_initializer())
   ...: 
   ...:     print(sess.run(embeddings))
   ...:     


DataLossError: Checksum does not match: stored 3328590665 vs. calculated on the restored bytes 94957053
         [[Node: checkpoint_initializer = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](ch$
ckpoint_initializer/prefix, checkpoint_initializer/tensor_names, checkpoint_initializer/shape_and_slices)]]

Caused by op 'checkpoint_initializer', defined at:
  File "~/venv/bin/ipython", line 11, in <module>
    sys.exit(start_ipython())
  File "~/venv/lib/python3.6/site-packages/IPython/__init__.py", line 119, in start_ipython
    return launch_new_instance(argv=argv, **kwargs)
  File "~/venv/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "~/venv/lib/python3.6/site-packages/IPython/terminal/ipapp.py", line 355, in start
    self.shell.mainloop()
  File "~/venv/lib/python3.6/site-packages/IPython/terminal/interactiveshell.py", line 493, in mainloop
    self.interact()
  File "~/venv/lib/python3.6/site-packages/IPython/terminal/interactiveshell.py", line 484, in interact
    self.run_cell(code, store_history=True)
  File "~/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2718, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "~/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2822, in run_ast_nodes
    if self.run_code(code, result):
  File "~/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2882, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-5-68aeaedb27fc>", line 4, in <module>
    embed = hub.Module(embedding_module, trainable=False)
  File "~/venv/lib/python3.6/site-packages/tensorflow_hub/module.py", line 126, in __init__
    tags=self._tags)
  File "~/venv/lib/python3.6/site-packages/tensorflow_hub/native_module.py", line 282, in _create_impl
    name=name)
  File "~/venv/lib/python3.6/site-packages/tensorflow_hub/native_module.py", line 338, in __init__
    tf.train.init_from_checkpoint(self._checkpoint_path, self._variable_map)
  File "~/lib/python3.6/site-packages/tensorflow/python/training/checkpoint_utils.py", line 221, in i
nit_from_checkpoint
    _set_variable_or_list_initializer(var, ckpt_file, tensor_name_in_ckpt)
  File "~/venv/lib/python3.6/site-packages/tensorflow/python/training/checkpoint_utils.py", line 335, in _
set_variable_or_list_initializer
    _set_checkpoint_initializer(v, ckpt_file, tensor_name, slice_info.spec)
  File "~/venv/lib/python3.6/site-packages/tensorflow/python/training/checkpoint_utils.py", line 299, in _
set_checkpoint_initializer
    ckpt_file, [tensor_name], [slice_spec], [base_type], name=name)[0]
  File "~/venv/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1463, in restore_v2
    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
  File "~/venv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _a
pply_op_helper
    op_def=op_def)
  File "~/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
    op_def=op_def)
  File "~/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1718, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

DataLossError (see above for traceback): Checksum does not match: stored 3328590665 vs. calculated on the restored bytes 94957053
         [[Node: checkpoint_initializer = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](che
ckpoint_initializer/prefix, checkpoint_initializer/tensor_names, checkpoint_initializer/shape_and_slices)]]

[arnoegw]

Hmm, strange. Please help me understand if this is an issue with the cached result of downloading&decompressing, or an issue with its subsequent use. According to your logs, the module 'https://tfhub.dev/google/nnlm-en-dim128/1' has been cached in /tmp/tfhub_modules/32f2b2259e1cc8ca58c876921748361283e73997

Please open a shell session, cd into that cache directory, and see if the following commands gives you the same file sizes and checksums:

$ ls -gGR
.:
total 24
drwxr-x--- 2  4096 Mar 19 10:02 assets/
-rw-r----- 1 11093 Mar 19 10:02 saved_model.pb
-rw-r----- 1     2 Mar 19 10:02 tfhub_module.pb
drwxr-x--- 2  4096 Mar 19 10:02 variables/

./assets:
total 8252
-rw-r----- 1 8449127 Mar 19 10:02 tokens.txt

./variables:
total 486896
-rw-r----- 1 498570752 Mar 19 10:02 variables.data-00000-of-00001
-rw-r----- 1       141 Mar 19 10:02 variables.index

$ find . -type f | xargs md5sum
6fff604a2ca98db081d03a93b5c3eb21  ./saved_model.pb
67fbacf272d157a66875d14fdd6bc0cb  ./variables/variables.index
86c053ab977204d7ac3e70ed8169133a  ./variables/variables.data-00000-of-00001
829ca3b0f994475ae7786114c4a7c526  ./assets/tokens.txt
6d13541d2cedd620921d41aece9c01d3  ./tfhub_module.pb

If you get different checksums (or file sizes), there was a download problem. In that case, please delete the cache directory, make sure you have enough free disk space, and re-run your program (which should re-download the module).

If you get the same checksums but the original error message persists, I need to dig deeper.

[tokestermw]

Hi, so here are the results of the commands. Looks like the checksum for ./variables/variables.data-00000-of-00001 are different.

motoki@here:/tmp/tfhub_modules/32f2b2259e1cc8ca58c876921748361283e73997$ ls -gGR
.:
total 24
drwxr-xr-x 2  4096 Apr 30 14:22 assets
-rw-rw-r-- 1 11093 Apr 30 14:22 saved_model.pb
-rw-rw-r-- 1     2 Apr 30 14:22 tfhub_module.pb
drwxr-xr-x 2  4096 Apr 30 14:22 variables

./assets:
total 8252
-rw-rw-r-- 1 8449127 Apr 30 14:22 tokens.txt

./variables:
total 486892
-rw-rw-r-- 1 498570752 Apr 30 14:23 variables.data-00000-of-00001
-rw-rw-r-- 1       141 Apr 30 14:22 variables.index
motoki@here:/tmp/tfhub_modules/32f2b2259e1cc8ca58c876921748361283e73997$ find . -type f | xargs md5sum
67fbacf272d157a66875d14fdd6bc0cb  ./variables/variables.index
4d9090c03df283800f8b5590d0175847  ./variables/variables.data-00000-of-00001
829ca3b0f994475ae7786114c4a7c526  ./assets/tokens.txt
6d13541d2cedd620921d41aece9c01d3  ./tfhub_module.pb
6fff604a2ca98db081d03a93b5c3eb21  ./saved_model.pb

I did try deleting and re-downloading and I get another checksum for ./variables/variables.data-00000-of-00001.

motoki@here:/tmp/tfhub_modules/32f2b2259e1cc8ca58c876921748361283e73997$ find . -type f | xargs md5sum
67fbacf272d157a66875d14fdd6bc0cb  ./variables/variables.index
30ed10f1887c0d361fb518254198b042  ./variables/variables.data-00000-of-00001
829ca3b0f994475ae7786114c4a7c526  ./assets/tokens.txt
6d13541d2cedd620921d41aece9c01d3  ./tfhub_module.pb
6fff604a2ca98db081d03a93b5c3eb21  ./saved_model.pb

What's strange is, I remember loading the model worked the first time I downloaded the model. I was playing with changing cache dirs, and deleting the cache, and now the model doesn't load regardless of the model.

[tokestermw]

Interestingly, I ran it with a fresh Python 3.5 (as opposed to 3.6) venv and I am getting segfault messages.

import tensorflow as tf
import tensorflow_hub as hub

import os
# default is '/tmp/tfhub_modules/'
# TFHUB_CACHE_DIR = '/tmp/my_module_cache'
# os.environ['TFHUB_CACHE_DIR'] = TFHUB_CACHE_DIR

with tf.Graph().as_default():
  embed = hub.Module("https://tfhub.dev/google/nnlm-en-dim128/1")
  embeddings = embed(["A long sentence.", "single-word", "http://example.com"])

  with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    sess.run(tf.tables_initializer())

    print(sess.run(embeddings))
:--
INFO:tensorflow:Using /tmp/tfhub_modules to cache modules.
INFO:tensorflow:Downloading TF-Hub Module 'https://tfhub.dev/google/nnlm-en-dim128/1'.
INFO:tensorflow:Downloaded TF-Hub Module 'https://tfhub.dev/google/nnlm-en-dim128/1'.
Segmentation fault (core dumped)

Then, I refreshed with Python 3.6 venv and got the same DataLossError as above, and then I refreshed again with Python 3.5 venv and got a DataLossError instead of a segfault. 😅

[arnoegw]

Let's focus on the faulty downloads that you have seen repeatedly with Python 3.6 (Ubuntu 16.04). Has anyone else seen these? I've been investigating for a while but did not find anything.

Let me provide some context so that others can better look at it.

The URL passed to module gets augmented to https://tfhub.dev/google/nnlm-en-dim128/1?tf-hub-format=compressed, which redirects to https://storage.googleapis.com/tensorflow-hub/google/nnlm-en-dim128/1.tar.gz.

https://github.com/tensorflow/hub/blob/r0.1/tensorflow_hub/compressed_module_resolver.py#L101
opens that.

https://github.com/tensorflow/hub/blob/r0.1/tensorflow_hub/resolver.py#L105
iterates over the entries of the tarball (using tarfile's transparent decompression), gets the fileobj for each file in it, makes a matching GFile fileobj for output, and throws the two at shutil.copyfileobj():
https://github.com/python/cpython/blob/3.6/Lib/shutil.py#L76

That seems just about right.

The extracted fileobj knows its size from what is stored in the tarball, not from the end of the stream:
https://github.com/python/cpython/blob/3.6/Lib/tarfile.py#L610
So it's not clear to me just now if the correct file size that you see means a lot.

Can you observe problems somewhere along that call chain? Maybe too few bytes downloaded?

Is there any clue in the diff between the variables.data-00000-of-00001 in your cache, and what you get from manually downloading and untarring the module?

[andresusanopinto]

Closed due to inactivity.

[gowthamkpr]

@arnoegw Did you find a solution for this issue. Thanks!

[arnoegw]

This was never really resolved. But then, no further instances of the problem have been reported, so it is probably very rare. Also, we have no way to reproduce and analyze this. So there seems nothing left to do here.

[arnoegw]

To clarify, I believe this is different from issue https://github.com/tensorflow/hub/issues/305 (despite the similar error message), because this here talks about problems after downloading a Hub module, while that other one appears to talk about TensorFlow's ordinary checkpointing during training.

[afaq-ahmad]

At the the time export inference graph I am also facing the same problem

[MISSIVA20]

j'ai eu cette erreur lorsque j'ai essyer d'exporter le modèle ssd mobile netV2 quelqu'un a pu résolu le problème???
DataLossError (see above for traceback): Checksum does not match: stored 1858275978 vs. calculated on the restored bytes 2637572513
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

[afaq-ahmad]

At the the time export inference graph I am also facing the same problem

When I used previous checkpoint, it successfully converted to a frozen graph

[MISSIVA20]

i do same thank you

Le mer. 8 juil. 2020 à 11:31, Afaq Ahmad [e-mail] a
écrit :

At the the time export inference graph I am also facing the same problem

When I used previous checkpoint, it successfully converted to a frozen
graph

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/hub/issues/44#issuecomment-655405630, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AOGYGJNNIJQPIXVHBUMVN7TR2Q4FPANCNFSM4E5VBJKA
.

[MISSIVA20]

its work with succes

Le mer. 8 juil. 2020 à 12:01, missiva Chatour [e-mail] a
écrit :

i do same thank you

Le mer. 8 juil. 2020 à 11:31, Afaq Ahmad [e-mail] a
écrit :

At the the time export inference graph I am also facing the same problem

When I used previous checkpoint, it successfully converted to a frozen
graph

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/hub/issues/44#issuecomment-655405630, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AOGYGJNNIJQPIXVHBUMVN7TR2Q4FPANCNFSM4E5VBJKA
.