Tf-pose-estimation: Restore pretain model 「mobilenet_thin」but failed.

[edvardHua]

I uses shell script to download the pretrain mobilenet_thin model.
Then, i used in training procedure but encounters error, the error msg is shown below:

shell [2018-02-06 14:46:23,277] [train] [INFO] model weights initialization [2018-02-06 14:46:27,530] [train] [INFO] Restore pretrained weights... 2018-02-06 14:46:30.449865: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number) 2018-02-06 14:46:30.450230: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)

Is there any method to fix it?
Thanks a lot!

[ildoonet]

I guess you need to download model file again.

After that, Please provide the command line and exact source code line which throws an exception.

[edvardHua]

@ildoonet I download the pretrain model file again and the training configuration is shown below:

Training command

python3 train.py --gpus 3 --model mobilenet_thin --datapath ... --imgpath ...

The complete error log of training

╭─root@ubuntu-11 ~/workspace/tf-pose-estimation/src  ‹master*›
╰─# python3 train.py --gpus 3 --model mobilenet_thin
[2018-02-08 13:34:09,747] [train] [INFO] define model+
[2018-02-08 13:34:09,748] [pose_dataset] [INFO] dataflow img_path=/root/workspace/dataset/COCO/images/
loading annotations into memory...
Done (t=1.39s)
creating index...
index created!
[2018-02-08 13:34:11,164] [pose_dataset] [INFO] /root/workspace/dataset/COCO/annotations/ dataset 14186
[0208 13:34:11 @parallel.py:165] [MultiProcessPrefetchData] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d.
[0208 13:34:11 @parallel.py:165] [MultiProcessPrefetchData] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d.
[2018-02-08 13:34:11,293] [pose_dataset] [INFO] dataflow img_path=/root/workspace/dataset/COCO/images/
loading annotations into memory...
Done (t=2.83s)
creating index...
index created!
[2018-02-08 13:34:14,208] [pose_dataset] [INFO] /root/workspace/dataset/COCO/annotations/ dataset 3449
[0208 13:34:14 @parallel.py:165] [MultiProcessPrefetchData] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d.
[0208 13:34:14 @parallel.py:165] [MultiProcessPrefetchData] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d.
[2018-02-08 13:34:15,022] [train] [INFO] tensorboard val image: 20
[2018-02-08 13:34:15,023] [train] [INFO] Tensor("fifo_queue_Dequeue:0", shape=(24, 368, 368, 3), dtype=float32, device=/device:GPU:0)
[2018-02-08 13:34:15,023] [train] [INFO] Tensor("fifo_queue_Dequeue:1", shape=(24, 46, 46, 19), dtype=float32, device=/device:GPU:0)
[2018-02-08 13:34:15,024] [train] [INFO] Tensor("fifo_queue_Dequeue:2", shape=(24, 46, 46, 38), dtype=float32, device=/device:GPU:0)
[2018-02-08 13:35:03,462] [train] [INFO] define model-
INFO:train:define model-
2018-02-08 13:35:10.918669: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-02-08 13:35:11.857563: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least oneNUMA node, so returning NUMA node zero
2018-02-08 13:35:11.858465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.6325
pciBusID: 0000:01:00.0
totalMemory: 10.91GiB freeMemory: 10.02GiB
2018-02-08 13:35:12.462496: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least oneNUMA node, so returning NUMA node zero
2018-02-08 13:35:12.463611: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 1 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.6325
pciBusID: 0000:04:00.0
totalMemory: 10.91GiB freeMemory: 10.75GiB
2018-02-08 13:35:13.257259: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least oneNUMA node, so returning NUMA node zero
2018-02-08 13:35:13.258211: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 2 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.6325
pciBusID: 0000:08:00.0
totalMemory: 10.91GiB freeMemory: 10.75GiB
2018-02-08 13:35:13.288967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Device peer to peer matrix
2018-02-08 13:35:13.289537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1051] DMA: 0 1 2
2018-02-08 13:35:13.290701: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 0:   Y Y Y
2018-02-08 13:35:13.290947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 1:   Y Y Y
2018-02-08 13:35:13.291194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 2:   Y Y Y
2018-02-08 13:35:13.291454: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-02-08 13:35:13.291919: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:1) -> (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:04:00.0, compute capability: 6.1)
2018-02-08 13:35:13.292187: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:2) -> (device: 2, name: GeForce GTX 1080 Ti, pci bus id: 0000:08:00.0, compute capability: 6.1)
[2018-02-08 13:35:14,785] [train] [INFO] model weights initialization
INFO:train:model weights initialization
[2018-02-08 13:35:28,473] [train] [INFO] Restore pretrained weights...
INFO:train:Restore pretrained weights...
2018-02-08 13:35:31.290170: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.290170: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.290192: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.290170: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.290170: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.290254: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.290271: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.290357: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.290910: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.291024: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.291062: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.291195: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.291473: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.291501: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.291381: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.291593: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.291857: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.291939: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.291974: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.292152: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.292445: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.292501: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.292586: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.292784: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.292919: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.292952: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.293020: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.293120: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.293168: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.293355: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.293443: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.293597: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.293622: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.293747: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.294007: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.294007: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.294097: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.294162: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.294337: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.294546: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.294657: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.294680: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.294693: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.294920: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.295032: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.295074: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.295163: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.295357: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.295389: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.295641: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.295681: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.295948: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.295981: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.296109: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.296132: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.296221: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.296363: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.296529: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.296562: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.296627: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.296932: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.296998: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.297941: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.298109: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.298160: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.298321: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.300993: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.301056: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
2018-02-08 13:35:31.301072: W tensorflow/core/framework/op_kernel.cc:1192] Data loss: not an sstable (bad magic number)
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1323, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1302, in _run_fn
    status, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.DataLossError: not an sstable (bad magic number)
         [[Node: save_1/RestoreV2_18 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save_1/Const_0_0, save_1/RestoreV2_18/tensor_names,save_1/RestoreV2_18/shape_and_slices)]]
         [[Node: save_1/RestoreV2_17/_21 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_152_save_1/RestoreV2_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 195, in <module>
    loader.restore(sess, pretrain_path)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1666, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 889, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1317, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.DataLossError: not an sstable (bad magic number)
         [[Node: save_1/RestoreV2_18 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save_1/Const_0_0, save_1/RestoreV2_18/tensor_names,save_1/RestoreV2_18/shape_and_slices)]]
         [[Node: save_1/RestoreV2_17/_21 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_152_save_1/RestoreV2_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Caused by op 'save_1/RestoreV2_18', defined at:
  File "train.py", line 194, in <module>
    loader = tf.train.Saver(net.restorable_variables())
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1218, in __init__
    self.build()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1227, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1263, in _build
    build_save=build_save, build_restore=build_restore)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 751, in _build_internal
    restore_sequentially, reshape)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 427, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 267, in restore_op
    [spec.tensor.dtype])[0])
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 1021, in restore_v2
    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1470, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

DataLossError (see above for traceback): not an sstable (bad magic number)
         [[Node: save_1/RestoreV2_18 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save_1/Const_0_0, save_1/RestoreV2_18/tensor_names,save_1/RestoreV2_18/shape_and_slices)]]
         [[Node: save_1/RestoreV2_17/_21 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_152_save_1/RestoreV2_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

The code for loading pretrain model in network.py

def get_network(type, placeholder_input, sess_for_load=None, trainable=True):
    ...
    elif type == 'mobilenet_thin':
        net = MobilenetNetworkThin({'image': placeholder_input}, conv_width=0.75, conv_width2=0.50, trainable=trainable)
        pretrain_path = 'models/pretrained/mobilenet_v1_1.0_224_2017_06_14/mobilenet_v1_1.0_224.ckpt'
        last_layer = 'MConv_Stage6_L{aux}_5'
    ...
    return

Did i miss something?

[ildoonet]

@edvardHua
You need to run it the parent directory of 'src'.

[linchaozhang]

i have the same problem.
you should modifier the pretrain_pathpretrain_path = 'models/pretrained/mobilenet_v1_1.0_224_2017_06_14/mobilenet_v1_1.0_224.ckpt' in get_network, to pretrain_path = 'models/pretrained/mobilenet_v1_1.0_224_2017_06_14/'
@edvardHua

[edvardHua]

The reason is file corruption, problem sovled, thx.

[NPetsky]

had the same error, for some reason the download.sh script only downloads html links not the real files
I had to download them manually then it worked

[juciny]

I also have this problem...
Thanks @linchaozhang
changing pretrain_path solved my problem

[meetdave06]

How to generate .pb file for mobilenet_accurate using run_checkpoint.py? Currently it points to a folder called trained which doesn't exist