Models: [slim] Python 3 support

[pokey]

System information

• What is the top-level directory of the model you are using: slim
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS 10.12.3 (16D32)
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.1.0
• Bazel version (if compiling from source): N/A
• CUDA/cuDNN version: N/A
• GPU model and memory: N/A
• Exact command to reproduce: cd slim && python download_and_convert_data.py --dataset_name=flowers --dataset_dir=data

Here's the output from the environment capture script:

== cat /etc/issue ===============================================
Darwin Brandons-MBP.connect 16.4.0 Darwin Kernel Version 16.4.0: Thu Dec 22 22:53:21 PST 2016; root:xnu-3789.41.3~3/RELEASE_X86_64 x86_64
Mac OS X 10.12.3

== are we in docker =============================================
No

== compiler =====================================================
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin16.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

== uname -a =====================================================
Darwin Brandons-MBP.connect 16.4.0 Darwin Kernel Version 16.4.0: Thu Dec 22 22:53:21 PST 2016; root:xnu-3789.41.3~3/RELEASE_X86_64 x86_64

== check pips ===================================================
numpy (1.12.1)
protobuf (3.2.0)
tensorflow (1.1.0)

== check for virtualenv =========================================
False

== tensorflow import ============================================
tf.VERSION = 1.1.0
tf.GIT_VERSION = v1.1.0-rc0-61-g1ec6ed5
tf.COMPILER_VERSION = v1.1.0-rc0-61-g1ec6ed5
Sanity check: array([1], dtype=int32)

== env ==========================================================
LD_LIBRARY_PATH is unset
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================

== cuda libs  ===================================================

Describe the problem

The slim/download_and_convert_data.py script doesn't work with Python 3.6.0. It gives the following output:

```Python traceback
Traceback (most recent call last):
File "download_and_convert_data.py", line 39, in
from datasets import download_and_convert_cifar10
File "/Users/pokey/src/deep-stats/models/slim/datasets/download_and_convert_cifar10.py", line 29, in
import cPickle
ModuleNotFoundError: No module named 'cPickle'

To fix this, I patched as follows:

```diff
diff --git a/slim/datasets/download_and_convert_cifar10.py b/slim/datasets/download_and_convert_cifar10.py
index 2cb787d..ee8928b 100644
--- a/slim/datasets/download_and_convert_cifar10.py
+++ b/slim/datasets/download_and_convert_cifar10.py
@@ -26,7 +26,7 @@ from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function

-import cPickle
+import pickle
 import os
 import sys
 import tarfile
@@ -73,7 +73,7 @@ def _add_to_tfrecord(filename, tfrecord_writer, offset=0):
     The new offset.
   """
   with tf.gfile.Open(filename, 'r') as f:
-    data = cPickle.load(f)
+    data = pickle.load(f)

   images = data['data']
   num_images = images.shape[0]

However, I then get the following:

```Python traceback

Downloading flower_photos.tgz 100.0%
Successfully downloaded flower_photos.tgz 228813984 bytes.
Converting image 1/3320 shard 0Traceback (most recent call last):
File "download_and_convert_data.py", line 73, in
tf.app.run()
File "/Users/pokey/.pyenv/versions/deep-stats/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "download_and_convert_data.py", line 65, in main
download_and_convert_flowers.run(FLAGS.dataset_dir)
File "/Users/pokey/src/deep-stats/models/slim/datasets/download_and_convert_flowers.py", line 202, in run
dataset_dir)
File "/Users/pokey/src/deep-stats/models/slim/datasets/download_and_convert_flowers.py", line 139, in _convert_dataset
image_data = tf.gfile.FastGFile(filenames[i], 'r').read()
File "/Users/pokey/.pyenv/versions/deep-stats/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 122, in read
pywrap_tensorflow.ReadFromStream(self._read_buf, length, status))
File "/Users/pokey/.pyenv/versions/deep-stats/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 90, in _prepare_value
return compat.as_str_any(val)
File "/Users/pokey/.pyenv/versions/deep-stats/lib/python3.6/site-packages/tensorflow/python/util/compat.py", line 106, in as_str_any
return as_str(value)
File "/Users/pokey/.pyenv/versions/deep-stats/lib/python3.6/site-packages/tensorflow/python/util/compat.py", line 84, in as_text
return bytes_or_text.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
```

[drpngx]

@sguada are we aware of any issues with py3 and slim?

[nealwu]

There are definitely issues, but it's not clear to me how best to fix them. See also https://github.com/tensorflow/models/pull/1346.

[cyberutkarsh]

@nealwu : @drpngx I am having the same issues.... So whats the workaround in this case? Is it recommended to continue to use bazel to convert/train on the dataset?

[nealwu]

For now I would use Python 2 to work with these models if possible. bazel should still be fine; I don't think there are any issues there.

[Harshini-Gadige]

Is this still an issue ?

[wt-huang]

Closing as this is resolved.