Models: SSL certificate verification failed in mnist.py

Created on 5 Dec 2018 · 2Comments · Source: tensorflow/models

System information

What is the top-level directory of the model you are using: models/official/mnist
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS 10.14.1
TensorFlow installed from (source or binary): binary (with pip)
TensorFlow version (use command below): v1.12.0-rc2-3-ga6d8ffae09 1.12.0
Bazel version (if compiling from source): N/A
CUDA/cuDNN version: none
GPU model and memory: no GPU support (on macOS)
Exact command to reproduce: python mnist.py

Describe the problem

I cannot download the MNIST data due to a certificate failure:

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)

I believe I followed the steps in the README. I installed Python 3.6.7, created a virtual environment, cloned this repo, added it to the PYTHONPATH, and ran python mnist.py.

Following this thread on StackOverflow, I replaced line 70 of models/official/mnist/dataset.py:

  url = 'https://storage.googleapis.com/cvdf-datasets/mnist/' + filename + '.gz'

with the insecure version:

  url = 'http://storage.googleapis.com/cvdf-datasets/mnist/' + filename + '.gz'

This works, but it's a hack. I tried directing Python to a different binary of OpenSSL, such as LibreSSL, but it seems that I would have to recompile Python instead of installing from a package installer. I would think that TensorFlow would work right out of the box, so I'm not sure if this is a bug or if I am doing something wrong.

Source code / logs

My bash script that installed the setup is:

# Install Python 3.6.7 manually from this page
# https://www.python.org/downloads/release/python-367/

# Then verify installation with
python3 --version
pip3 --version

# Install virtualenv for current user
export PYTHONUSERBASE=$HOME
pip3 install virtualenv

# Create and activate a new virtual environment
virtualenv --system-site-packages -p python3 ./venv
source ./venv/bin/activate

# Upgrade pip in this environment
pip3 install --upgrade pip

# Install TensorFlow (no GPU support on macOS):
pip3 install --upgrade tensorflow

# Verify that TensorFlow is working with:
python -c "import tensorflow as tf; tf.enable_eager_execution(); print(tf.reduce_sum(tf.random_normal([1000, 1000])))"

# Use MNIST as an official model from this repo
git clone [email protected]:tensorflow/models.git
export PYTHONPATH="$PYTHONPATH:$HOME/models"
pip3 install --user -r models/official/requirements.txt

The bash log is below in two parts, before and after the change in protocol.

Before the change:

(venv) $ grep -r "https://storage.googleapis.com/cvdf-datasets/mnist" models
Binary file models/official/mnist/__pycache__/dataset.cpython-36.pyc matches
models/official/mnist/dataset.py:  url = 'https://storage.googleapis.com/cvdf-datasets/mnist/' + filename + '.gz'
models/tutorials/image/mnist/convolutional.py:SOURCE_URL = 'https://storage.googleapis.com/cvdf-datasets/mnist/'
(venv) $ python models/official/mnist/mnist.py 
2018-12-05 12:28:17.752081: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
I1205 12:28:19.777540 4566869440 tf_logging.py:115] Initializing RunConfig with distribution strategies.
I1205 12:28:19.777722 4566869440 tf_logging.py:115] Not using Distribute Coordinator.
I1205 12:28:19.777999 4566869440 tf_logging.py:115] Using config: {'_model_dir': '/tmp/mnist_model', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.contrib.distribute.python.one_device_strategy.OneDeviceStrategy object at 0x12aeb72e8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x12aeb7358>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}
Downloading https://storage.googleapis.com/cvdf-datasets/mnist/train-images-idx3-ubyte.gz to /var/folders/zx/j_gjm0ld081b_mcqmg3gp9l1zp59y6/T/tmpkx13zmq0.gz
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "models/official/mnist/mnist.py", line 236, in <module>
    absl_app.run(main)
  File "~/venv/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "~/venv/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "models/official/mnist/mnist.py", line 230, in main
    run_mnist(flags.FLAGS)
  File "models/official/mnist/mnist.py", line 211, in run_mnist
    mnist_classifier.train(input_fn=train_input_fn, hooks=train_hooks)
  File "~/venv/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 354, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "~/venv/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1205, in _train_model
    return self._train_model_distributed(input_fn, hooks, saving_listeners)
  File "~/venv/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1273, in _train_model_distributed
    input_fn, model_fn_lib.ModeKeys.TRAIN, self._train_distribution)
  File "~/venv/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1064, in _get_iterator_from_input_fn
    lambda: self._call_input_fn(input_fn, mode))
  File "~/venv/lib/python3.6/site-packages/tensorflow/contrib/distribute/python/one_device_strategy.py", line 66, in distribute_dataset
    self._call_dataset_fn(dataset_fn), [self._device],
  File "~/venv/lib/python3.6/site-packages/tensorflow/python/training/distribute.py", line 537, in _call_dataset_fn
    result = dataset_fn()
  File "~/venv/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1064, in <lambda>
    lambda: self._call_input_fn(input_fn, mode))
  File "~/venv/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1162, in _call_input_fn
    return input_fn(**kwargs)
  File "models/official/mnist/mnist.py", line 192, in train_input_fn
    ds = dataset.train(flags_obj.data_dir)
  File "~/models/official/mnist/dataset.py", line 112, in train
    'train-labels-idx1-ubyte')
  File "~/models/official/mnist/dataset.py", line 84, in dataset
    images_file = download(directory, images_file)
  File "~/models/official/mnist/dataset.py", line 73, in download
    urllib.request.urlretrieve(url, zipped_filepath)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 248, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)>

After the change in protocol to HTTP, as evidenced by the result of the grep command:

(venv) $ grep -r "://storage.googleapis.com/cvdf-datasets/mnist" models
Binary file models/official/mnist/__pycache__/dataset.cpython-36.pyc matches
models/official/mnist/dataset.py:  url = 'http://storage.googleapis.com/cvdf-datasets/mnist/' + filename + '.gz'
models/tutorials/image/mnist/convolutional.py:SOURCE_URL = 'https://storage.googleapis.com/cvdf-datasets/mnist/'
(venv) $ python models/official/mnist/mnist.py 
2018-12-05 12:28:57.989102: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
I1205 12:28:59.933488 4764476864 tf_logging.py:115] Initializing RunConfig with distribution strategies.
I1205 12:28:59.933668 4764476864 tf_logging.py:115] Not using Distribute Coordinator.
I1205 12:28:59.933948 4764476864 tf_logging.py:115] Using config: {'_model_dir': '/tmp/mnist_model', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': <tensorflow.contrib.distribute.python.one_device_strategy.OneDeviceStrategy object at 0x12c5942e8>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x12c594358>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}
Downloading http://storage.googleapis.com/cvdf-datasets/mnist/train-images-idx3-ubyte.gz to /var/folders/zx/j_gjm0ld081b_mcqmg3gp9l1zp59y6/T/tmp4667tagp.gz
Downloading http://storage.googleapis.com/cvdf-datasets/mnist/train-labels-idx1-ubyte.gz to /var/folders/zx/j_gjm0ld081b_mcqmg3gp9l1zp59y6/T/tmp7qxgbdwg.gz
I1205 12:29:01.782899 4764476864 tf_logging.py:115] Calling model_fn.
I1205 12:29:02.112184 4764476864 tf_logging.py:115] Done calling model_fn.
I1205 12:29:02.141009 4764476864 tf_logging.py:115] Create CheckpointSaverHook.
I1205 12:29:02.263503 4764476864 tf_logging.py:115] Graph was finalized.
I1205 12:29:02.439916 4764476864 tf_logging.py:115] Running local_init_op.
I1205 12:29:02.446758 4764476864 tf_logging.py:115] Done running local_init_op.
I1205 12:29:02.677207 4764476864 tf_logging.py:115] Saving checkpoints for 0 into /tmp/mnist_model/model.ckpt.
I1205 12:29:05.109802 4764476864 tf_logging.py:115] cross_entropy = 2.3048978, learning_rate = 1e-04, train_accuracy = 0.05
I1205 12:29:05.109989 4764476864 tf_logging.py:115] loss = 2.3048978, step = 0
I1205 12:29:14.766379 4764476864 tf_logging.py:115] global_step/sec: 10.3554
I1205 12:29:14.766859 4764476864 tf_logging.py:115] cross_entropy = 0.3506276, learning_rate = 1e-04, train_accuracy = 0.48 (9.657 sec)
I1205 12:29:14.766991 4764476864 tf_logging.py:115] loss = 0.3506276, step = 100 (9.657 sec)
I1205 12:29:24.416563 4764476864 tf_logging.py:115] global_step/sec: 10.3625
I1205 12:29:24.417036 4764476864 tf_logging.py:115] cross_entropy = 0.111308776, learning_rate = 1e-04, train_accuracy = 0.65 (9.650 sec)
I1205 12:29:24.417161 4764476864 tf_logging.py:115] loss = 0.111308776, step = 200 (9.650 sec)
I1205 12:29:33.866394 4764476864 tf_logging.py:115] global_step/sec: 10.5822
I1205 12:29:33.866874 4764476864 tf_logging.py:115] cross_entropy = 0.22374004, learning_rate = 1e-04, train_accuracy = 0.725 (9.450 sec)
I1205 12:29:33.867074 4764476864 tf_logging.py:115] loss = 0.22374004, step = 300 (9.450 sec)
I1205 12:29:43.441606 4764476864 tf_logging.py:115] global_step/sec: 10.4436
I1205 12:29:43.442087 4764476864 tf_logging.py:115] cross_entropy = 0.14755917, learning_rate = 1e-04, train_accuracy = 0.772 (9.575 sec)
I1205 12:29:43.442266 4764476864 tf_logging.py:115] loss = 0.14755917, step = 400 (9.575 sec)
I1205 12:29:53.074919 4764476864 tf_logging.py:115] global_step/sec: 10.3806
I1205 12:29:53.075423 4764476864 tf_logging.py:115] cross_entropy = 0.14501753, learning_rate = 1e-04, train_accuracy = 0.8 (9.633 sec)
I1205 12:29:53.075557 4764476864 tf_logging.py:115] loss = 0.14501753, step = 500 (9.633 sec)
I1205 12:30:02.875329 4764476864 tf_logging.py:115] Saving checkpoints for 600 into /tmp/mnist_model/model.ckpt.
I1205 12:30:03.097067 4764476864 tf_logging.py:115] Finalize system.
I1205 12:30:03.207376 4764476864 tf_logging.py:115] Loss for final step: 0.13695708.
Downloading http://storage.googleapis.com/cvdf-datasets/mnist/t10k-images-idx3-ubyte.gz to /var/folders/zx/j_gjm0ld081b_mcqmg3gp9l1zp59y6/T/tmp8h4voiac.gz
Downloading http://storage.googleapis.com/cvdf-datasets/mnist/t10k-labels-idx1-ubyte.gz to /var/folders/zx/j_gjm0ld081b_mcqmg3gp9l1zp59y6/T/tmpo0jnubw3.gz
I1205 12:30:04.159009 4764476864 tf_logging.py:115] Calling model_fn.
I1205 12:30:04.264031 4764476864 tf_logging.py:115] Done calling model_fn.
I1205 12:30:04.278503 4764476864 tf_logging.py:115] Starting evaluation at 2018-12-05-12:30:04
I1205 12:30:04.333271 4764476864 tf_logging.py:115] Graph was finalized.
I1205 12:30:04.334044 4764476864 tf_logging.py:115] Restoring parameters from /tmp/mnist_model/model.ckpt-600
I1205 12:30:04.367141 4764476864 tf_logging.py:115] Running local_init_op.
I1205 12:30:04.374906 4764476864 tf_logging.py:115] Done running local_init_op.
I1205 12:30:08.018642 4764476864 tf_logging.py:115] Finished evaluation at 2018-12-05-12:30:08
I1205 12:30:08.018784 4764476864 tf_logging.py:115] Saving dict for global step 600: accuracy = 0.9681, global_step = 600, loss = 0.1036345
I1205 12:30:08.061439 4764476864 tf_logging.py:115] Saving 'checkpoint_path' summary for global step 600: /tmp/mnist_model/model.ckpt-600

Evaluation results:
    {'accuracy': 0.9681, 'loss': 0.1036345, 'global_step': 600}

...

Source