Tensorflow: Mac + Python 3.6.1: Attempting to download mnist data results in CERTIFICATE_VERIFY_FAILED error

[paulcwatts]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Mac OS X 10.12.5
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): v1.2.0-rc2-21-g12f033d 1.2.0
• Bazel version (if compiling from source): n/a
• CUDA/cuDNN version: none
• GPU model and memory: n/a
• Exact command to reproduce:

$ python3 --version
Python 3.6.1

$ python3 -m virtualenv venv
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.6'
New python executable in .../venv/bin/python3
Also creating executable in .../venv/bin/python
Installing setuptools, pip, wheel...done.

$ source venv/bin/activate
$ pip install tensorflow
Collecting tensorflow
  Using cached tensorflow-1.2.0-cp36-cp36m-macosx_10_11_x86_64.whl
  ....

$ python
Python 3.6.1 (v3.6.1:69c0db5050, Mar 21 2017, 01:21:04) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from tensorflow.examples.tutorials.mnist import input_data
>>> mnist = input_data.read_data_sets("/tmp/data/")
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 401, in wrap_socket
    _context=self, _session=session)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 808, in __init__
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../venv/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py", line 235, in read_data_sets
    SOURCE_URL + TRAIN_IMAGES)
  File ".../venv/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py", line 208, in maybe_download
    temp_file_name, _ = urlretrieve_with_retry(source_url)
  File ".../venv/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py", line 165, in wrapped_fn
    return fn(*args, **kwargs)
  File ".../venv/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py", line 190, in urlretrieve_with_retry
    return urllib.request.urlretrieve(url, filename)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 248, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)>

This doesn't reproduce with TensorFlow 1.1.

[aselle]

Could you try with python2 (default system install of py)? Also, how did you install with brew or macports (we use brew for our testing so that's more likely to work).

[paulcwatts]

I installed Python 3.6 using the dmg installer from the Python website (not through brew or macports).

It seems to work with Python 2.7.10. I had thought that in Python 2.x urlretrieve didn't validate the server certificate, but apparently it does as of 2.7.9: https://docs.python.org/2/library/urllib.html

[aselle]

It works me on python 3.4 on Linux. I'd try installing python3.6 using brew. It could be the python3 distrib you are using has out of data certificates.

[paulcwatts]

This might have also been because I recently switched to a new machine and it's still only partially installed :)

For future reference: if you want to use the Python dmg installer, you also have to read Python 3's ReadMe and run the /Applications/Python 3.6/Install Certificates.command bash script to install newer certs.

Thanks for the quick response, @aselle!

[EthanAI]

If you want to just copy paste into Terminal:
/Applications/Python\ 3.6/Install\ Certificates.command

[brightbytes-dude]

I'm getting exactly this same issue on MacOSX 10.11.6

I first followed exactly the steps here for the virtualenv setup for Python 3.n: https://www.tensorflow.org/install/install_mac

Then, following the first 2 steps in this tutorial https://www.tensorflow.org/get_started/mnist/beginners I see the following; any suggestions for working-around this (e.g. could I download this from somewhere else?) would be greatly appreciated.

(tensorflow) ~/tensorflow  $ python
Python 3.6.2 (v3.6.2:5fd33b5926, Jul 16 2017, 20:11:06) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from tensorflow.examples.tutorials.mnist import input_data
>>> mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 401, in wrap_socket
    _context=self, _session=session)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 808, in __init__
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:748)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/aaron/tensorflow/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py", line 235, in read_data_sets
    SOURCE_URL + TRAIN_IMAGES)
  File "/Users/aaron/tensorflow/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py", line 208, in maybe_download
    temp_file_name, _ = urlretrieve_with_retry(source_url)
  File "/Users/aaron/tensorflow/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py", line 165, in wrapped_fn
    return fn(*args, **kwargs)
  File "/Users/aaron/tensorflow/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py", line 190, in urlretrieve_with_retry
    return urllib.request.urlretrieve(url, filename)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 248, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:748)>

[paulcwatts]

@brightbytes-dude Did you run the command listed here: https://github.com/tensorflow/tensorflow/issues/10779#issuecomment-309134512

/Applications/Python\ 3.6/Install\ Certificates.command

[brightbytes-dude]

That did the trick, thanks!! (I stupidly assumed that was only relevant for the dmg installation.)

[martin-gorner]

Could you please change the MNIST download URL in contrib/learn/python/learn/datasets/mnist.py to not use https ? That will help people who are having this problem. In tried in a browser and the http download link works.

line to change:
SOURCE_URL = 'https://storage.googleapis.com/cvdf-datasets/mnist/'
new line:
SOURCE_URL = 'https://storage.googleapis.com/cvdf-datasets/mnist/'

[janusfxxx]

Hello,
I'm new to python and tensorflow. Trying the MNIST example and receive the about the same error messages as paulcwatts (above)

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
Traceback (most recent call last):
File "C:UsersAndreasDocumentsPythonliburllibrequest.py", line 1318, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "C:UsersAndreasDocumentsPythonlibhttpclient.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:UsersAndreasDocumentsPythonlibhttpclient.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:UsersAndreasDocumentsPythonlibhttpclient.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:UsersAndreasDocumentsPythonlibhttpclient.py", line 1026, in _send_output
self.send(msg)
File "C:UsersAndreasDocumentsPythonlibhttpclient.py", line 964, in send
self.connect()
File "C:UsersAndreasDocumentsPythonlibhttpclient.py", line 1400, in connect
server_hostname=server_hostname)
File "C:UsersAndreasDocumentsPythonlibssl.py", line 401, in wrap_socket
_context=self, _session=session)
File "C:UsersAndreasDocumentsPythonlibssl.py", line 808, in __init__
self.do_handshake()
File "C:UsersAndreasDocumentsPythonlibssl.py", line 1061, in do_handshake
self._sslobj.do_handshake()
File "C:UsersAndreasDocumentsPythonlibssl.py", line 683, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:748)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
File "C:UsersAndreasDocumentsPythonlibsite-packagestensorflowcontriblearnpythonlearndatasetsmnist.py", line 240, in read_data_sets
source_url + TRAIN_IMAGES)
File "C:UsersAndreasDocumentsPythonlibsite-packagestensorflowcontriblearnpythonlearndatasetsbase.py", line 208, in maybe_download
temp_file_name, _ = urlretrieve_with_retry(source_url)
File "C:UsersAndreasDocumentsPythonlibsite-packagestensorflowcontriblearnpythonlearndatasetsbase.py", line 165, in wrapped_fn
return fn(args, *kwargs)
File "C:UsersAndreasDocumentsPythonlibsite-packagestensorflowcontriblearnpythonlearndatasetsbase.py", line 190, in urlretrieve_with_retry
return urllib.request.urlretrieve(url, filename)
File "C:UsersAndreasDocumentsPythonliburllibrequest.py", line 248, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "C:UsersAndreasDocumentsPythonliburllibrequest.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "C:UsersAndreasDocumentsPythonliburllibrequest.py", line 526, in open
response = self._open(req, data)
File "C:UsersAndreasDocumentsPythonliburllibrequest.py", line 544, in _open
'_open', req)
File "C:UsersAndreasDocumentsPythonliburllibrequest.py", line 504, in _call_chain
result = func(*args)
File "C:UsersAndreasDocumentsPythonliburllibrequest.py", line 1361, in https_open
context=self._context, check_hostname=self._check_hostname)
File "C:UsersAndreasDocumentsPythonliburllibrequest.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError:

The main difference is that I'm on WINDOWS 10 and I cannot find the certificates command.
Sorry for maybe asking "stupid questions" but those errors do not "go away"

[themikaelson]

I have same problem with ssl and this line fix it,
/Applications/Python 3.6/Install Certificates.command
thanks @EthanAI @aselle @paulcwatts

[mrcity]

If there's anyone stumbling across this page but looking for the answer to permit Tensorflow Hub in Ubuntu, I made it work like this:

# Directory where user certs can be stored
cd /usr/local/share/ca-certificates
# Below may require elevated permissions, such as sudo su
pip3 install --upgrade certifi  # Installs a certificate file
# Move the installed cert to the place Ubuntu expects it
cp /home/ubuntu/.local/lib/python3.5/site-packages/certifi/cacert.pem .
# Convert it into the format Ubuntu expects
openssl x509 -in cacert.pem -inform pem -outform der -out cacert.crt
# Actually load the new cert into the OS
update-ca-certificates

On top of this, some magical combination of configuration of my corporate proxy, restarting the Jupyter notebook kernel, and running TF Hub from a script on the terminal rather than from a notebook eventually caused TF Hub to work both on the terminal and in the notebook.

[victor61zhong]

running that command also failed.
./Install Certificates.command
-- pip install --upgrade certifi
Collecting certifi
Cache entry deserialization failed, entry ignored
Could not fetch URL https://pypi.python.org/simple/certifi/: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777) - skipping
Could not find a version that satisfies the requirement certifi (from versions: )
No matching distribution found for certifi
Traceback (most recent call last):
File "", line 44, in
File "", line 25, in main
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 291, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6', '-E', '-s', '-m', 'pip', 'install', '--upgrade', 'certifi']' returned non-zero exit status 1.

[iridiumsoft]

on MacOS /Applications/Python\ 3.6/Install\ Certificates.command command works for me

[litcoderr]

/Applications/Python\ 3.6/Install\ Certificates.command did the magic on MacOSX High Sierra. Thx

[damienstanton]

What is the solution for those using a virtual environment?

Update: It looks like running that same system command pointed out above 👆 does the trick when in an active venv as well. Good to know!

[hungnmai]

If you want to just copy paste into Terminal:
/Applications/Python\ 3.6/Install\ Certificates.command

thank u!

[AnimatorJoe]

This might have also been because I recently switched to a new machine and it's still only partially installed :)

For future reference: if you want to use the Python dmg installer, you also have to read Python 3's ReadMe and run the /Applications/Python 3.6/Install Certificates.command bash script to install newer certs.

Thanks for the quick response, @aselle!

Thanks! It worked! :D

[PassionZale]

Thanks

[biranchi2018]

Could you please change the MNIST download URL in contrib/learn/python/learn/datasets/mnist.py to not use https ? That will help people who are having this problem. In tried in a browser and the http download link works.

line to change:
SOURCE_URL = 'https://storage.googleapis.com/cvdf-datasets/mnist/'
new line:
SOURCE_URL = 'https://storage.googleapis.com/cvdf-datasets/mnist/'

I see both the urls are same.

[chanduthedev]

Could you please change the MNIST download URL in contrib/learn/python/learn/datasets/mnist.py to not use https ? That will help people who are having this problem. In tried in a browser and the http download link works.

line to change:
SOURCE_URL = 'https://storage.googleapis.com/cvdf-datasets/mnist/'
new line:
SOURCE_URL = 'https://storage.googleapis.com/cvdf-datasets/mnist/'

It worked for me.

[WeiYangBin]

We only need input at the Terminal
open "/Applications/Python 3.6/Install Certificates.command"

[kshahat]

Could you please change the MNIST download URL in contrib/learn/python/learn/datasets/mnist.py to not use https ? That will help people who are having this problem. In tried in a browser and the http download link works.
line to change:
SOURCE_URL = 'https://storage.googleapis.com/cvdf-datasets/mnist/'
new line:
SOURCE_URL = 'https://storage.googleapis.com/cvdf-datasets/mnist/'

I see both the urls are same.

change 'https://...' to 'http://...'

[UCLA123]

/Applications/Python 3.7/Install Certificates.command
Work for me

[neonbulb12]

How to do it in Manjaro Linux? I have Python3.7 tensorflow.

[maksatkanat]

If you want to just copy paste into Terminal:
/Applications/Python\ 3.6/Install\ Certificates.command

Thanks man! It works for me!
I just run the code in jupyter notebook:
!/Applications/Python\ 3.6/Install\ Certificates.command

[CRLannister]

Incase of linux...

go to .local/python3.X/lib/python3.6/site-packages/keras/utils/data_utils.py

and below import statements add these----

```import requests
requests.packages.urllib3.disable_warnings()
import ssl

try:
_create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
# Legacy Python that doesn't verify HTTPS certificates by default
pass
else:
# Handle target environment that doesn't support HTTPS verification
ssl._create_default_https_context = _create_unverified_https_context
```

now try new instance of python and ..hopefully it works :smile_cat:

[Stewiejobs90]

This might have also been because I recently switched to a new machine and it's still only partially installed :)

For future reference: if you want to use the Python dmg installer, you also have to read Python 3's ReadMe and run the /Applications/Python 3.6/Install Certificates.command bash script to install newer certs.

Thanks for the quick response, @aselle!

You saved my life with this comment! <3

[dhiren-hamal]

Could you please change the MNIST download URL in contrib/learn/python/learn/datasets/mnist.py to not use https ? That will help people who are having this problem. In tried in a browser and the http download link works.

line to change:
SOURCE_URL = 'https://storage.googleapis.com/cvdf-datasets/mnist/'
new line:
SOURCE_URL = 'https://storage.googleapis.com/cvdf-datasets/mnist/'

This works for me on ubuntu
by changing ~/miniconda/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/keras/datasets/mnist.py

New Line
origin_folder = 'http://storage.googleapis.com/tensorflow/tf-keras-datasets/'

[ghost]

Here's what worked for me:

1. Download the mnist dataset file by hand (the url is given in the error message)
2. Copy that file into ~/.keras/datasets/

That's all. The keras download utility looks in that folder for cached data before going over the network.

(Python 3.7.3, tensorflow 2.1.0)

[ashu4199]

/Applications/Python 3.6/Install Certificates.command does this work for virtual environment as well?