Stdpopsim: -g HapmapII_GRCh37 error

[agladstein]

I get an error when doing:

stdpopsim homsap -g HapmapII_GRCh37 -c chr22 -o foo.ts --model ooa_2 2 3

2019-11-29 10:43:38,515 [93420] CRITICAL root: Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1392, in connect
    server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 412, in wrap_socket
    session=session
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 850, in _create
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1108, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1045)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/agladsteinNew/dev/stdpopsim/stdpopsim_env/bin/stdpopsim", line 11, in <module>
    load_entry_point('stdpopsim==0.1.dev230+gad8481b', 'console_scripts', 'stdpopsim')()
  File "/Users/agladsteinNew/dev/stdpopsim/stdpopsim_env/lib/python3.7/site-packages/stdpopsim-0.1.dev230+gad8481b-py3.7.egg/stdpopsim/cli.py", line 479, in stdpopsim_main
    run(args)
  File "/Users/agladsteinNew/dev/stdpopsim/stdpopsim_env/lib/python3.7/site-packages/stdpopsim-0.1.dev230+gad8481b-py3.7.egg/stdpopsim/cli.py", line 470, in run
    args.runner(args)
  File "/Users/agladsteinNew/dev/stdpopsim/stdpopsim_env/lib/python3.7/site-packages/stdpopsim-0.1.dev230+gad8481b-py3.7.egg/stdpopsim/cli.py", line 373, in run_simulation
    length_multiplier=args.length_multiplier)
  File "/Users/agladsteinNew/dev/stdpopsim/stdpopsim_env/lib/python3.7/site-packages/stdpopsim-0.1.dev230+gad8481b-py3.7.egg/stdpopsim/species.py", line 135, in get_contig
    recomb_map = gm.get_chromosome_map(chrom.id)
  File "/Users/agladsteinNew/dev/stdpopsim/stdpopsim_env/lib/python3.7/site-packages/stdpopsim-0.1.dev230+gad8481b-py3.7.egg/stdpopsim/genetic_maps.py", line 144, in get_chromosome_map
    self.download()
  File "/Users/agladsteinNew/dev/stdpopsim/stdpopsim_env/lib/python3.7/site-packages/stdpopsim-0.1.dev230+gad8481b-py3.7.egg/stdpopsim/genetic_maps.py", line 110, in download
    urllib.request.urlretrieve(self.url, filename=download_file)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 247, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 563, in error
    result = self._call_chain(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 755, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1360, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1045)>

I'm on a mac.

[jeromekelleher]

Interesting. Can you try:

curl http://ftp-trace.ncbi.nih.gov/1000genomes/ftp/technical/working/20110106_recombination_hotspots/HapmapII_GRCh37_RecombinationHotspots.tar.gz > /dev/null

Please?

If that works, also try loading that url with

import requests
url = "http..."
response = requests.get(url)
response.raise_for_status()

[agladstein]

Looks like that worked

HapmapII_GRCh37_RecombinationHotspots.tar.gz > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   349  100   349    0     0   2311      0 --:--:-- --:--:-- --:--:--  2326

>>> import requests
>>> url = "http://ftp-trace.ncbi.nih.gov/1000genomes/ftp/technical/working/20110106_recombination_hotspots/HapmapII_GRCh37_RecombinationHotspots.tar.gz" 
>>> response = requests.get(url)
>>> response.raise_for_status()
>>> 

[jeromekelleher]

Ah, sorry, those are both redirects to https. Can you change the url to "https://"?

[agladstein]

that also worked. No output from response.raise_for_status()

[jeromekelleher]

Weird. So, if you use requests downloading from the URL succeeds but if you use stdpopsim it fails? Is it a reliable failure or an intermittent one?

I'd like to get to the bottom of it, as if we're having odd issues like this which don't happen when we use requests, then that's a good reason to switch over to using requests (which I'd avoided to keep the number of dependencies down).

[Chris1221]

There's a specific problem with urllib on python3.7 on Mac. Trying the same command in 3.6.8 would probably not give the same error, as the management of certificates is more intuitive (and uses the keychain). To fix it, there's a script in the python folder (though I'm not sure how you've installed python) which pops up once you've installed python, but it is easy to miss. I'm not sure if we need to alter anything on our end, but if this fixes the issue for you @agladstein, we might consider including it in an FAQ.

Active SO issue: here
Python bug tracker issue: here

For posterity, my version is found in /Applications/Python\ 3.7/Install\ Certificates.command. If you can't find it, I can just post the script here.

[jeromekelleher]

Aha, thanks @Chris1221. Do you know if this issue should be present if we were using requests? (Seems like it should, since requests uses urllib under the hood? Perhaps requests does it's own cert management or something though.)

[Chris1221]

I would also expect it to be an issue in requests as well, though it looks like @agladstein was able to successfully use requests without updating the cert management, so perhaps they do something different. FWIW, I didn't see any issues on SO where they were specifically using requests and having this problem. Not 100% sure though.

[agladstein]

Thanks @Chris1221! That worked.
I did open /Applications/Python\ 3.7/Install\ Certificates.command

So, we can either add dependencies and use requests or have that solution somewhere in the docs. If we put it in the docs, where should we put it?
Is an FAQ a good idea?

[Chris1221]

Hmm, sorry maybe that was too hasty of a suggestion. It might not be the best place to put it, but in the tutorial we can just mention under L100 in tutorial.rst that there's a known problem with urllib on mac, and link to the SO post, since this is the first place that a new user would probably encounter the issue (if following the tutorial). (My personal opinion is that, since this is kind of a known issue with a major library, it's not totally our responsibility to exhaustively cover a solution, it's of course up to you guys). Once we have an "installation" page, that might be a decent place to put a warning box saying something along the lines of

If you are on OSX and using Python version 3.7 or greater, you should install SSL certificates prior to using stdpopsim. This is a straightforward process, and a thorough guide can be found here

Do you think that would be appropriate?

[agladstein]

that's true, but I'm a little worried about installation becoming too cumbersome for every delicate environment.
For a new user, installing special things and tracking down issues can be a huge barrier and deterrent for using new things.
And we are pitching this for people without a lot of experience.

[Chris1221]

Good points! I hadn’t thought of it like that.

[jeromekelleher]

I tend to agree with @Chris1221 --- do we need to document this upstream problem @agladstein?

[agladstein]

I think it would be good to at least mention it somewhere.
I can add it into the tutorial if that seems like the best place for now.
Or would it be better to start an FAQ section?

[jeromekelleher]

I don't think the tutorial is the place, as it really shouldn't happen (it's not our fault macs are broken). At most a ''note'' block in the installation section.

Come to think of it, most people won't even manage to install stdpopsim in the first place if they have this problem, since it'll occur when they use pip. So, I'm inclined to just ignore this one.

[agladstein]

Okay. That's fine. There is an issue about now. So, if anyone hits that problem they can find this issue.