Requests: OpenSSL.SSL.SysCallError: (-1, 'Unexpected EOF')

[fygul]

I run following code to retrieve a page on www.cnyes.com

import requests
url = 'https://www.cnyes.com/global/IndexImmediateQuotedPrice/'
r = requests.get(url)

Following errors occur

Actual Result

C:\ProgramData\Anaconda3\envs\p36python.exe D:/projects/DailyReport/scripts/test/test_requests.py
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\contrib\pyopenssl.py", line 441, in wrap_socket
cnx.do_handshake()
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\OpenSSL\SSL.py", line 1806, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\OpenSSL\SSL.py", line 1539, in _raise_ssl_error
raise SysCallError(-1, "Unexpected EOF")
OpenSSL.SSL.SysCallError: (-1, 'Unexpected EOF')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen
chunked=chunked)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\connectionpool.py", line 850, in _validate_conn
conn.connect()
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\connection.py", line 326, in connect
ssl_context=context)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\util\ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\contrib\pyopenssl.py", line 448, in wrap_socket
raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\requests\adapters.py", line 440, in send
timeout=timeout
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\util\retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.cnyes.com', port=443): Max retries exceeded with url: /global/IndexImmediateQuotedPrice/ (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')",),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:/projects/DailyReport/scripts/test/test_requests.py", line 9, in
r = requests.get(url)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\requests\api.py", line 72, in get
return request('get', url, params=params, *kwargs)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\requests\api.py", line 58, in request
return session.request(method=method, url=url, *kwargs)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\requests\sessions.py", line 508, in request
resp = self.send(prep, *send_kwargs)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\requests\sessions.py", line 618, in send
r = adapter.send(request, *kwargs)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\requests\adapters.py", line 506, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.cnyes.com', port=443): Max retries exceeded with url: /global/IndexImmediateQuotedPrice/ (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')",),))

Some Information on Security Panel of Google Chrome DevTools

I use DevTools to Inspect www.cnyes.com:

Connection
Protocol TLS 1.0
Key exchange RSA
Cipher 3DES_EDE_CBC with HMAC-SHA1

Refer to the red circle on attach image.

Firefox Developer Tools shows Cipher suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.2.2"
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.5"
  },
  "platform": {
    "release": "8.1",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "1010008f",
    "version": "17.5.0"
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "100020bf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": true
}

Windows 8.1
Python 3.6.5
pip install request[security]

pip list
Package Version

---

asn1crypto 0.24.0
certifi 2018.4.16
cffi 1.11.5
chardet 3.0.4
cryptography 2.2.2
idna 2.6
pip 10.0.1
pycparser 2.18
pyOpenSSL 17.5.0
requests 2.18.4
setuptools 39.1.0
six 1.11.0
urllib3 1.22
wheel 0.31.0
wincertstore 0.2

[fygul]

I try to add DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA to CIPHERS, but it doesn't work.

import requests
from requests.adapters import HTTPAdapter
from urllib3.poolmanager import PoolManager
from urllib3.util.ssl_ import create_urllib3_context

CIPHERS = (
    'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
    'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:'
    '!eNULL:!MD5:DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA'
)


class DESAdapter(HTTPAdapter):
    def init_poolmanager(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=CIPHERS)
        kwargs['ssl_context'] = context
        self.poolmanager = PoolManager(
            *args, ** kwargs)


s = requests.Session()
s.mount('https://www.cnyes.com', DESAdapter())
res = s.get('https://www.cnyes.com/global/IndexImmediateQuotedPrice/')

[wty1143]

Have you found the solution ? I am facing the same problem with you.
Thanks!

[fygul]

@wty1143
I use Selenium/PhantomJs instead, temporarily.

[wty1143]

@fygul Haha, I am using selenium too, let's keep in touch to see if there's an update.

[nateprewitt]

Hi @fygul, have you tried adding TLS_RSA_WITH_3DES_EDE_CBC_SHA to your cipher list and uninstalling PyOpenSSL as suggested in #3774? It also looks like you may not have tried the suggested cipher suite at the top of that issue. Would you mind checking that as well?

3DES ciphers are not secure and have been removed from the default Requests cipher suite. It’s highly advisable to contact the server administrator about this to get the service fixed.

[fygul]

@nateprewitt

I tried to uninstall pyOpenSSL, and added more cipher suites, but none of them helped.

TLS_RSA_WITH_3DES_EDE_CBC_SHA ==> DES-CBC3-SHA. I already added it.

Several cipher suited was found in SSL Report for www.cnyes.com

I searched their names on the following pages:

• https://wiki.mozilla.org/Security/Server_Side_TLS
• https://www.openssl.org/docs/man1.1.0/apps/ciphers.html

Following is my packages, code and error message:

pip freeze
asn1crypto==0.24.0
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
cryptography==2.2.2
idna==2.6
pycparser==2.18
requests==2.18.4
six==1.11.0
urllib3==1.22
wincertstore==0.2

import ssl
import requests
from requests.adapters import HTTPAdapter
from urllib3.poolmanager import PoolManager
from urllib3.util.ssl_ import create_urllib3_context

CIPHERS = (
    'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
    'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:'
    '!eNULL:!MD5:DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:'
    'RC4-MD5:RC4-SHA'
)


class DESAdapter(HTTPAdapter):
    # def init_poolmanager(self, *args, **kwargs):
    def init_poolmanager(self, connections, maxsize, block=False, *args, **kwargs):
        context = create_urllib3_context(ciphers=CIPHERS)
        kwargs['ssl_context'] = context
        # self.poolmanager = PoolManager(*args, ** kwargs)
        # self.poolmanager = PoolManager(ssl_version=ssl.PROTOCOL_SSLv3, *args, ** kwargs)
        self.poolmanager = PoolManager(
            num_pools=connections, maxsize=maxsize,
            # block=block, ssl_version=ssl.PROTOCOL_SSLv3, *args, **kwargs)
            block=block, *args, **kwargs)


s = requests.Session()
s.mount('https://www.cnyes.com', DESAdapter())
res = s.get('https://www.cnyes.com/global/IndexImmediateQuotedPrice/')

Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen
chunked=chunked)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\connectionpool.py", line 850, in _validate_conn
conn.connect()
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\connection.py", line 326, in connect
ssl_context=context)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\util\ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\ProgramData\Anaconda3\envs\p36\lib\ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "C:\ProgramData\Anaconda3\envs\p36\lib\ssl.py", line 814, in __init__
self.do_handshake()
File "C:\ProgramData\Anaconda3\envs\p36\lib\ssl.py", line 1068, in do_handshake
self._sslobj.do_handshake()
File "C:\ProgramData\Anaconda3\envs\p36\lib\ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:833)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\requests\adapters.py", line 440, in send
timeout=timeout
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\urllib3\util\retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.cnyes.com', port=443): Max retries exceeded with url: /global/IndexImmediateQuotedPrice/ (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:833)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:/projects/DailyReport/scripts/test/test_requests_3.py", line 50, in
res = s.get('https://www.cnyes.com/global/IndexImmediateQuotedPrice/')
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\requests\sessions.py", line 521, in get
return self.request('GET', url, *kwargs)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\requests\sessions.py", line 508, in request
resp = self.send(prep, *send_kwargs)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\requests\sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "C:\ProgramData\Anaconda3\envs\p36\lib\site-packages\requests\adapters.py", line 506, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.cnyes.com', port=443): Max retries exceeded with url: /global/IndexImmediateQuotedPrice/ (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:833)'),))

---

I'll try to contact cnyes.com to improve their web site.

[wty1143]

@nateprewitt
I also have tried the method you mentioned, but none of them fixed this issue.
However, when I tested the same code with my old version requests (2.0.1 I think) with old version urllib3, the issue was gone.

So I am wondering what's the critical difference between these two versions and why the manual cipher solution did not work well in this case.

Again, thanks for your help.

[nateprewitt]

@fygul, so this server is in a pretty sad state (ssl labs report). It's primary cipher suites are RC4-MD5 and RC4-SHA which aren't secure and it supports 3DES_EDE_CBC with HMAC-SHA1 with TLS1.0 to appease browsers. If you change the cipher string to only CIPHERS = 'DES-CBC3-SHA' it will connect, so there's something else in the cipher string that the server is hanging on.

Since this isn't supported with Requests default cipher suite and there's a path forward for debugging, I'm going to close this out.

@wty1143, I'd suggest taking a look at your cipher string and seeing if this fixes your issue. If not, you can either use your browsers developer tools or openssl s_client -connect yourdomainhere.com:443 to determine what cipher suite is being negotiated.