Ccxt: How to disable SSL certificate verification in Python?

[synchronizing]

I have read the docs up and down and I can't seem to find a reference for disabling SSL certificate verification. As of right now, I currently have a project where I am doing an intentional man-in-the-middle attack to switch proxies on need-bases.

client <-> Proxy Switcher (server acting as proxy)
Proxy Switcher (emulated client) <-> Exchanges

On the return of anything but a 200, the proxy switcher automatically switches proxy and try again. A man-in-the-middle is needed to verify that the HTTPS requests are coming back with the proper headers, and this part seems to work fine. However, communicating between the client and proxy switcher seems to be returning back (expected) SSL certificate issues, as proxy switcher automatically generates its own local certificate. This, though, I would like to disable.

Without adding the certificate to my local trust env, I receive the error:

(Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

Which is warranted, but would be best if could be disabled by some exchange flag. When adding the certificate to my local trust env, I receive the error

(Caused by SSLError(SSLCertVerificationError("hostname 'pro.coinbase.com' doesn't match 'Felipes-MacBook-Pro.local'")))

Which is also warranted, but a check I would much rather disable. Any help would be appreciated. My idea would be something simple as:

ex = getattr(ccxt, exchange)(
    {
        "session": cfscrape.create_scraper(),
        "enableRateLimit": False,
        "verify_ssl_certificates": False,
    }
)

Note: I have tried the verify flag with no success.

[kroitor]

@synchronizing

1. What's your version of Python?
2. Are you using the sync or the async version of the lib?

[kroitor]

@synchronizing ↓ does this help?

session = cfscrape.create_scraper()
session.verify = False
ex = getattr(ccxt, exchange)(
    {
        "session": session,
        "enableRateLimit": False,
    }
)

[synchronizing]

@synchronizing

1. What's your version of Python?
2. Are you using the sync or the async version of the lib?

Python 3.7.2 and using sync (tested with cfscrape, and without). Will give it a try on async as well, just to be sure, as I know aiohttp has fewer SSL verifications.

[kroitor]

@synchronizing let us know if the comment above does not resolve the issue for you: https://github.com/ccxt/ccxt/issues/5394#issuecomment-506918563

[synchronizing]

@synchronizing ↓ does this help?

session = cfscrape.create_scraper()
session.verify = False
ex = getattr(ccxt, exchange)(
    {
        "session": session,
        "enableRateLimit": False,
    }
)

Tested with the above with the same SSL verification error, unfortunately.

[synchronizing]

Tested with ccxt_async, and it seems to work fine -- passes through the proxy finder as well, without throwing back any issue. SSL certificated added in env (not sure if it would work without.)

However, cfscrape is still desired for coinbasepro.

[kroitor]

This page says it should have worked with the sync version as well: https://2.python-requests.org/en/master/user/advanced/#ssl-cert-verification

@synchronizing can you post a complete short snippet of your code to reproduce it, say, 10-20 lines? We need to make sure that there's no other interference, therefore we need a complete snippet, including the instantiation.

[synchronizing]

Sure thing @kroitor -- as of now the code is wrapped in an API server, so give me a few to extract the relevant lines to a separate text file for easy testing on your end.

Also: is the session.verify a boolean, or a certificate location string? From previous issues I saw here, I thought it was simply a bool flag.

[kroitor]

Also: is the session.verify a boolean, or a certificate location string? From previous issues I saw here, I thought it was simply a bool flag.

It should work either way, as a boolean or as a string-path, if the docs are correct.

[synchronizing]

Also: is the session.verify a boolean, or a certificate location string? From previous issues I saw here, I thought it was simply a bool flag.

It should work either way, as a boolean or as a string-path, if the docs are correct.

Sounds good. Give me a few to compile the problem down to a few lines of code.

[synchronizing]

Utilizing the following code:

import ccxt

exchange = ccxt.binance()
exchange.session.verify = False # With, or without line.
fetch = exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
print(fetch)

With export to a man-in-the-middle proxy:

export http_proxy=http://127.0.0.1:8888
export https_proxy=http://127.0.0.1:8888

I still receive error on the sync version of ccxt. If you would like to test it out with the man-in-the-middle, you can find it on my repo here. Just run the example/example_server.py file.

I just came to realize the reason it might have worked with aiohttp is that even with setting exchange.session.trust_env and exchange.session.trust_env_aiohttp, neither flags respect the set http_proxy and https_proxy env variable.

[synchronizing]

Update: aiohttp does not work either.

async def fetch_stuff():
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", "verify": False})
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch


print(asyncio.get_event_loop().run_until_complete(asyncio.gather(fetch_stuff())))

With implicit aiohttp_proxy set (since http_proxy and https_proxy is does not seem to be respected), the SSL checks still return error. I know for a fact aiohttp uses the flag ssl (rather than verify, like the requests library, to enable/disable SSL checks) but I assume there is no existing flag for ccxt.

[kroitor]

async def fetch_stuff():
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", "verify": False})
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch

↑ This is not a correct way of configuring it. You should add an async session and set verify = False on it, before passing it to the binance constructor. The derived exchange class does not support the verify option.

More about it here:

• https://github.com/ccxt/ccxt/blob/master/python/ccxt/async_support/base/exchange.py#L75

[kroitor]

It sync-misbehavior might be a bug in the MITM proxy: https://www.google.com/search?q=python+https+proxy+ssl+verify+requests. Looks like you're not the only person having difficulties when using proxies + ssl verify.

[synchronizing]

async def fetch_stuff():
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", "verify": False})
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch

↑ This is not a correct way of configuring it. You should add an async session and set verify = False on it, before passing it to the binance constructor. The derived exchange class does not support the verify option.

When you say this, would this be the correct format?

async def fetch_stuff():
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888"})
    exchange.session.verify = False
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch

print(asyncio.get_event_loop().run_until_complete(asyncio.gather(fetch_stuff())))

[synchronizing]

It might be a bug in the MITM proxy: https://www.google.com/search?q=python+https+proxy+ssl+verify+requests. Looks like you're not the only person having difficulties when using proxies + ssl verify.

Tell me about it -- SLL + proxies is a nightmare, as I've come to find out 😩. However, mitm is acting as the destination server for the client, so it's not actually communicating the request forward to the destination server as a normal proxy would. Instead, it initiates an emulated client that does that, and then returns the request of the emulated client back to the actual client, reading the request in the middle. The issue is the initial communication between mitm and the client with ccxt, mainly due to a self-signed certificate in the middle. Even with verify flag set to false, the error seems to persist.

[kroitor]

would this be the correct format?

Nope. This would be the correct format:

import aiohttp
import asyncio

event_loop = asyncio.get_event_loop()

async def fetch_stuff():
    connector = aiohttp.TCPConnector(ssl=False, loop=event_loop)
    session = aiohttp.ClientSession(loop=event_loop, connector=connector, trust_env=True)
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", 'session': session})
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch

print(event_loop.run_until_complete(asyncio.gather(fetch_stuff())))

Does that help?

[synchronizing]

would this be the correct format?

Nope. This would be the correct format:

import aiohttp
import asyncio

event_loop = asyncio.get_event_loop()

async def fetch_stuff():
    trust_environment_variables = False
    connector = aiohttp.TCPConnector(ssl=False, loop=event_loop)
    session = aiohttp.ClientSession(loop=event_loop, connector=connector, trust_env=trust_environment_variables)
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", 'session': session})
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch

print(event_loop.run_until_complete(asyncio.gather(fetch_stuff())))

Does that help?

Yes, it did! I received a request in the mitm which got processed but didn't get properly returned (which is fault from mitm). Much appreciated man! Let me give it a try with sync ccxt.

[kroitor]

@synchronizing something like the above should work for the sync version as well. Just need to dig the internets on the proper way to configure it. However, this is beyond CCXT, unfortunately.

[synchronizing]

@synchronizing something like the above should work for the sync version as well. Just need to dig the internets on the proper way to configure it. However, this is beyond CCXT, unfortunately.

Perfectly understandable -- all I was hoping for was a proper CONNECT and GET with mitm, as from there on out I knew ccxt had completed the proper ssl steps.

[kroitor]

@synchronizing are you ok if we close this for now?

[synchronizing]

Forgive me for the repetitive questioning: but with cfscraper and standard sync functionality, can we set a session? While I have you on the wire.

If this is beyond the project due to cfscraper, no worries. The help above has already been fantastical and I truly do appreciate it.

Also, completely irrelevant: Don't fall for AdBlocker Pro -- they were bought out by some company a while ago, and still display advertisements. Open source solution is uBlock Origin which very few people seem to be aware is out there and is also a superb blocker in comparison (including YouTube advertisement, thank the Lord.)

[kroitor]

Forgive me for the repetitive questioning: but with cfscraper and standard sync functionality, can we set a session?

Yes, that should be possible. However, there may be bugs outside of CCXT:

• https://github.com/kennethreitz/requests/issues/3829

I will add the exchange.verify option for the sync version shortly and will need your help to test it against your proxy.

[synchronizing]

Forgive me for the repetitive questioning: but with cfscraper and standard sync functionality, can we set a session?

Yes, that should be possible. However, there may be bugs outside of CCXT:

• kennethreitz/requests#3829

I will add the exchange.verify option for the sync version shortly and will need your help to test it against your proxy.

Sounds like a plan! Again, I appreciate the help. Feel free to close this issue and re-open when I am needed for testing.

[kroitor]

@synchronizing should not take too long, will let you know when it's there, 30-60 minutes.

[synchronizing]

@synchronizing should not take too long, will let you know when it's there, 30-60 minutes.

Sounds good with me, I'll be here.

[kroitor]

Ok, here's what we need to do:

1. Wait 15 minutes for the build 1.18.844 to arrive
2. Update your CCXT version (make sure it's updated properly)
3. Try the following code:

# sync 

import ccxt
exchange = ccxt.binance({'verify': False})
fetch = exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
print(fetch)

or

# async

async def fetch_stuff():
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", "verify": False})
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch

[kroitor]

We will be happy if you let us know whether it did solve the issue for you or not.

[synchronizing]

We will be happy if you let us know whether it did solve the issue for you or not.

Ofc, will be happy to help. For the async, intended feature for assigning verify directly on binance constructor?

[kroitor]

For the async, intended feature for assigning verify directly on binance constructor?

Yep, I've added the verify constructor-option to both the sync and the async version. You can also set it afterwards after creating the exchange instance:

exchange = ccxt.binance()
exchange.verify = False

[synchronizing]

For the async, intended feature for assigning verify directly on binance constructor?

Yep, I've added the verify option to both the sync and the async version.

Beautiful. Will give it a shot right now.

[kroitor]

@synchronizing found a couple of minor issues along the way, added the fixes, so, it will arrive shortly (5-10 minutes). Standing by.

[synchronizing]

@synchronizing found a couple of minor issues along the way, added the fixes, so, it will arrive shortly (5-10 minutes). Standing by.

What version should I be on the look out for?

[kroitor]

@synchronizing 1.18.844 (the one upcoming).

[kroitor]

Ok, it has arrived, looking forward to hearing from you.

[synchronizing]

sync version is working as expected on 1.18.844:

import ccxt as ccxt

exchange = ccxt.binance(
    {
        "proxies": {"http": "http://127.0.0.1:8888", "https": "http://127.0.0.1:8888"},
        "verify": False,
    }
)
fetch = exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
print(fetch)

async version does not work with the following:

import ccxt.async_support as ccxt
import asyncio

async def fetch_stuff():
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", "verify": False})
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch

print(asyncio.get_event_loop().run_until_complete(asyncio.gather(fetch_stuff())))

SSL error is thrown for the above code. However, it does work when loading session as shown previously here. Assumingly, verify might not be setting the ssl flag in the TCPConnector to False internally within ccxt.

[kroitor]

Assumingly, verify might not be setting the ssl flag in the TCPConnector to False internally within ccxt.

Ok, I've added one more edit to it, let us know if 1.18.845 does not resolve the issue on the async side. Closing this for now. Feel free to reopen it or just ask further questions, if any. We will be happy if you report back anyways.

[synchronizing]

Sounds good -- will give it a try, thank you again.

[synchronizing]

async is still not working with 1.18.845 with "verify" : False set.

import ccxt.async_support as ccxt
import asyncio

async def fetch_stuff():
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", "verify": False})
    print(exchange.verify)
    fetch = await exchange.fetch_ohlcv("ETH/BTC", "1m", 1514764800000)
    await exchange.close()
    return fetch

print(asyncio.get_event_loop().run_until_complete(asyncio.gather(fetch_stuff())))

[kroitor]

@synchronizing doesn't work with verify: False, but works with https://github.com/ccxt/ccxt/issues/5394#issuecomment-506921151 ?

[synchronizing]

@synchronizing doesn't work with verify: False, but works with #5394 (comment) ?

Correct, unfortunately.

[kroitor]

@synchronizing are you sure about that? This is strange, because they should be technically equal...

[kroitor]

@synchronizing ah, nvm, found another bug there with the ordering of the calls, will fix it in a moment.

[synchronizing]

@synchronizing are you sure about that? This is strange, because they should be technically equal...

They seem to be different on further testing:

import ccxt.async_support as ccxt
import asyncio
import aiohttp

async def fetch_stuff():
    connector = aiohttp.TCPConnector(ssl=False, loop=asyncio.get_event_loop())
    exchange = ccxt.binance({"aiohttp_proxy": "http://127.0.0.1:8888", "verify": False})
    print(exchange.session._connector._ssl)
    print(connector._ssl)
    await exchange.close()

asyncio.get_event_loop().run_until_complete(fetch_stuff())

Outputs:

<ssl.SSLContext object at 0x10f55c480>
False

@synchronizing ah, nvm, found another bug there with the ordering of the calls, will fix it in a moment.

Awesome, lmk!

[kroitor]

@synchronizing ok, let's try again with 1.18.846. Unfortunately, I can't really test it on my side atm, your help with debugging it is very much appreciated! The autobuild takes 10-15 minutes.

[synchronizing]

@synchronizing ok, let's try again with 1.18.846. Unfortunately, I can't really test it on my side atm, your help with debugging it is very much appreciated! The autobuild takes 10-15 minutes.

My pleasure man -- I appreciate all the work on your end. I'll be on the lookout for the new release to give it a try.

[kroitor]

https://travis-ci.org/ccxt/ccxt/builds

[synchronizing]

https://travis-ci.org/ccxt/ccxt/builds

Working as expected! Thank you again, cheers.

[brandsimon]

@kroitor
Should we log a warning if SSL Certificates are disabled or overwritten, since this can result in stolen keys? MITM Attack and there are some exchanges using login-data like keys (for example dx.exchange).

[synchronizing]

@kroitor
Should we log a warning if SSL Certificates are disabled or overwritten, since this can result in stolen keys? MITM Attack and there are some exchanges using login-data like keys (for example dx.exchange).

Let me just add:

• aiohttp (ccxt.async_support) does not throw out SSL certificate warning.
• requests (ccxt) does throw SSL certificate warning through urllib3.

[kroitor]

@brandsimon in this particular case, the MITM attack is done deliberately by the owner. But in general, yes, I think it would be great to have a warning so that we keep people aware! )

@synchronizing thx for the hints!