Requests: Python 3.7.6: urllib3.exceptions.ProxySchemeUnknown

Created on 23 Dec 2019  路  10Comments  路  Source: psf/requests

Proxy setup without scheme is not working on Python 3.7.6

While it works on Python 3.7.5:

Python 3.7.5 (default, Nov 23 2019, 05:59:34) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("http://httpbin.org/get", proxies={"http": "103.250.166.4:6666"})
<Response [200]>

Is not under Python 3.7.6

Python 3.7.6 (default, Dec 21 2019, 08:28:11) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("http://httpbin.org/get", proxies={"http": "103.250.166.4:6666"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 412, in send
    conn = self.get_connection(request.url, proxies)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 309, in get_connection
    proxy_manager = self.proxy_manager_for(proxy)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 199, in proxy_manager_for
    **proxy_kwargs)
  File "/usr/local/lib/python3.7/site-packages/urllib3/poolmanager.py", line 470, in proxy_from_url
    return ProxyManager(proxy_url=url, **kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/poolmanager.py", line 420, in __init__
    raise ProxySchemeUnknown(proxy.scheme)
urllib3.exceptions.ProxySchemeUnknown: Not supported proxy scheme None

It seems that some changes on Python urllib.parse.urlparse brake requests.utils.prepend_scheme_if_needed

>>> utils.prepend_scheme_if_needed('103.250.166.4:6666', 'http')
'103.250.166.4://6666'

So it looks like this utils has to update to match Python changes (I think the change that produces this is https://github.com/python/cpython/commit/5a88d50ff013a64fbdb25b877c87644a9034c969). In both cases I'm using requests 2.22.0.

Reporting here since urllib.parse.urlparse behaviour is wrong in both Python versions and I think the problem is requests.utils.prepend_scheme_if_needed problem:

Python 3.7.5

>>> from urllib import parse
>>> parse.urlparse("103.250.166.4:6666")
ParseResult(scheme='', netloc='', path='103.250.166.4:6666', params='', query='', fragment='')

Python 3.7.6

>>> from urllib import parse
>>> parse.urlparse("103.250.166.4:6666")
ParseResult(scheme='103.250.166.4', netloc='', path='6666', params='', query='', fragment='')
picture rafaduran

Most helpful comment

yes, this fixes the issue for 3.7.6 or 3.8.x
either by --proxy http://<ip>:<port>
or in (Windows') %APPDATA%pippip.ini, [global] section:
http://<ip>:<port>

picture cbpr on 6 Jan 2020
👍2

All 10 comments

Thanks for reporting this issue and doing investigation!

You can fix this by using http://<ip>:<port> instead of just an IP.

sethmlarson picture sethmlarson on 23 Dec 2019
👍2

you can solve the problem??
i have the same issue with python 3.8

Proxy setup without scheme is not working on Python 3.7.6

While it works on Python 3.7.5:

Python 3.7.5 (default, Nov 23 2019, 05:59:34) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("http://httpbin.org/get", proxies={"http": "103.250.166.4:6666"})
<Response [200]>

Is not under Python 3.7.6

Python 3.7.6 (default, Dec 21 2019, 08:28:11) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("http://httpbin.org/get", proxies={"http": "103.250.166.4:6666"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 412, in send
    conn = self.get_connection(request.url, proxies)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 309, in get_connection
    proxy_manager = self.proxy_manager_for(proxy)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 199, in proxy_manager_for
    **proxy_kwargs)
  File "/usr/local/lib/python3.7/site-packages/urllib3/poolmanager.py", line 470, in proxy_from_url
    return ProxyManager(proxy_url=url, **kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/poolmanager.py", line 420, in __init__
    raise ProxySchemeUnknown(proxy.scheme)
urllib3.exceptions.ProxySchemeUnknown: Not supported proxy scheme None

It seems that some changes on Python urllib.parse.urlparse brake requests.utils.prepend_scheme_if_needed

>>> utils.prepend_scheme_if_needed('103.250.166.4:6666', 'http')
'103.250.166.4://6666'

So it looks like this utils has to update to match Python changes (I think the change that produces this is python/cpython@5a88d50). In both cases I'm using requests 2.22.0.

Reporting here since urllib.parse.urlparse behaviour is wrong in both Python versions and I think the problem is requests.utils.prepend_scheme_if_needed problem:

Python 3.7.5

>>> from urllib import parse
>>> parse.urlparse("103.250.166.4:6666")
ParseResult(scheme='', netloc='', path='103.250.166.4:6666', params='', query='', fragment='')

Python 3.7.6

>>> from urllib import parse
>>> parse.urlparse("103.250.166.4:6666")
ParseResult(scheme='103.250.166.4', netloc='', path='6666', params='', query='', fragment='')

you can solve the problem??
i have same issue with python 3.8

Crackde picture Crackde on 29 Dec 2019
👍1

@Crackde:

You can fix this by using http://<ip>:<port> instead of just an IP.

sethmlarson picture sethmlarson on 29 Dec 2019

yes, this fixes the issue for 3.7.6 or 3.8.x
either by --proxy http://<ip>:<port>
or in (Windows') %APPDATA%pippip.ini, [global] section:
http://<ip>:<port>

cbpr picture cbpr on 6 Jan 2020
👍2

I observed the same error. As already mentioned above, i also can solve the problem by adding a leading http:// to the --proxy command line argument or in %APPDATA%\pip\pip.ini configuration file.

System information:
OS: Windows 10 Enterprise (Build 17134)
Python: 3.8.1 (x86)
pip: 19.3.1
joh-ku picture joh-ku on 7 Jan 2020

It looks like Seth has answered this and it's working as expected now. Going to resolve, thanks everyone.

nateprewitt picture nateprewitt on 1 Feb 2020

The fix is not applicable if one is using another library and has no direct control over the proxy urls.

I think the solution should be to fix the way utils.prepend_scheme_if_neededmethod calls the urllib3 helpers. as it currently is not working as intended (it should at least not produce invalid urls):

utils.prepend_scheme_if_needed('103.250.166.4:6666', 'http')
'103.250.166.4://6666'

The issues seems that urlsplit searches for the first ":" and assumes everything to the right to be the scheme - unless it contains an illegal character(a non-alphanumeric character except +-.).
This also overrides the default scheme that the prepend_scheme_if_needed passes to urlparse, which passes it to urlsplit.

103.250.166.4:6666--> scheme = 103.250.166.4, path=6666

prefixing it with "//" - as is expected by urlparse - works:

//103.250.166.4:6666 --> scheme = None (default is used) , netloc=103.250.166.4:6666

Maybe prepend_scheme can first check for the presence of a scheme and then call urlparse with the prepended "//"?
Or is this an issue with urllib?

awfuluncrn picture awfuluncrn on 18 Feb 2020

When I try to update by conda, I had received this following report

Traceback (most recent call last):
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\exceptions.py", line 1079, in __call__
return func(args, *kwargs)
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\cli\main.py", line 84, in _main
exit_code = do_call(args, p)
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\cli\conda_argparse.py", line 82, in do_call
return getattr(module, func_name)(args, parser)
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\cli\main_update.py", line 20, in execute
install(args, parser, 'update')
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\cli\install.py", line 265, in install
should_retry_solve=(_should_retry_unfrozen or repodata_fn != repodata_fns[-1]),
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\solve.py", line 117, in solve_for_transaction
should_retry_solve)
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\solve.py", line 158, in solve_for_diff
force_remove, should_retry_solve)
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\solve.py", line 262, in solve_final_state
ssc = self._collect_all_metadata(ssc)
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\common\io.py", line 88, in decorated
return f(args, *kwds)
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\solve.py", line 415, in _collect_all_metadata
index, r = self._prepare(prepared_specs)
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\solve.py", line 1011, in _prepare
self.subdirs, prepared_specs, self._repodata_fn)
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\index.py", line 228, in get_reduced_index
repodata_fn=repodata_fn)
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\subdir_data.py", line 105, in query_all
result = tuple(concat(executor.map(subdir_query, channel_urls)))
File "C:\Users\Shungo\anaconda3\lib\concurrent\futures_base.py", line 598, in result_iterator
yield fs.pop().result()
File "C:\Users\Shungo\anaconda3\lib\concurrent\futures_base.py", line 435, in result
return self.__get_result()
File "C:\Users\Shungo\anaconda3\lib\concurrent\futures_base.py", line 384, in __get_result
raise self._exception
File "C:\Users\Shungo\anaconda3\lib\concurrent\futures\thread.py", line 57, in run
result = self.fn(self.args, *self.kwargs)
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\subdir_data.py", line 98, in
package_ref_or_match_spec))
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\subdir_data.py", line 110, in query
self.load()
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\subdir_data.py", line 174, in load
_internal_state = self._load()
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\subdir_data.py", line 248, in _load
repodata_fn=self.repodata_fn)
File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\subdir_data.py", line 485, in fetch_repodata_remote_request
timeout=timeout)
File "C:\Users\Shungo\anaconda3\lib\site-packages\requests\sessions.py", line 546, in get
return self.request('GET', url, *kwargs)
File "C:\Users\Shungo\anaconda3\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, *send_kwargs)
File "C:\Users\Shungo\anaconda3\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, *kwargs)
File "C:\Users\Shungo\anaconda3\lib\site-packages\requests\adapters.py", line 412, in send
conn = self.get_connection(request.url, proxies)
File "C:\Users\Shungo\anaconda3\lib\site-packages\requests\adapters.py", line 309, in get_connection
proxy_manager = self.proxy_manager_for(proxy)
File "C:\Users\Shungo\anaconda3\lib\site-packages\requests\adapters.py", line 199, in proxy_manager_for
*proxy_kwargs)
File "C:\Users\Shungo\anaconda3\lib\site-packages\urllib3\poolmanager.py", line 470, in proxy_from_url
return ProxyManager(proxy_url=url, **kw)
File "C:\Users\Shungo\anaconda3\lib\site-packages\urllib3\poolmanager.py", line 420, in __init__
raise ProxySchemeUnknown(proxy.scheme)
urllib3.exceptions.ProxySchemeUnknown: Not supported proxy scheme None

Last one mean it is similar problem.
However, I could not understand how to change setting proxy.

ShunTono picture ShunTono on 6 May 2020

I try to change it, however, the manner may be bad

(base) C:\Users\Shungo>conda update conda
Collecting package metadata (current_repodata.json): failed

>>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

Traceback (most recent call last):
  File "C:\Users\Shungo\anaconda3\lib\site-packages\requests\adapters.py", line 412, in send
    conn = self.get_connection(request.url, proxies)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\requests\adapters.py", line 305, in get_connection
    proxy_url = parse_url(proxy)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\urllib3\util\url.py", line 392, in parse_url
    return six.raise_from(LocationParseError(source_url), None)
  File "<string>", line 3, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: https://<ip>:<port>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\exceptions.py", line 1079, in __call__
    return func(*args, **kwargs)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\cli\main.py", line 84, in _main
    exit_code = do_call(args, p)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\cli\conda_argparse.py", line 82, in do_call
    return getattr(module, func_name)(args, parser)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\cli\main_update.py", line 20, in execute
    install(args, parser, 'update')
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\cli\install.py", line 265, in install
    should_retry_solve=(_should_retry_unfrozen or repodata_fn != repodata_fns[-1]),
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\solve.py", line 117, in solve_for_transaction
    should_retry_solve)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\solve.py", line 158, in solve_for_diff
    force_remove, should_retry_solve)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\solve.py", line 262, in solve_final_state
    ssc = self._collect_all_metadata(ssc)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\common\io.py", line 88, in decorated
    return f(*args, **kwds)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\solve.py", line 415, in _collect_all_metadata
    index, r = self._prepare(prepared_specs)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\solve.py", line 1011, in _prepare
    self.subdirs, prepared_specs, self._repodata_fn)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\index.py", line 228, in get_reduced_index
    repodata_fn=repodata_fn)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\subdir_data.py", line 105, in query_all
    result = tuple(concat(executor.map(subdir_query, channel_urls)))
  File "C:\Users\Shungo\anaconda3\lib\concurrent\futures\_base.py", line 598, in result_iterator
    yield fs.pop().result()
  File "C:\Users\Shungo\anaconda3\lib\concurrent\futures\_base.py", line 435, in result
    return self.__get_result()
  File "C:\Users\Shungo\anaconda3\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
  File "C:\Users\Shungo\anaconda3\lib\concurrent\futures\thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\subdir_data.py", line 98, in <lambda>
    package_ref_or_match_spec))
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\subdir_data.py", line 110, in query
    self.load()
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\subdir_data.py", line 174, in load
    _internal_state = self._load()
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\subdir_data.py", line 248, in _load
    repodata_fn=self.repodata_fn)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\conda\core\subdir_data.py", line 485, in fetch_repodata_remote_request
    timeout=timeout)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\requests\sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\Shungo\anaconda3\lib\site-packages\requests\adapters.py", line 414, in send
    raise InvalidURL(e, request=request)
requests.exceptions.InvalidURL: Failed to parse: https://<ip>:<port>
ShunTono picture ShunTono on 6 May 2020

I sove this by modify pip_vendor\requests\adapters.py
def get_connection(self, url, proxies=None):
"""Returns a urllib3 connection for the given URL. This should not be
called from user code, and is only exposed for use when subclassing the
:class:HTTPAdapter <requests.adapters.HTTPAdapter>.

    :param url: The URL to connect to.
    :param proxies: (optional) A Requests-style dictionary of proxies used on this request.
    :rtype: urllib3.ConnectionPool
    """
    proxy = select_proxy(url, proxies)
    print(proxy)

    proxy = None #<<<<<< ADD THIS LINE
    if proxy:
        proxy = prepend_scheme_if_needed(proxy, 'http')
        proxy_url = parse_url(proxy)
        if not proxy_url.host:
            raise InvalidProxyURL("Please check proxy URL. It is malformed"
                                  " and could be missing the host.")
        proxy_manager = self.proxy_manager_for(proxy)
        conn = proxy_manager.connection_from_url(url)
    else:
        # Only scheme should be lower case
        parsed = urlparse(url)
        url = parsed.geturl()
        conn = self.poolmanager.connection_from_url(url)

    return conn
ningfc picture ningfc on 15 Nov 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

mitar picture mitar  路  4Comments
eromoe picture eromoe  路  3Comments
Matt3o12 picture Matt3o12  路  3Comments
jake491 picture jake491  路  3Comments
xsren picture xsren  路  3Comments