Urllib3: connection.py: BytesWarning: Comparison between bytes and string: if SKIP_HEADER not in values:

[jdufresne]

Subject

Starting with the 1.26 series, I started receiving a BytesWarning in my project's test suite.

.../urllib3/connection.py:218: BytesWarning: Comparison between bytes and string
  if SKIP_HEADER not in values:

I run my test suite with Python's -b option to help catch misuses of bytes and string objects. My hunch is that urllib3 is sometimes treating headers as bytes and other times as string and then comparing them with each other.

Environment

Describe your environment.
At least, paste here the output of:

import platform
import urllib3

print("OS", platform.platform())
print("Python", platform.python_version())
print("urllib3", urllib3.__version__)

OS Linux-5.8.16-200.fc32.x86_64-x86_64-with-glibc2.2.5
Python 3.8.6
urllib3 1.26.2

Steps to Reproduce

This is reproducible using urllib3's own test suite:

Invoke using:

$ python -b -m pytest -W a test/with_dummyserver/test_socketlevel.py
...
test/with_dummyserver/test_socketlevel.py::TestProxyManager::test_connect_ipv6_addr
  /home/jon/.virtualenvs/urllib3/lib64/python3.9/site-packages/urllib3/connection.py:193: BytesWarning: Comparison between bytes and string
    if SKIP_HEADER not in values:

Expected Behavior

No ByteWarning emitted.

Actual Behavior

.../connection.py:193: BytesWarning: Comparison between bytes and string
    if SKIP_HEADER not in values:

[sethmlarson]

Nice catch, we'll have to enable this in our test suite and remove the issue.

[The-Compiler]

I'm seeing this as well FWIW - I did a quick bisect and this was introduced in 16b7b332fd1b84c2d465f11d17658c1e83d3f20f (#2018, "Add SKIP_HEADER for skipping automatically added headers").

[mgorny]

Ping. This is causing a lot of cron spam for a lot of our users.

[fred-vogt]

Breaks python ... -bb ..., until this is fixed, can be worked around by downgrading in pip:

urllib3==1.25.11

[pquentin]

@mgorny Can you please clarify what Gentoo cron spam is?

The issue here is that putheader accepts an array of value that are usually strings, but can also be bytes. For example, the standard library calls putheader("Host": encoded_netloc) in https://github.com/python/cpython/blob/v3.9.1/Lib/http/client.py#L1129-L1133. Opened https://github.com/urllib3/urllib3/pull/2141 to fix this.

[mgorny]

@pquentin, I mean one of our most important tools happen to trigger this, and many users run it via cron in quiet mode. Now that Python suddenly started outputting this warning, quiet mode is no longer quiet and people are getting mails from cron that something is wrong.

[pquentin]

This was released as part of urllib3 1.26.3. Sorry for the noise!