Mailinabox: requests/pyopenssl `TypeError: must be str, not bytes` error

[henningwerner]

Traceback (most recent call last):
  File "management/ssl_certificates.py", line 807, in <module>
    provision_certificates_cmdline()
  File "management/ssl_certificates.py", line 450, in provision_certificates_cmdline
    status = provision_certificates(env, agree_to_tos_url=agree_to_tos_url, logger=my_logger, force_domains=force_domains, show_extended_problems=show_extended_problems)
  File "management/ssl_certificates.py", line 337, in provision_certificates
    logger=my_logger)
  File "/usr/local/lib/python3.4/dist-packages/free_tls_certificates/client.py", line 64, in issue_certificate
    agree_to_tos_url, validation_method, acme_server, logger)
  File "/usr/local/lib/python3.4/dist-packages/free_tls_certificates/client.py", line 112, in validate_domain_ownership
    acme_server=acme_server)
  File "/usr/local/lib/python3.4/dist-packages/free_tls_certificates/client.py", line 317, in create_client
    client = acme.client.Client(acme_server, key)
  File "/usr/local/lib/python3.4/dist-packages/acme/client.py", line 63, in __init__
    self.net.get(directory).json())
  File "/usr/local/lib/python3.4/dist-packages/acme/client.py", line 624, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/usr/local/lib/python3.4/dist-packages/acme/client.py", line 606, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 350, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 835, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connection.py", line 311, in connect
    cert_reqs=resolve_cert_reqs(self.cert_reqs),
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/util/ssl_.py", line 267, in create_urllib3_context
    context.set_ciphers(ciphers or DEFAULT_CIPHERS)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 385, in set_ciphers
    self._ctx.set_cipher_list(ciphers)
TypeError: must be str, not bytes

[guyzmo]

Is the error happening every time the cron job run?

If it is, to me it looks like it's possible to be a fault either in the free_tls_certificate»acme»requests chain. There are two most propable sources of bytes() in py3: either reading from a file opened in binary mode, or raw data returned by requests.

First, try to run the following command (the one being ran by the cron job) as root:

cd /root/mailinabox && management/daily_tasks.sh

if you're not getting the same traceback as you're receiving in your mailbox, then you've got a pretty weird error, and it's gonna be tough to understand.

But if you do, it's possible to make you replicate the error, and figure out where the hell ciphers is coming from to end up improperly formatted, though it'll go through some code to add in there…

[henningwerner]

Same error.

[guyzmo]

happening once? once in a while, or each time?

is running the cron job manually replicate the error?

[henningwerner]

I've only run it once manually.
And yes after waiting a while I got the same error as above.
The error appears during runtime only once.

The error seems to be reproducible.

[JoshData]

Since this error is related to pyOpenSSL, it's probably similar to what was reported in #1017. But since @henningwerner reported in #1017 that they ran the suggested steps there already, I guess those steps don't fix this problem.

Since I can't replicate the problem it's a little hard to figure out. I'd try first forcing the Python packages that depend on pyOpenSSL to update:

pip3 freeze | egrep "acme|requests" # check versions before
pip3 install -U acme requests
pip3 freeze | egrep "acme|requests" # check versions after

[TheGo2SWATking]

root@box:~/mailinabox# pip3 freeze | egrep "acme|requests"
acme==0.9.3
requests==2.12.2
root@box:~/mailinabox# pip3 install -U acme requests
Requirement already up-to-date: acme in /usr/local/lib/python3.4/dist-packages
Downloading/unpacking requests from https://pypi.python.org/packages/84/68/f0acceafe80354aa9ff4ae49de0572d27929b6d262f0c55196424eb86b2f/requests-2.12.3-py2.py3-none-any.whl#md5=9b4e857c4a27a9c282cb1abb53ebaba2
  Downloading requests-2.12.3-py2.py3-none-any.whl (575kB): 575kB downloaded
Installing collected packages: requests
  Found existing installation: requests 2.12.2
    Uninstalling requests:
      Successfully uninstalled requests
Successfully installed requests
Cleaning up...
root@box:~/mailinabox# pip3 freeze | egrep "acme|requests"
acme==0.9.3
requests==2.12.3

And here's the output of the daily_tasks.sh file after running the pip3 commands:

root@box:~/mailinabox# management/daily_tasks.sh
Traceback (most recent call last):
  File "management/ssl_certificates.py", line 807, in <module>
    provision_certificates_cmdline()
  File "management/ssl_certificates.py", line 450, in provision_certificates_cmdline
    status = provision_certificates(env, agree_to_tos_url=agree_to_tos_url, logger=my_logger, force_domains=force_domains, show_extended_problems=show_extended_problems)
  File "management/ssl_certificates.py", line 337, in provision_certificates
    logger=my_logger)
  File "/usr/local/lib/python3.4/dist-packages/free_tls_certificates/client.py", line 64, in issue_certificate
    agree_to_tos_url, validation_method, acme_server, logger)
  File "/usr/local/lib/python3.4/dist-packages/free_tls_certificates/client.py", line 112, in validate_domain_ownership
    acme_server=acme_server)
  File "/usr/local/lib/python3.4/dist-packages/free_tls_certificates/client.py", line 317, in create_client
    client = acme.client.Client(acme_server, key)
  File "/usr/local/lib/python3.4/dist-packages/acme/client.py", line 63, in __init__
    self.net.get(directory).json())
  File "/usr/local/lib/python3.4/dist-packages/acme/client.py", line 624, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/usr/local/lib/python3.4/dist-packages/acme/client.py", line 606, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 350, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 835, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connection.py", line 311, in connect
    cert_reqs=resolve_cert_reqs(self.cert_reqs),
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/util/ssl_.py", line 267, in create_urllib3_context
    context.set_ciphers(ciphers or DEFAULT_CIPHERS)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 385, in set_ciphers
    self._ctx.set_cipher_list(ciphers)
TypeError: must be str, not bytes

[guyzmo]

looking at both tracebacks, I believe they might be related, but they're not the same, and the fix in the other issue won't solve anything here.

I believe a good thing would be to find out where ciphers comes from. A good idea would be for someone to throw a debugger at it:

in /usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py, guard the line 385:

try:
    self._ctx.set_cipher_list(ciphers)
except TypeError as err:
    import pdb; pdb.set_trace()

when the pdb console appears, print what ciphers contains:

pdb> print(type(ciphers), ciphers)

I expect type(ciphers) to give bytes (or there wouldn't be an error).

Then:

pdb> up
pdb> print(cipher)

if it's showing up None or some empty list or False value, then the bug is local to that module, and should be reported to the urllib3 (or requests) module.

If it's set up to something, it's important to track where it's from.

[guyzmo]

Following the code flow that issue shouldn't be happening. When I follow the traceback:

• connection.py:

            self.ssl_context = create_urllib3_context(
                ssl_version=resolve_ssl_version(self.ssl_version),
                cert_reqs=resolve_cert_reqs(self.cert_reqs),
                # ciphers is not defined, i.e. defaults to None!
            )

in ssl_.py

# DEFAULT_CIPHERS contains an str() string:

DEFAULT_CIPHERS = ':'.join([
 'ECDH+AESGCM',
# […]
    '!MD5',
])


def create_urllib3_context(ssl_version=None, cert_reqs=None,
options=None, ciphers=None):
# […]
 context.set_ciphers(ciphers or DEFAULT_CIPHERS) # so here we're using DEFAULT_CIPHERS

in pyopenssl.py, that issue should not be happening.

[…]
self._ctx = OpenSSL.SSL.Context(self.protocol)
[…]
        if isinstance(ciphers, six.text_type):
            ciphers = ciphers.encode('utf-8')
        self._ctx.set_cipher_list(ciphers)

So actualy:

looking at both tracebacks, I believe they might be related, but they're not the same, and the fix in the other issue won't solve anything here.

scratch that, it's openssl that's cocked up and for some reason it can't take an utf-8 encoded str(). So reinstalling it might solve the case…

Though maybe you're loading the wrong OpenSSL library, having OpenSSL loaded from py2 wouldn't be nice ☺ Check that with:

>>> import OpenSSL
>>> OpenSSL.__file__
'/usr/local/lib/python3.4/dist-packages/OpenSSL/__init__.py'

What's weird is that I just tried locally, and set_cipher_list() can take both str() and bytes() as argument.

[henningwerner]

So for me, what should I do?
How can I help you?

[guyzmo]

well, if you're a bit at ease with code, please do what I suggest in this answer.

All in all, it looks to me the issue comes from OpenSSL .so file. In the solution you suggested, you install both "pyOpenSSL" and "python-openssl" which look like the same package… but it might be that "python-openssl" isn't installing the right version of the lib.

As I cannot reproduce it on my side, I'm not sure how to guide you. But to me it looks like a version incoherence between the OpenSSL .so lib and the .py lib that you installed, likely due to using two packages manager (apt and pip).

[NatCC]

I was having the same problem as above; I think I have solved it on my box, based upon the incoherence mentioned earlier.

Uninstall pyOpenSSL using pip3 and then install it again as root:
pip3 uninstall pyOpenSSL
pip3 install pyOpenSSL

After running the two commands, try running the management/daily_tasks.sh script again. For me, it ran without a hitch and reprovisioned the certificates that were running out.

[TheGo2SWATking]

I did what @NatCC suggested, and it does work fine now. The problem I'm facing now, which I stated in the original discourse issue, is that I'm trying to add an additional subdomain (autodiscover.example.com) to my mailinabox. I've already updated external DNS, and added it as a mail alias to get it to appear in the TLS certificates page. When I click the blue provision button, I get this error:

autodiscover.example.com

Something unexpected went wrong: [NeedToInstallFile('http://autodiscover.example.com/.well-known/acme-challenge/-yK8l_mSHZbFKIRxDNiPY9Cw_-kk7QAcPhVFSDTUj1I', '-yK8l_mSHZbFKIRxDNiPY9Cw_-kk7QAcPhVFSDTUj1I.6FfPp-Ovtg_39pZRnZSfT-ypgbiyJpueVG97TSZkVLk', '-yK8l_mSHZbFKIRxDNiPY9Cw_-kk7QAcPhVFSDTUj1I')]

Log:
Reading account key from /home/user-data/ssl/lets_encrypt/account.pem.
Validating existing account saved to /home/user-data/ssl/lets_encrypt/registration.json.
Reusing existing challenges for autodiscover.example.com.
Validation file is not present --- a file must be installed on the web server.
Reading account key from /home/user-data/ssl/lets_encrypt/account.pem.
Validating existing account saved to /home/user-data/ssl/lets_encrypt/registration.json.
Reusing existing challenges for autodiscover.example.com.
Validation file is not present --- a file must be installed on the web server.

In addition to that, when running daily_tasks.sh I get this error:

Exception ignored in: >
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/acme/client.py", line 511, in __d el__
self.session.close()
File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 689, in close
for v in self.adapters.values():
File "/usr/lib/python3.4/_collections_abc.py", line 512, in __iter__
for key in self._mapping:
File "/usr/lib/python3.4/collections/__init__.py", line 91, in __iter__
curr = root.next
ReferenceError: weakly-referenced object no longer exists
Exception ignored in: >
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/acme/client.py", line 511, in __d el__
self.session.close()
File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 689, in close
for v in self.adapters.values():
File "/usr/lib/python3.4/_collections_abc.py", line 512, in __iter__
for key in self._mapping:
File "/usr/lib/python3.4/collections/__init__.py", line 91, in __iter__
curr = root.next
ReferenceError: weakly-referenced object no longer exists

All of this goes away if I remove the mail alias, therefore removing the subdomain, but it comes back if I try to add the subdomain again. The exact same thing happens when I try to add a different subdomain (test.example.com).

[henningwerner]

Done following:

sudo pip3 uninstall pyOpenSSL
sudo pip3 install pyOpenSSL
sudo pip3 install pyOpenSSL --upgrade

Last step I've done because, pip3 said it's already installed...
But now it works, curious.

[mzoeller]

I had the same problem. Above suggestions did not help. Removing all pyOpenSSL and Ubuntu's "python3-openssl" lib (apt-get remove python3-openssl) and installing pyOpenSSL though pip finally resolved the issue. Curious since since I never installed anything else on this box (apart from Ubuntu and mail-in-a-box).

[guyzmo]

@mzoeller not curious, because MIAB installs stuff on the box.

[millermarkj]

I've tried all of the suggestions posted here and nothing has resolved this problem for me.

[guyzmo]

@millermarkj is it exactly the same traceback? or is it a traceback that looks alike?

[JoshData]

When in doubt, just move all of your data to a fresh machine:

https://mailinabox.email/maintenance.html#moving-boxes

[mattbeedle]

What worked for me, collected from comments above:

sudo apt-get remove --purge python3-pip python-pip python3-openssl
sudo shutdown -r 0

The restart step above is important! I tried several times without that and it didn't work. Once restarted run the following:

sudo apt-get install python3-pip
pip3 install pyOpenSSL
pip3 install pyOpenSSL --upgrade
cd /root/mailinabox && management/daily_tasks.sh

If the last step returns no output then it worked :)

[TabTwo]

@mattbeedle tried the steps you mentioned.
This led to

Traceback (most recent call last):
  File "management/ssl_certificates.py", line 807, in <module>
    provision_certificates_cmdline()
  File "management/ssl_certificates.py", line 450, in provision_certificates_cmdline
    status = provision_certificates(env, agree_to_tos_url=agree_to_tos_url, logger=my_logger, force_domains=force_domains, show_extended_problems=show_extended_problems)
  File "management/ssl_certificates.py", line 272, in provision_certificates
    import acme.messages
  File "/usr/local/lib/python3.4/dist-packages/acme/messages.py", line 4, in <module>
    from acme import challenges
  File "/usr/local/lib/python3.4/dist-packages/acme/challenges.py", line 9, in <module>
    import OpenSSL
ImportError: No module named 'OpenSSL'

reinstalling python3-openssl then led again to "TypeError: must be str, not bytes"

next I tried the steps from @henningwerner

sudo pip3 uninstall pyOpenSSL
sudo pip3 install pyOpenSSL
sudo pip3 install pyOpenSSL --upgrade

this led to

root@box:~/mailinabox# cd /root/mailinabox && management/daily_tasks.sh
Exception ignored in: <bound method ClientNetwork.__del__ of <acme.client.ClientNetwork object at 0x7ff131626eb8>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/acme/client.py", line 511, in __del__
    self.session.close()
  File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 689, in close
    for v in self.adapters.values():
  File "/usr/lib/python3.4/_collections_abc.py", line 512, in __iter__
    for key in self._mapping:
  File "/usr/lib/python3.4/collections/__init__.py", line 91, in __iter__
    curr = root.next
ReferenceError: weakly-referenced object no longer exists

but the certs were updated and the box is running again.
Mail-in-a-Box is up to date. You are running version v0.21b.

[davinian]

I had similar problems, Lets Encrypt would not renew - had one day left so scanning through all tickets I tried the following and now have new certs!

removed and reinstalled pyOpenSSL

sudo apt-get remove python3-openssl
sudo pip3 uninstall pyOpenSSL
sudo pip3 install pyOpenSSL

and the updated

apt-get install -y --reinstall python-pkg-resources
python3 -m pip install --upgrade setuptools
python3 -m pip install --upgrade distribute
python3 -m pip install --upgrade pip

re-run the Mail-in-a-Box upgrade (I had already updated to v0.21c)
curl -s https://mailinabox.email/setup.sh | sudo bash

and then run the daily task
cd /root/mailinabox && management/daily_tasks.sh

It might be that just upgrading and running the daily task would have fixed the problem, but I have no way of knowing.

[TheGo2SWATking]

@davinian Thanks for the post, though I wasn't able to get my box working with those instructions. Maybe it'll help someone else who is experiencing a similar issue.