Invoke: UnicodeDecodeError if task's doc string is non-ASCII UTF-8 encoded string

[marc1n]

tasks.py:

#  -*- coding: utf-8 -*-
from invoke import task

@task
def test():
    """Zażółć gęślą jaźń <- polish test sentence"""
    pass   

command line:

$ invoke -V
Invoke 0.17.0

$ invoke -l
Available tasks:

Traceback (most recent call last):
  File "/usr/local/bin/invoke", line 11, in <module>
    sys.exit(program.run())
  File "/usr/local/lib/python2.7/dist-packages/invoke/program.py", line 278, in run
    self._parse(argv)
  File "/usr/local/lib/python2.7/dist-packages/invoke/program.py", line 372, in _parse
    self.list_tasks()
  File "/usr/local/lib/python2.7/dist-packages/invoke/program.py", line 574, in list_tasks
    self.print_columns(pairs)
  File "/usr/local/lib/python2.7/dist-packages/invoke/program.py", line 606, in print_columns
    print(spec + help_chunks[0])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 2: ordinal not in range(128)

[pekkaklarck]

I'm fairly certain this would work if you'd use u"""Zażółć gęślą jaźń <- polish test sentence""" instead.

The above said, it would be great if invoke would automatically decode docstrings if they are not Unicode. I would assume it's enough to assume docstrings are UTF-8 encoded in that case. It has the benefit of being ASCII compatible and I don't see why someone would use any other encoding. Adding a system for users to tell which encoding they have used would be a lot more complicated.

[bitprophet]

Could also from __future__ import unicode_literals instead of applying the u, I believe. Something I've started doing in my 2.7 code :)

I'm slightly torn on @pekkaklarck's suggestion to explicitly decode bytes-type docstrings as if they contained UTF-8...it would avoid issues like the OP's, but it also arguably prevents folks from having to learn proper Unicode handling.

Also, there are in fact folks using non ASCII, non UTF-8 encodings - e.g. folks in the UK on Windows use some older one whose name I forget but which came up in the big Unicode shuffle here a year or two back. Not common, but does happen.

Given the ease at which trip-ups like Marcin's are fixable by the user, I don't know if it's worth the admittedly small risk. Open to counterarguments :)

[marc1n]

Thanks @bitprophet for from __future__ import unicode_literals suggestion. It is the right solution for me.

[pekkaklarck]

I don't feel too strongly about this, but got two comments about the matter anyway.

I don't see any risk in assuming docstrings that are bytes being UTF-8. At the moment using non ASCII characters in byte docstrings always fails, after that change only docstrings being non UTF-8 would fail. I doubt there are many people using non UTF-8 encoding in their source files (ASCII being UTF-8 compatible), but this change wouldn't make their life any more complicated it is now.

Notice also that using from __future__ import unicode_literals doesn't always work. The main reason is that there are surprisingly many APIs in Python 2 standard library that only work with byte strings and blow up when using Unicode strings. In my Python 2/3 projects I've opted u'string' format when I want a string to be Unicode.

[pnovotnak]

Hey, just wanted to chime in here and say that the workarounds aren't working for me.

$ python --version
Python 3.6.3

$ pip freeze
...
invoke==0.22.0
...

Docstrings with "Ø" fail for me. I've added # -*- coding: utf-8 -*- to the top of my file, prefixed my strings with u and tried importing unicode_literals from __future__ import unicode_literals.

#  -*- coding: utf-8 -*-

...

@task
def zmq_forwarder(_, pull_bind=settings.Ø['FORWARDER_PULL'],
                  pub_bind=settings.Ø['FORWARDER_PUB'],
                  secrets_dir=settings.Ø['SECRETS_DIR']):
    u""" ØMQ forwarder device for collecting input at 'subscribe' from multitude
    of web server threads, publishing it at at a single socket--'publish' for
    clients to consume
    """
    ...

For me this is not a big issue at all, but it does seem like something worth cleaning up.

[IllanFed]

You have Ø symbol not only in docstrings, see to function arguments declaration, after settings:

def zmq_forwarder(_, pull_bind=settings.Ø['FORWARDER_PULL'],
                  pub_bind=settings.Ø['FORWARDER_PUB'],
                  secrets_dir=settings.Ø['SECRETS_DIR']):

[pnovotnak]

Yup, thanks for letting me know! Removing them from just the docstrings fixes the issue, though.