Spacy: Wasabi Unicode Error from \u2714 (heavy check mark) when linking model via CLI

[jfelectron]

How to reproduce the behavior

Shouldn't assume that stdout supports Unicode...

• python -m spacy link en_core_web_sm en --force
  Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
  "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
  exec code in run_globals
  File "/home/zoltar_venv/lib/python2.7/site-packages/spacy/__main__.py", line 38, in
  plac.call(commands[command], sys.argv[1:])
  File "/home/zoltar_venv/local/lib/python2.7/site-packages/plac_core.py", line 328, in call
  cmd, result = parser.consume(arglist)
  File "/home/zoltar_venv/local/lib/python2.7/site-packages/plac_core.py", line 207, in consume
  return cmd, self.func((args + varargs + extraopts), *kwargs)
  File "/home/zoltar_venv/local/lib/python2.7/site-packages/spacy/cli/link.py", line 77, in link
  msg.good("Linking successful", details)
  File "/home/zoltar_venv/local/lib/python2.7/site-packages/wasabi/printer.py", line 72, in good
  return self._get_msg(title, text, style=MESSAGES.GOOD, show=show, exits=exits)
  File "/home/zoltar_venv/local/lib/python2.7/site-packages/wasabi/printer.py", line 213, in _get_msg
  return self.text(title, text, color=style, icon=style, show=show, exits=exits)
  File "/home/zoltar_venv/local/lib/python2.7/site-packages/wasabi/printer.py", line 122, in text
  print(title)
  UnicodeEncodeError: 'ascii' codec can't encode character u'\u2714' in position 9: ordinal not in range(128)

Your Environment

• Operating System: Ubuntu 18.10
• Python Version Used: 2.7.16
• spaCy Version Used: 2.1
• Environment Information:

[ines]

Hmm, that's strange – it should definitely handle that and decode the strings accordingly.

Edit: What does locale.getpreferredencoding() return for you?

[jfelectron]

Let me see, this happens inside a container within our CI environment.

[jfelectron]

The container is a different env, could be the problem.

Python 2.7.12 (default, Nov 19 2016, 06:48:10)

locale.getpreferredencoding()
'UTF-8'

[ines]

Ahhh, that might explain what's going on. Internally, wasabi calls string.encode(encoding, "ignore").decode("utf8"). So maybe for cases like this, we also have to check for sys.stdout.encoding? I'll look into this.

I'll also add an environment variable for disabling pretty printing (which will remove color, icons etc.). There's already WASABI_LOG_FRIENDLY, but that only removes animations and ANSI escape sequences for colors etc.

[jfelectron]

The link command is executed in a BASH script by a CI system, so stdout is not a Python/Ipython interpreter. Is there a workaround if I want to try to use 2.1?

[honnibal]

The link command is executed in a BASH script by a CI system, so stdout is not a Python/Ipython interpreter.

🤔 Not quite sure I follow there? I mean, it's very normal that the script is running from bash...

I think it's just a question of checking the right thing to see whether unicode is supported. In the meantime, is using Python3 an option?

[honnibal]

Try setting PYTHONIOENCODING=UTF-8?

[ines]

I also just pushed a new version of wasabi (0.1.4) that supports a WASABI_NO_PRETTY environment variable. So you can do export WASABI_NO_PRETTY=1 or something, and it should strip out icons and other formatting.

[ines]

Just published wasabi v0.2.0, which should fix this by default. I was able to reproduce the problem in the CI and also added some nicer fallbacks.

cc: @bdewilde Could you try upgrading wasabi and check if it works without the env var now?

[bdewilde]

The new version works great — thank you, @ines!

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.