Invoke: ascii encoding failure under Python 3 on EC2 / Elastic Beanstalk

[bitprophet]

See description in this comment on #350 by @aleksihakli: https://github.com/pyinvoke/invoke/issues/350#issuecomment-232641766

---

Quick note: hide=True just skips the automatic mirroring to stdout/err; presumably if one manually were to try printing or decoding run(...).stdout it'd explode there too.

---

I'm wondering why this is breaking for @aleksihakli; is their environment a normal-ish EC2 instance with a human user at a terminal? Or is Elastic Beanstalk executing it in a more limited environment?

They're on Python 3.4, so Invoke isn't manually encoding the string, it's assuming Python 3's io writer classes are doing the correct thing. My best guess is that in this case something is triggering use of a writer class that does _not_ do the correct thing; or that the outer environment has properties we haven't encountered yet.

Hoping Aleksi can shed some light here :)

[stevenhair]

I see the same issue with Invoke 0.13.0 on Python 3.5 running on CentOS 7 when running mocha tests for our Node builds.

Calling run('mocha specs') fails with UnicodeEncodeError: 'ascii' codec can't encode characters in position 58: ordinal not in range(128).

You're right that calling run with hide=True does not fail, but subsequently printing result.stdout will fail with the same error.

[bitprophet]

@stevenhair are you running those interactively or e.g. via CI of some kind?

The issue here is that in our testing, Python 3's default sys.stdout/err handle arbitrary unicode writes pretty well, so you've found some edge case. If you can provide more details (esp if there's a way for me to reproduce your test run locally) that'd be helpful - thanks!

[stevenhair]

It's part of our Jenkins CI build. I can confirm that running locally works without a problem on both OS X 10.11 and CentOS 7.1. However, when you run it on CentOS 7.2 or Ubuntu 14.04 in Docker (either locally or in CI), then it throws the error.

I made a proof of concept project here, and here's the build log illustrating the issue.

[stevenhair]

Actually, it may just be a problem running in Docker in general. If you build that docker machine and then manually run inv test, it errors as well.

[bitprophet]

OK, so another mark under "probably something specific to noninteractive use". Thanks for setting up the repro case, I'll try to use that when I look next!

[jefflage]

@bitprophet It seems the 0.12.3 milestone was skipped and 0.13.0 has been released. Is this bug still open?

[bitprophet]

@jefflage It is, milestones are a bit wonky pre-1.0. Github ticket status is a clearer indicator than milestone version numbers, fwiw. If I consider something closed/done/wontfix, I always mash 'close'.

I've been focusing on feature work for the last few months but intend to put a bunch of love into bugfixes (on all my projects) very soon now.

[stevenhair]

@bitprophet another clue: https://travis-ci.org/stevenhair/vigilant-potato/builds/187228305

It turns out that the default encoding for CentOS is ANSI. If you switch it to UTF-8, everything works fine.

[rolandgeider]

FYI, I had a very similar problem (docker + bower + invoke) and the solution was to change the locale in the docker container

[denisemauldin]

@rolandgeider How did you do this?

I'm running into this error with pyinvoke on python 2 on a Docker container using alpine. I've installed and set the encoding to en_US.UTF-8 and the npm install is throwing a UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 15: ordinal not in range(128)

If I use locale.getpreferredencoding() before the Context.run command, it shows UTF-8.
If I use locale.getpreferredencoding(False) before the Context.run command, it shows ASCII.

Either way I can't seem to get the alpine Docker container to set the locale so that the UTF-8 output by npm install doesn't crash pyinvoke

[MrwanBaghdad]

Is there any update for this I having the same issue. My scenario is pretty similar to @denisemauldin and @stevenhair. The error is thrown during the jenkins build which is a docker in docker setup. I tried to change the invoke.yaml encoding parameter to UTF-8 but without any change. Does the subthreads respect the main thread encoding?

[MrwanBaghdad]

I think I found the solution setting the environment variable LANG to UTF-8 entry make it works.
So in my case adding the line below to my Dockerfile solve the UnicodeError that was thrown by npm.
ENV LANG=en_US.UTF-8

[denisemauldin]

@MrwanBaghdad I'm glad that solved your issue. I've tried that and still running into errors. We ended up having to pipe the npm install output into a file so that it wouldn't display to screen in order to get our docker build to run. I still just wish that pyinvoke would fix this issue.

[JoeAlamo]

I've had this same error, when running pyinvoke inside my build container when running a GitLab CI pipeline.

Added

variables:
    LANG: en_US.UTF-8

Which has resolved it.