Node: test, tools: tools\test.py --help fails on Windows in shell with Unicode codepage

[vsemozhetbyt]

• Version: Node.js 8.0.0 nightly 2017 05 01, Python 2.7.13
• Platform: Windows 7 x64
• Subsystem: test
  cmd.exe with Unicode codepage by default:

J:\temp\_git\node-fork>tools\test.py --help
Traceback (most recent call last):
  File "J:\temp\_git\node-fork\tools\test.py", line 1769, in <module>
    sys.exit(Main())
  File "J:\temp\_git\node-fork\tools\test.py", line 1593, in Main
    (options, args) = parser.parse_args()
  File "C:\Python27\lib\optparse.py", line 1400, in parse_args
    stop = self._process_args(largs, rargs, values)
  File "C:\Python27\lib\optparse.py", line 1440, in _process_args
    self._process_long_opt(rargs, values)
  File "C:\Python27\lib\optparse.py", line 1515, in _process_long_opt
    option.process(opt, value, values, self)
  File "C:\Python27\lib\optparse.py", line 789, in process
    self.action, self.dest, opt, value, values, parser)
  File "C:\Python27\lib\optparse.py", line 811, in take_action
    parser.print_help()
  File "C:\Python27\lib\optparse.py", line 1670, in print_help
    file.write(self.format_help().encode(encoding, "replace"))
LookupError: unknown encoding: cp65001

J:\temp\_git\node-fork>chcp 1252
Active code page: 1252

J:\temp\_git\node-fork>tools\test.py --help
Usage: test.py [options]

Options:
  -h, --help            show this help message and exit
  -m MODE, --mode=MODE  The test modes in which to run (comma-separated)
  -v, --verbose         Verbose output
  --logfile=LOGFILE     write test output to file. NOTE: this only applies the
                        tap progress indicator
  -p PROGRESS, --progress=PROGRESS
                        The style of progress indicator (verbose, dots, color,
                        mono, tap)
  --report              Print a summary of the tests to be run
  -s SUITE, --suite=SUITE
                        A test suite
  -t TIMEOUT, --timeout=TIMEOUT
                        Timeout in seconds
  --arch=ARCH           The architecture to run tests for
  --snapshot            Run the tests with snapshot turned on
  --special-command=SPECIAL_COMMAND
  --node-args=NODE_ARGS
                        Args to pass through to Node
  --expect-fail         Expect test cases to fail
  --valgrind            Run tests through valgrind
  --check-deopts        Check tests for permanent deoptimizations
  --cat                 Print the source of the tests
  --flaky-tests=FLAKY_TESTS
                        Regard tests marked as flaky (run|skip|dontcare)
  --warn-unused         Report unused rules
  -j J                  The number of parallel tasks to run
  -J                    Run tasks in parallel on all cores
  --time                Print timing information after running
  --suppress-dialogs    Suppress Windows dialogs for crashing tests
  --no-suppress-dialogs
                        Display Windows dialogs for crashing tests
  --shell=SHELL         Path to V8 shell
  --store-unexpected-output
                        Store the temporary JS files from tests that fails
  --no-store-unexpected-output
                        Deletes the temporary JS files from tests that fails
  -r RUN, --run=RUN     Divide the tests in m groups (interleaved) and run
                        tests from group n (--run=n,m with n < m)
  --temp-dir=TEMP_DIR   Optional path to change directory used for tests
  --repeat=REPEAT       Number of times to repeat given tests
  --abort-on-timeout    Send SIGABRT instead of SIGTERM to kill processes that
                        time out

[gibfahn]

cc/ @nodejs/platform-windows @nodejs/python

[refack]

I like this bug, great work @vsemozhetbyt
I've self assigned this.
(In my company I would look very hard for edge-case attracting people for QA 😍 they are priceless. A bug in QA if so much cheaper than a bug in the field!)

[refack]

This is what's called a can of worms... So it's a Windows/Python bug.
http://bugs.python.org/issue1602
http://stackoverflow.com/questions/878972/windows-cmd-encoding-change-causes-python-crash

We can fix each tool with a header:

import codecs
codecs.register(lambda name: codecs.lookup('utf-8') if name == 'cp65001' else None)

or set an environment var:

set PYTHONIOENCODING=UTF-8

I recommend we document this and suggest the dev to set the env var...

[seishun]

-1 on adding hacks to work around Python bugs.

[mscdex]

Why not try the workarounds/fixes referenced in that linked Python issue?

[silverwind]

Can we require Python 3.6 to run tests?

[refack]

Can we require Python 3.6 to run tests?

Except GYP doesn't work with python 3 😆

[refack]

IMHO a comment about setting

set PYTHONIOENCODING=UTF-8

with the right keywords (LookupError: unknown encoding: cp65001, python, test) will be sufficient.

[Trott]

Should this remain open?

[silverwind]

Is there a version of Windows where the default code page is unicode (cp65001) yet? If not, I'd close this as something the user brought onto himself. Not nice, but there really is no possible solution until we can require python3.6 for this script.

[refack]

hopefully we'll get python3 compatibility soon, so this be a non issue