Jupyter-book: Add windows 10 support

[choldgraf]

Currently there are a few things that seem broken or unreliable in Windows 10. This issue is to keep track of these.

• [x] File paths are not reliably used properly (#538 #573)
• [x] Formatting of Sphinx etc output has encoding issues (https://github.com/executablebooks/jupyter-book/issues/574)

If others find similar issues, please raise them here!

[phaustin]

build failure for quantecon example: https://github.com/executablebooks/quantecon-example/issues/12

[phaustin]

same problem and solution for myst_nb/sphinx: https://github.com/executablebooks/MyST-NB/issues/174

[phaustin]

I think a useful question for a windows expert:

"How can we ensure that pip entry_points run python with the -X utf8 flag enabled?"

see: https://docs.python.org/3/using/windows.html#utf-8-mode

[phaustin]

here's a deep dive into utf8 and python: https://vstinner.github.io/python37-new-utf8-mode.html

[phaustin]

windows encoding update: I installed greek as a second language this morning and it looks like the encoding problems have gone away. So once we fix the non-utf8 related problems for the quantecon book we can round up some windows users and try and get some more statistics.

[phaustin]

this PR shows how to get a working build of the quantecon example: https://github.com/executablebooks/quantecon-example/pull/13

[phaustin]

I've uploaded an anaconda package that provides a wrapper to jb to set PYTHONUTF8=1 and run jb with a single argument: https://anaconda.org/eoas_ubc/runjb

Here's the readme

Longer term, I think jupyter-book should follow nbformat, sphinx and bookdown and make the io encoding explicitly utf-8 for every read_text and open. That's a fair amount of work though, and this does the same thing in a single line. A better approach, which also works on pypi, would be to find out how to invoke the -X utf8 flag from a setup.py console_script (at which point this wrapper isn't needed).

[phaustin]

Queried distutils-sig

[asteppke]

Following the startup guide (https://jupyterbook.org/start/overview.html) this is the very first error that I hit:

pip install -U "jupyter-book>=0.7.0b2" jupyter-book create strain jupyter-book toc strain
and then
````
$ jupyter-book build strain

Running Sphinx v2.4.4
making output directory... done

Encoding error:
'charmap' codec can't encode character 'uf06a' in position 3835: character maps to
The full traceback has been saved in C:UsersALEXAN~1AppDataLocalTempsphinx-err-513s9e1b.log, if you want to report the issue to the developers.
Traceback (most recent call last):
File "c:toolsminiconda3envsstrainlibrunpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:toolsminiconda3envsstrainlibrunpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:toolsminiconda3envsstrainScriptsjupyter-book.exe__main__.py", line 7, in
File "c:toolsminiconda3envsstrainlibsite-packagesclickcore.py", line 829, in __call__
return self.main(args, *kwargs)
File "c:toolsminiconda3envsstrainlibsite-packagesclickcore.py", line 782, in main
rv = self.invoke(ctx)
File "c:toolsminiconda3envsstrainlibsite-packagesclickcore.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:toolsminiconda3envsstrainlibsite-packagesclickcore.py", line 1066, in invoke
return ctx.invoke(self.callback, *ctx.params)
File "c:toolsminiconda3envsstrainlibsite-packagesclickcore.py", line 610, in invoke
return callback(args, **kwargs)
File "c:toolsminiconda3envsstrainlibsite-packagesjupyter_bookcommands__init__.py", line 132, in build
_error(
File "c:toolsminiconda3envsstrainlibsite-packagesjupyter_bookutils.py", line 65, in _error
raise kind(box)

ValueError:

There was an error in building your book. Look above for the error message.

===============================================================================
````

When using the workaround set PYTHONUTF8=1 it runs a little bit further but still stops at:

````
$ jupyter-book build strain
Running Sphinx v2.4.4
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 5 source files that are out of date
updating environment: [new config] 5 added, 5 changed, 0 removed
Executing: notebooks 80%] notebooks

Exception occurred:
File "c:toolsminiconda3envsstrainlibasyncioevents.py", line 501, in add_reader
raise NotImplementedError
NotImplementedError
The full traceback has been saved in C:UsersALEXAN~1AppDataLocalTempsphinx-err-xy1efr2r.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at https://github.com/sphinx-doc/sphinx/issues. Thanks!
Traceback (most recent call last):
File "c:toolsminiconda3envsstrainlibrunpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:toolsminiconda3envsstrainlibrunpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:toolsminiconda3envsstrainScriptsjupyter-book.exe__main__.py", line 7, in
File "c:toolsminiconda3envsstrainlibsite-packagesclickcore.py", line 829, in __call__
return self.main(args, *kwargs)
File "c:toolsminiconda3envsstrainlibsite-packagesclickcore.py", line 782, in main
rv = self.invoke(ctx)
File "c:toolsminiconda3envsstrainlibsite-packagesclickcore.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:toolsminiconda3envsstrainlibsite-packagesclickcore.py", line 1066, in invoke
return ctx.invoke(self.callback, *ctx.params)
File "c:toolsminiconda3envsstrainlibsite-packagesclickcore.py", line 610, in invoke
return callback(args, **kwargs)
File "c:toolsminiconda3envsstrainlibsite-packagesjupyter_bookcommands__init__.py", line 132, in build
_error(
File "c:toolsminiconda3envsstrainlibsite-packagesjupyter_bookutils.py", line 65, in _error
raise kind(box)

ValueError:

There was an error in building your book. Look above for the error message.

===============================================================================

Exception ignored in:
Traceback (most recent call last):
File "c:toolsminiconda3envsstrainlibsite-packageszmqsugarsocket.py", line 67, in __del__
File "c:toolsminiconda3envsstrainlibsite-packageszmq_future.py", line 160, in close
File "c:toolsminiconda3envsstrainlibsite-packageszmqasyncio__init__.py", line 60, in _clear_io_state
File "c:toolsminiconda3envsstrainlibasyncioevents.py", line 504, in remove_reader
NotImplementedError:

````
I am happy to debug further but for now this looks like some more work is required that goes beyond a change of path separators or encoding issues.

[phaustin]

Any help would be very much appreciated. Jupyterbook is just beginning to be used on windows, so it's not too surprising that bugs like this are surfacing. We're also looking for good examples to add to CI testing.

[asteppke]

Alright, so the bug above comes from a change in Python 3.8 itself and how the downstream packages have not yet adapted to this:

• https://github.com/sphinx-doc/sphinx/issues/7310
• https://docs.python.org/3/library/asyncio-platforms.html#asyncio-platform-support
• https://github.com/tornadoweb/tornado/issues/2608
• https://www.tornadoweb.org/en/stable/releases/v6.0.4.html#general-changes

I am not sure if jupyterbook can communicate the workaround to Sphinx.

[phaustin]

I can definitely live with a pin to 3.7 in the meantime, thanks. FYI: the other known windows issue (besides utf8) relates to nested content structures like https://github.com/executablebooks/quantecon-example/blob/master/book/_toc.yml
Running this example (with PYTHONUTF8=1) fails with an "index.md not found" error.
The workaround is to just flatten the contents into a single folder, but any analysis of what is going on would be appreciated.

[asteppke]

Downgrading to Python 3.7.6 helped, the example then runs without any issues. So that seems like a good starting point together with PYTHONUTF8=1.

[phaustin]

Well that's interesting/frustrating -- just to comfim -- that includes keeping the timeseries date in https://github.com/executablebooks/quantecon-example/blob/master/book/docs/pandas.md#exercise-2-1 at 1928? When I run this with python 3.7.6 on windows it errors because the timestamp is earlier than 1970, and one of the imported modules doesn't support negative unix timestamps.

[asteppke]

Sorry, no, I am still referring to the most basic example from the template given in https://jupyterbook.org/start/overview.html. This runs now successfully.

When including the pandas.md example document the error is
OverflowError: mktime argument out of range
Changing the date to 1978 works around this problem and the rest executes fine here.

[phaustin]

that's progress, thanks. Must be something about my environment, we'll see if it pops up again once we've got a github action for a windows build.

[choldgraf]

Do folks think it would be useful to add a section for "Running Jupyter Book on windows" here: https://jupyterbook.org/advanced/advanced.html ? I'd be happy to review PRs that gave it a shot w/ some best-practices

[phaustin]

yes, I can do that today. Right now the recommendation would be to:

1) use miniconda with an environment we specify that provides shell wrappers to set PYTHONUTF8=1

that's it, although there's uncertainty around https://github.com/executablebooks/jupyter-book/issues/573 and toc paths -- we'll need some test cases.

Once we get CI for windows, we'll go ahead and add encoding='utf8' to the io in jupyter-cache, jupyter-book and jupyter-sphinx, which should make the wrappers and miniconda unnecessary and we can remove this guidance.

[chrisjsewell]

This should be closed in v0.8.0 😄, but feel free to re-open/open another issue if anything is outstanding: https://jupyterbook.org/reference/_changelog.html#v0-8-0-2020-09-01