Jupyter-book: Errors when running on Windows

[emdupre]

Lifted from @kirstieJane feedback in #134

1. jupyter-book build mybookname fails with:

   UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 693: character maps to <undefined>
   

2. jupyter-book create mybookname --demo --overwrite fails with:

   PermissionError: [WinError 5] Access is denied: '.\\mybookname\\.git\\objects\\01\\57a78648328b87569df49479a9813ce273e495'
   

My first guess at solutions: we should add in an .decode('utf-8') when reading in the markdown for (1) and update the documentation to add that the --overwrite command might need to be run with root privileges for (2).

[jasmainak]

why is jupyter-book trying to access the .git folder? :-)

[KirstieJane]

Ooooh! Good question @jasmainak!

I didn’t follow the instructions and made my repo on GitHub (and cloned it locally) so I had a .git folder already. The _actual_ instructions tell you to do that later.

So there’s a mistake on my part there....but also maybe there’s something to update so Jupyter book ignores the .git folder! (I don’t know the answer, totally noob here!!)

Thank you for taking a look @emdupre! I hope my ramblings aren’t too incoherent!!

Happy to test anything as needed 🌸

[jasmainak]

I'm also new to the codebase :-) But looking at it, there seems to be an sh.rmtree that could be the culprit. I think it's better to selectively remove files rather than doing a blind copytree or rmtree whenever applicable. One could also suppress the errors by passing an argument to rmtree but I'm one always in favor of fixing the errors instead of suppressing them. As the Zen of Python says:

Errors should never pass silently.

[jasmainak]

Also, are you able to reproduce the error faithfully? It could be a temporary issue as another process might be locking the folder ...

[netoferraz]

I have encountered similar errors:

SO: Windows 7 Professional SP1
Python 3.7.1

C:\Users\jferraz\Documents\SBQ\CGI\pmqc\wiki
(venv) λ jupyter-book build mybookname
c:\users\jferraz\documents\sbq\cgi\pmqc\wiki\venv\lib\site-packages\jupyter_book\build.py:129: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  site_yaml = yaml.load(ff.read())
c:\users\jferraz\documents\sbq\cgi\pmqc\wiki\venv\lib\site-packages\jupyter_book\build.py:137: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  toc = yaml.load(ff.read())
Convert and copy notebook/md files...
  0%|                                                                                            | 0/35 [00:00<?, ?it/s] Traceback (most recent call last):
  File "C:\Users\jferraz\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\jferraz\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\jferraz\Documents\SBQ\CGI\pmqc\wiki\venv\Scripts\jupyter-book.exe\__main__.py", line 9, in <module>
  File "c:\users\jferraz\documents\sbq\cgi\pmqc\wiki\venv\lib\site-packages\jupyter_book\main.py", line 31, in main
    commands[args.command]()
  File "c:\users\jferraz\documents\sbq\cgi\pmqc\wiki\venv\lib\site-packages\jupyter_book\build.py", line 267, in build_book
    lines = ff.readlines()
  File "C:\Users\jferraz\AppData\Local\Programs\Python\Python37\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 693: character maps to <undefined>

[emdupre]

Hi @netoferraz, thanks for chiming in ! It's good to know that this is occurring for multiple users. It may take us a few days to look into it (a little slow on development cycles right now), but please do submit a patch if you're able and interested !

In the meantime, we have instructions for building with Docker that should allow you to circumvent Windows-specific issues: https://jupyter.org/jupyter-book/guide/03_build.html#building-your-site-locally-with-containers-docker

[jasmainak]

I think the culprit is some unicode character in the files. If you want to fix it, these lines would have to be changed so that they specify an encoding.

[stafforddavidj]

I just encountered this same bug today.

C:\jupyter> jupyter-book build mybookname

...

Traceback (most recent call last):                                                                                             
  File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\runpy.py", line 193, in _run_module_as_main      
    "__main__", mod_spec)                                                                                                      
  File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\runpy.py", line 85, in _run_code                 
    exec(code, run_globals)                                                                                                    
  File "C:\Users\Garbagio\AppData\Roaming\Python\Python36\Scripts\jupyter-book.exe\__main__.py", line 9, in <module>           
  File "C:\Users\Garbagio\AppData\Roaming\Python\Python36\site-packages\jupyter_book\main.py", line 31, in main                
    commands[args.command]()                                                                                                   
  File "C:\Users\Garbagio\AppData\Roaming\Python\Python36\site-packages\jupyter_book\build.py", line 267, in build_book        
    lines = ff.readlines()                                                                                                     
  File "c:\program files (x86)\microsoft visual studio\shared\python36_64\lib\encodings\cp1252.py", line 23, in decode         
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]                                                          
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 693: character maps to <undefined>

Adding UTF-8 to the 'encoding' argument of the open() function in build.py allowed a successful build afterward.

# Clean markdown for Jekyll quirks (e.g. extra escape characters)
        with open(path_new_file, 'r', encoding='utf8') as ff:
            lines = ff.readlines()

[choldgraf]

@stafforddavidj thanks for noting that! Any chance you'd be willing to make a PR updating that line so we can see if the tests are still happy? :-)

[choldgraf]

@KirstieJane wanna check if this is fixed when running "master" now?

[cczhu]

Hi all, I'm working with the current master (dda4a03) in a conda 4.6.14 / Python 3.6.8 environment under Windows 10, and encountered this same issue. I could only get things working after passing the UTF-8 encoding flag when opening files for writing:

        # Write the result as UTF-8.
        with open(path_new_file, 'w', encoding='utf8') as ff:
            ff.writelines(lines)

in addition to the fix for reading suggested by @stafforddavidj. Should that be added as well, or is that a unique quirk of my setup? Thanks!

[choldgraf]

it sounds like the utf8 encoding needs to be there for it to work on windows...but I thought this had been fixed so now I'm confused :-/ isn't that what was fixed in https://github.com/jupyter/jupyter-book/commit/cbc512df254938c04207df97d63b170f3c2dd07e ?

[cczhu]

I think https://github.com/jupyter/jupyter-book/commit/cbc512df254938c04207df97d63b170f3c2dd07e fixes it for ff.readlines(), but not ff.writelines(lines), though I'm not sure if the latter is only necessary for my build environment.

[choldgraf]

just realized that I had dropped this one - that's a good point @cczhu , if the fix you proposed solves this problem for you, then I think we should make a PR - I'm happy to review if you open one up!

[cczhu]

@choldgraf sure thing, opened PR #225 with the proposed fix.

[choldgraf]

I believe this is now fixed with @cczhu 's patches, thanks!

[Racooneer]

I hope I can reopen the issue this way...

I am unable to convert any book on windows with the similar issue as above

Running Sphinx v2.4.3

Encoding error:
'charmap' codec can't encode character '\uf06a' in position 3835: character maps to <undefined>
The full traceback has been saved in C:\Users\...\AppData\Local\Temp\sphinx-err-2mladg7m.log, if you want to report the issue to the developers.
Traceback (most recent call last):
  File "c:\programdata\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\Scripts\jupyter-book.exe\__main__.py", line 7, in <module>
  File "c:\programdata\anaconda3\lib\site-packages\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "c:\programdata\anaconda3\lib\site-packages\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "c:\programdata\anaconda3\lib\site-packages\click\core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\programdata\anaconda3\lib\site-packages\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\programdata\anaconda3\lib\site-packages\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "c:\programdata\anaconda3\lib\site-packages\jupyter_book\commands\__init__.py", line 133, in build
    "There was an error in building your book. "
  File "c:\programdata\anaconda3\lib\site-packages\jupyter_book\utils.py", line 65, in _error
    raise kind(box)
ValueError:
===============================================================================

There was an error in building your book. Look above for the error message.

===============================================================================

The last thing I tried to test, whether the issue depends on my .ipynb files:

1. uninstall of jupyter-book and clean install
   pip install -U "jupyter-book>=0.7.0b"

2. create book
   jupyter-book create 98_testbook

3. created an ipynb with one markdown cell that contains the words "I am empty"

4. try to build it
   jupyter-book build 98_testbook

This produced the same error message as above as for any other included ipynb I tried.

Before I did this, one thing was to just pip install jupyter book and this produced the error that I should upgrade it. But at least, it let me successfully run jupyter-book page mynb.ipynb. This does not work anymore either.

Thanks, I appreaciate any help :)

[phaustin]

Here's a section on the current status for windows users: tldr -- you need to set an environment variable PYTHONUTF8=1 in either powershell or cmd.exe, then run jupyter-book:
https://github.com/phaustin/jupyter-book/blob/windows/docs/advanced/advanced.md#working-on-windows