Conda-build: Windows Python 2.7 build issues with pip

[jakirkham]

Seeing a build issue with Sphinx on Windows Python 2.7, but not Python 3 or on other platforms with either Python. Appears some file is missing only on this platform and Python version. Not really sure what is going on.

cc @chohner

Actual Behavior

> python -m pip install --no-deps --ignore-installed . 
Processing c:\bld\sphinx_1527592727519\work
Exception:
Traceback (most recent call last):
  File "C:\bld\sphinx_1527592727519\_h_env\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:\bld\sphinx_1527592727519\_h_env\lib\site-packages\pip\commands\install.py", line 335, in run
    wb.build(autobuilding=True)
  File "C:\bld\sphinx_1527592727519\_h_env\lib\site-packages\pip\wheel.py", line 756, in build
    self.requirement_set.prepare_files(self.finder)
  File "C:\bld\sphinx_1527592727519\_h_env\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "C:\bld\sphinx_1527592727519\_h_env\lib\site-packages\pip\req\req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "C:\bld\sphinx_1527592727519\_h_env\lib\site-packages\pip\download.py", line 809, in unpack_url
    unpack_file_url(link, location, download_dir, hashes=hashes)
  File "C:\bld\sphinx_1527592727519\_h_env\lib\site-packages\pip\download.py", line 686, in unpack_file_url
    shutil.copytree(link_path, location, symlinks=True)
  File "C:\bld\sphinx_1527592727519\_h_env\lib\shutil.py", line 222, in copytree
    raise Error, errors
Error: [('C:\\bld\\sphinx_1527592727519\\work\\tests\\roots\\test-images\\testima\xa8ge.png', 'c:\\users\\appveyor\\appdata\\local\\temp\\1\\pip-bjlv99-build\\tests\\roots\\test-images\\testima\xa8ge.png', "[Errno 2] No such file or directory: 'C:\\\\bld\\\\sphinx_1527592727519\\\\work\\\\tests\\\\roots\\\\test-images\\\\testima\\xa8ge.png'")]

ref: https://ci.appveyor.com/project/conda-forge/sphinx-feedstock/build/1.0.99/job/0ieygt5nuioessp9

Expected Behavior

Nothing is missing from the source tree.

Steps to Reproduce

PR ( https://github.com/conda-forge/sphinx-feedstock/pull/35 )

Output of conda info

     active environment : base
    active env location : C:\Miniconda36-x64
            shell level : 1
       user config file : C:\Users\appveyor\.condarc
 populated config files : C:\Users\appveyor\.condarc
          conda version : 4.5.4
    conda-build version : 3.10.5
         python version : 3.6.5.final.0
       base environment : C:\Miniconda36-x64  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/win-64
                          https://repo.anaconda.com/pkgs/pro/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Miniconda36-x64\pkgs
                          C:\Users\appveyor\AppData\Local\conda\conda\pkgs
       envs directories : C:\Miniconda36-x64\envs
                          C:\Users\appveyor\AppData\Local\conda\conda\envs
                          C:\Users\appveyor\.conda\envs
               platform : win-64
             user-agent : conda/4.5.4 requests/2.18.4 CPython/3.6.5 Windows/2012ServerR2 Windows/6.3.9600
          administrator : True
             netrc file : None
           offline mode : False

[jakirkham]

FWIW this issue doesn't appear to occur with conda-build 2.

xref: https://github.com/conda-forge/sphinx-feedstock/pull/36
ref: https://ci.appveyor.com/project/conda-forge/sphinx-feedstock/build/1.0.102/job/7fyc3iswfkk9f024

[nehaljwani]

xref: https://github.com/AnacondaRecipes/sphinx-feedstock/commit/4b9be32d9de8b6c264b3bb3282d71b4429fa6046

IIRC, I had seen this too, and we just moved back from pip to using setuptools and it worked.

[jjhelmus]

IIRC pip creates some rather long directories which have a tendency to hit the Windows path length limit, python setup.py install ... is a bit more forgiving with the path length.

[jakirkham]

cc @isuruf @ocefpaf (for awareness)

[jakirkham]

Though it does seem interesting that using pip with conda-build 2 does not run into the issue. Any idea what the difference is?

[jakirkham]

Do you know if there is pip flag to affect the path length?

[ocefpaf]

Looks like a pip bug IMO. @jjhelmus did you raise this upstream? Do you want me to do it?

[isuruf]

It's not the path length. Filename has non-ASCII characters and you have to be careful with python 2.7.

[jakirkham]

Where are the non-ASCII characters?

[isuruf]

\xa8 in testima\xa8ge.png

[jakirkham]

Sorry saw a \ on Windows and just thought it was part of the path. Thanks for clarifying that.

[jakirkham]

This PR ( https://github.com/pypa/pip/pull/4486 ) looks promising. It landed in pip version 10.0.0. We could backport it or we could try upgrading. The latter means we need to try disabling build isolation ( https://github.com/pypa/pip/issues/5033 ). There should be an option to do this, but IDK if anyone has explored it.

[jakirkham]

Tried a few things in PR ( https://github.com/conda-forge/sphinx-feedstock/pull/38 ), but nothing seems to work. 😞

[jakirkham]

Seeing this again with python-libarchive-c. Same story as before. Only happens on Windows Python 2.7.

xref: https://github.com/conda-forge/python-libarchive-c-feedstock/issues/8

cc @marscher

[mingwandroid]

File is: Sphinx-1.7.5/tests/roots/test-images/testimäge.png, looks like some unicode thing.

[mingwandroid]

While not suggesting it's not important to fix this, why is python 2 being used in the base environment anyway? I've never (except when looking into a bug) run conda-build under python 2. I don't see any reason to do that.

[isuruf]

Base environment is python 3.6

[jakirkham]

Correct. Only the build is using Python 2.

There is probably some special combination of Windows, encodings, and legacy unresolved Python issues.

IME switching to setuptools instead of pip usually fixes this issue (as it does with python-libarchive-c). So it may be something wrong with pip itself. Or it could be pip is using part of the Python 2 standard library that doesn't behave well on Windows with UTF-8 characters in this case.

[mingwandroid]

For me, our sphinx package builds fine for python 2.7. We pass -vvv to pip I guess verbose prevents the progress-bar?

[jakirkham]

Think the progress bar point was a different issue I confused with this one. The python-libarchive-c build uses -vvv and still fails the same way.

Are there some other environment variables set in your builds? Wonder how your builds are avoiding this encoding error.