Pip: Tests don't work on Windows

[pfmoore]

The pip test suite doesn't really work on Windows at the moment. There are probably a lot of OS-dependent assumptions in there - it is difficult to tell because the tests are rarely, if ever, run on Windows. It would be nice, however, have cross-platform tests.

Without a Windows CI service, it will be difficult to stop the tests failing on Windows over time, so this may be something to defer until a CI service is available. Appveyor may be a suitable interim service, but I don't believe its performance is sufficient to act as a long-term CI solution.

[msabramo]

Most of the test_freeze tests pass now; the one failure is due to #2323, which @pfmoore is already working on I believe.

(pip) C:\Users\Administrator\Development\Python\pip>tox -e py27 -- tests\functional\test_freeze.py --tb=short
GLOB sdist-make: C:\Users\Administrator\Development\Python\pip\setup.py
py27 inst-nodeps: C:\Users\Administrator\Development\Python\pip\.tox\dist\pip-6.1.0.dev0.zip
py27 runtests: PYTHONHASHSEED='229'
py27 runtests: commands[0] | py.test --timeout 300 tests\functional\test_freeze.py --tb=short
============================= test session starts =============================
platform win32 -- Python 2.7.5 -- py-1.4.26 -- pytest-2.6.4
plugins: capturelog, cov, timeout, xdist
collected 8 items

tests/functional/test_freeze.py ....F...

================================== FAILURES ===================================
__________________________ test_freeze_bazaar_clone ___________________________
tests\functional\test_freeze.py:230: in test_freeze_bazaar_clone
    _check_output(result, expected)
tests\functional\test_freeze.py:35: in _check_output
    assert checker.check_output(expected, actual, ELLIPSIS), (
E   AssertionError:
E     ========== EXPECTED ==========
E     Script result: ...pip freeze
E     -- stdout: --------------------
E     ...-e bzr+file://c:\users\admini~1\appdata\local\temp\pytest-44\test_freeze_bazaar_clone0\workspace\scratch\version_pkg@1#egg=
version_pkg-0.1-...
E     ...
E     ========== ACTUAL ==========
E     Script result: pip freeze
E     -- stderr: --------------------
E     Error [Error 2] The system cannot find the file specified while executing command bzr info
E     cannot determine version of editable source in c:\users\admini~1\appdata\local\temp\pytest-44\test_freeze_bazaar_clone0\worksp
ace\scratch\bzr-package (bzr command not found in path)
E
E     -- stdout: --------------------
E     -e version-pkg==0.1
E
E     ========== ====== ==========
E
E   assert <bound method OutputChecker.check_output of <doctest.OutputChecker instance at 0x033440D0>>('Script result: ...pip freeze
\n-- stdout: --------------------\n...-e bzr+file://c:\users\admini~1\appdata\local\temp\pytest-44\test_freeze_bazaar_clone0\workspa
ce\scratch\version_pkg@1#egg=version_pkg-0.1-...\n...', 'Script result: pip freeze\n-- stderr: --------------------\nError [Error 2]
 The system cannot find the file specified...kspace\\scratch\\bzr-package (bzr command not found in path)\n\n-- stdout: ------------
--------\n-e version-pkg==0.1\n', 8)
E    +  where <bound method OutputChecker.check_output of <doctest.OutputChecker instance at 0x033440D0>> = <doctest.OutputChecker i
nstance at 0x033440D0>.check_output
===================== 1 failed, 7 passed in 86.67 seconds =====================
ERROR: InvocationError: 'C:\\Users\\Administrator\\Development\\Python\\pip\\.tox\\py27\\Scripts\\py.test.EXE --timeout 300 tests\\f
unctional\\test_freeze.py --tb=short'
____________________________________________________________ summary _____________________________________________________________
ERROR:   py27: commands failed

[msabramo]

I just found this comment in tests/functional/test_install_upgrade.py:

    # TODO(pnasrat) Using local_checkout fails on windows - oddness with the
    # test path urls/git.

[xavfernandez]

The unit tests now pass on windows on master :)

https://ci.appveyor.com/project/xavfernandez/pip

[msabramo]

Nice work, @xavfernandez!

[pfmoore]

The unit tests now pass on windows on master :)

W00t! Great work @xavfernandez!

[techtonik]

So this can be closed, right?

[xavfernandez]

Still all the functionnal tests missing...

[techtonik]

Finally found how to run them (need better docs):

tox -e py27 -- tests/functional/test_check.py

But no logs are saved. How to make tox save those?

[techtonik]

Okay. py.test doesn't support writing logs, so...

Most likely the failure is Windows limitation - https://stackoverflow.com/questions/1880321/why-does-the-260-character-path-length-limit-exist-in-windows and rejected fix in https://bugs.python.org/issue18199

[pradyunsg]

@pfmoore Would the final objective be to also run integration tests on AppVeyor?

[pfmoore]

I guess so, although my goal for this issue was simply to have a test infrastructure for Windows in place.

My inclination is to close this issue as complete. If someone wants to work on getting the functional and integration tests running on Windows, then maybe open a new issue focused on that. Personally, I don't have the time to commit to that, though.

[techtonik]

I ditched Windows from my life.

[pradyunsg]

Well, thanks to @benoit-pierre we're running all tests on Windows now. :)