pytest hangs if it tries to rewrite modules without having access rights on Windows 10

[Sup3rGeo]

Basically, it seems that if:

• python is installed in C:\Program files (in a situation that you also have to run pip install from an elevated command prompt in order to install packages)
• pytest runs (as a normal user) for the first time and tries to rewrite assertions, without having elevation

Then pytest just hangs. CTRL+C shows the following traceback:

Traceback (most recent call last):
  File "c:\program files\python36\lib\tempfile.py", line 262, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
PermissionError: [Errno 13] Permission denied: 'c:\\program files\\python36\\lib\\site-packages\\typhoon\\__pycache__\\tmpu12vgv2e'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files\python36\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\program files\python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python36\Scripts\pytest.exe\__main__.py", line 9, in <module>
  File "c:\program files\python36\lib\site-packages\_pytest\config\__init__.py", line 55, in main
    config = _prepareconfig(args, plugins)
  File "c:\program files\python36\lib\site-packages\_pytest\config\__init__.py", line 200, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "c:\program files\python36\lib\site-packages\pluggy\hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "c:\program files\python36\lib\site-packages\pluggy\manager.py", line 87, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\program files\python36\lib\site-packages\pluggy\manager.py", line 81, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "c:\program files\python36\lib\site-packages\pluggy\callers.py", line 203, in _multicall
    gen.send(outcome)
  File "c:\program files\python36\lib\site-packages\_pytest\helpconfig.py", line 89, in pytest_cmdline_parse
    config = outcome.get_result()
  File "c:\program files\python36\lib\site-packages\pluggy\callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "c:\program files\python36\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "c:\program files\python36\lib\site-packages\_pytest\config\__init__.py", line 661, in pytest_cmdline_parse
    self.parse(args)
  File "c:\program files\python36\lib\site-packages\_pytest\config\__init__.py", line 869, in parse
    self._preparse(args, addopts=addopts)
  File "c:\program files\python36\lib\site-packages\_pytest\config\__init__.py", line 815, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "c:\program files\python36\lib\site-packages\pluggy\manager.py", line 292, in load_setuptools_entrypoints
    plugin = ep.load()
  File "c:\program files\python36\lib\site-packages\importlib_metadata\__init__.py", line 90, in load
    module = import_module(match.group('module'))
  File "c:\program files\python36\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "c:\program files\python36\lib\site-packages\_pytest\assertion\rewrite.py", line 144, in exec_module
    _write_pyc(state, co, source_stat, pyc)
  File "c:\program files\python36\lib\site-packages\_pytest\assertion\rewrite.py", line 267, in _write_pyc
    with atomicwrites.atomic_write(pyc, mode="wb", overwrite=True) as fp:
  File "c:\program files\python36\lib\contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "c:\program files\python36\lib\site-packages\atomicwrites\__init__.py", line 156, in _open
    with get_fileobject(**self._open_kwargs) as f:
  File "c:\program files\python36\lib\site-packages\atomicwrites\__init__.py", line 172, in get_fileobject
    descriptor, name = tempfile.mkstemp(dir=dir)
  File "c:\program files\python36\lib\tempfile.py", line 344, in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "c:\program files\python36\lib\tempfile.py", line 268, in _mkstemp_inner
    if (_os.name == 'nt' and _os.path.isdir(dir) and
KeyboardInterrupt

I would expect it not to hang, but for it to terminate with a Permission denied message instead.

Using Windows 10, python 3.6.8 and pytest 5.0.0

[nicoddemus]

Hi @Sup3rGeo!

atomicwrites tries to create the temporary directory 'c:\\program files\\python36\\lib\\site-packages\\typhoon\\__pycache__\\tmpu12vgv2e' using tempfile.mkstemp, which in turn calls _mkstemp_inner:

def _mkstemp_inner(dir, pre, suf, flags, output_type):
    """Code common to mkstemp, TemporaryFile, and NamedTemporaryFile."""

    names = _get_candidate_names()
    if output_type is bytes:
        names = map(_os.fsencode, names)

    for seq in range(TMP_MAX):
        name = next(names)
        file = _os.path.join(dir, pre + name + suf)
        try:
            fd = _os.open(file, flags, 0o600)
        except FileExistsError:
            continue    # try again
        except PermissionError:
            # This exception is thrown when a directory with the chosen name
            # already exists on windows.
            if (_os.name == 'nt' and _os.path.isdir(dir) and
                _os.access(dir, _os.W_OK)):
                continue
            else:
                raise
        return (fd, _os.path.abspath(file))

    raise FileExistsError(_errno.EEXIST,
                          "No usable temporary file name found")

The traceback shows it is hanging on the PermissionError handling, which does call _os.access(dir, _os.W_OK) to check if it should keep retrying to create the temporary directory, otherwise it fails. It seems for some reason it is keeping looping there, instead of _os.access(dir, _os.W_OK) returning False and we getting the expected PermissionError.

Strange, unfortunately this is hard to debug. 😕

[sparrowt]

Hopefully #4730 would help (see also #4755)

[Sup3rGeo]

According to this stackoverflow question, it seems this is a long lasting issue for windows that has not been yet fixed:

https://bugs.python.org/issue22107

Let's hope the items mentioned by @sparrowt above will help workaround the issue for pytest.

[nicoddemus]

Let's hope the items mentioned by @sparrowt above will help workaround the issue for pytest.

It would definitely help, but only for py38 users I'm afraid... 🤔

[Blubbaa]

sadly this issue still exists on Windows 10, Python 3.8.6, pytest 6.1.2 😖

[nicoddemus]

I guess a workaround would be to check for permission before calling into atomic_write.

I've spent a few minutes trying to write a test for it:

@pytest.mark.skipif(not sys.platform.startswith("win"), reason="Windows only (#5844)")
def test_write_pyc(tmp_path: Path, request) -> None:
    cache_dir = tmp_path.joinpath("cache")
    cache_dir.mkdir()

    try:
        cache_dir.chmod(0)  # this doesn't work
        with pytest.raises(PermissionError):
            cache_dir.joinpath("foo").touch()
        # call _write_pyc and check it returns False instead of hanging
    finally:
        cache_dir.chmod(stat.S_IWRITE)

But unfortunately couldn't get cached_dir to throw a PermissionError for some reason. I'm leaving this here in case someone has some tips, as I'm short on time right now to investigate further. 👍