pytest2.8 invariantly writes to working directory + fails on readonly filesystem

[asottile]

I believe these to be the same issue so I'm only making a single report.

$ virtualenv venv
...
$ . venv/bin/activate
$ pip install pytest
...
$ py.test foo.py
============================= test session starts ==============================
platform linux2 -- Python 2.7.6, pytest-2.8.0, py-1.4.30, pluggy-0.3.1
rootdir: /tmp/foo, inifile: 

===============================  in 0.00 seconds ===============================
ERROR: file not found: foo.py
(venv)asottile@work:/tmp/foo$ ls -al .cache/v/cache/lastfailed 
-rw-rw-r-- 1 asottile asottile 2 Sep 20 17:03 .cache/v/cache/lastfailed

And the error from our CI server (which is running a repo inside docker)

15:06:43 docker run -t -v /nail/scratch/jenkins_prod_slave/workspace/packages-ubuntu-allocate_playground:/work:ro nginx_test_container /bin/bash -c "cd /work/itest && . /venv34/bin/activate && py.test -s test_nginx.py"
15:06:43 ============================= test session starts ==============================
15:06:43 platform linux -- Python 3.4.2, pytest-2.8.0, py-1.4.30, pluggy-0.3.1
15:06:43 rootdir: /work, inifile: 
15:06:43 
collecting 0 items
collecting 5 items
collected 5 items 
15:06:43 
15:06:44 test_nginx.py
...
15:06:51 .Traceback (most recent call last):
15:06:51   File "/venv34/lib/python3.4/site-packages/py/_error.py", line 64, in checked_call
15:06:51     return func(*args, **kwargs)
15:06:51 OSError: [Errno 30] Read-only file system: '/work/.cache/v/cache/lastfailed'
15:06:51 
15:06:51 During handling of the above exception, another exception occurred:
15:06:51 
15:06:51 Traceback (most recent call last):
15:06:51   File "/venv34/bin/py.test", line 11, in <module>
15:06:51     sys.exit(main())
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/config.py", line 48, in main
15:06:51     return config.hook.pytest_cmdline_main(config=config)
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 724, in __call__
15:06:51     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
15:06:51     return self._inner_hookexec(hook, methods, kwargs)
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
15:06:51     _MultiCall(methods, kwargs, hook.spec_opts).execute()
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 596, in execute
15:06:51     res = hook_impl.function(*args)
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/main.py", line 115, in pytest_cmdline_main
15:06:51     return wrap_session(config, _main)
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/main.py", line 110, in wrap_session
15:06:51     exitstatus=session.exitstatus)
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 724, in __call__
15:06:51     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
15:06:51     return self._inner_hookexec(hook, methods, kwargs)
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
15:06:51     _MultiCall(methods, kwargs, hook.spec_opts).execute()
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 595, in execute
15:06:51     return _wrapped_call(hook_impl.function(*args), self.execute)
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 249, in _wrapped_call
15:06:51     wrap_controller.send(call_outcome)
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/terminal.py", line 361, in pytest_sessionfinish
15:06:51     outcome.get_result()
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 278, in get_result
15:06:51     raise ex[1].with_traceback(ex[2])
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 264, in __init__
15:06:51     self.result = func()
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 596, in execute
15:06:51     res = hook_impl.function(*args)
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/cacheprovider.py", line 140, in pytest_sessionfinish
15:06:51     config.cache.set("cache/lastfailed", self.lastfailed)
15:06:51   File "/venv34/lib/python3.4/site-packages/_pytest/cacheprovider.py", line 73, in set
15:06:51     with path.open("w") as f:
15:06:51   File "/venv34/lib/python3.4/site-packages/py/_path/local.py", line 353, in open
15:06:51     return py.error.checked_call(open, self.strpath, mode)
15:06:51   File "/venv34/lib/python3.4/site-packages/py/_error.py", line 84, in checked_call
15:06:51     raise cls("%s%r" % (func.__name__, args))
15:06:51 py.error.EROFS: [Read-only file system]: open('/work/.cache/v/cache/lastfailed', 'w')
15:06:51 make: *** [itest] Error 1
...

[nicoddemus]

Thanks for the report! :smile:

Hmmm I see two possibilities:

1. Every call to config.cache.set will internally catch read-only errors, and silently fail in this case.
2. The internal cache plugin will explicitly capture errors during write, and print a pytest warning.
3. Add an option to disable lf-plugin.

I think 1 is too error prone, and 3 will remove much of the usefulness of the --lf option, because I usually always want it active when I'm running tests (and will always forget to pass --enable-lf). I think 2 is perhaps more appropriate, as the internal cache is now always enabled it perhaps should be extra careful for cache writing failures.

[RonnyPfannschmidt]

2 is how it should work, since we already made the warning recording a historic hook call, we can propperly issue a config.warn in such cases

[asottile]

The fix only addresses half of the problem. How do I get pytest to stop writing to my working directory and why did this change happen?

[bukzor]

The standard place to put this is ~/.cache.
To easily disambiguate caches against various $PWD's, you can mkdir -p $HOME/.cache/pytest/$PWD.
Even better would be to use $XDG_CACHE_HOME with a default value of $HOME/.cache.

[nicoddemus]

@asottile

How do I get pytest to stop writing to my working directory and why did this change happen?

I think the only way to prevent that currently is by passing -p no:cacheprovider in the command line, or by modifying your pytest.ini:

[pytest]
addopts = -p no:cacheprovider

(_In fact I will add this to the docs_)

This change happened because in 2.8 pytest-cache has been merged into the core. It brings with two important functionalities: re-run last failures or failures first and config.cache object, which lets plugins persist data between test sessions.

One of the drawbacks is that now pytest will always create a .cache directory in the rootdir of the test session, as the cache has to be active in order to --lf do its work. We did not foreseen the problems it could cause like your issue demonstrated, unfortunately (which already has been addressed in #1048, btw).

This will be fixed in 2.8.1 when it comes out (soon), but until then you can disable the cache completely as I noted above.

[nicoddemus]

@bukzor

The standard place to put this is ~/.cache. To easily disambiguate caches against various $PWD's, you can mkdir -p $HOME/.cache/pytest/$PWD. Even better would be to use $XDG_CACHE_HOME with a default value of $HOME/.cache

That's an excellent suggestion I think. Would you mind opening a new issue with it? Thanks! :smile:

[RonnyPfannschmidt]

we could probably use a hash + use a symlink back - using full paths is a huge error source due to path length limits on various platforms

[nicoddemus]

using full paths is a huge error source due to path length limits on various platforms

That's a good point! I know I've had my problems with this on Windows. :sweat_smile:

But I fear adding another layer of complexity to the system might bring more trouble than it is worth, to be honest. :grimacing:

[RonnyPfannschmidt]

i feel the need to make basetmp as well as cache locations configurable
we should brainstorm that in some way (since it also makes sense to put the cache + basetmp into something a CI system can easyly pick up for example

plus XDG_CONFIG_DIRS is the absolutely wrong thing to use in many non-containered ci situations

[bukzor]

@RonnyPfannschmidt That's wrongheaded on several counts.

a) A very convenient, and standard, way to configure tools' cache location is to set $XDG_CACHE_DIR. Many of my project's fixtures do this for other reasons already. Similarly, the most convenient, and standard, way to set "basetmp" would be $TMPDIR.

b) "something a CI system can easily pick up" is exactly these environment variables. Many systems will already support this, and those that don't will support injecting environment variables.

c) "non-containered ci" will either support ~/.cache correctly, or set $XDG_CACHE_DIR, because this is necessary for other tools that use the standard. Again, this is already handled in many of my projects because of other tools.

[bukzor]

moved to #1089

[asottile]

This has regressed (2.8.2):

15:13:09 .Traceback (most recent call last):
15:13:09   File "/venv34/lib/python3.4/site-packages/py/_error.py", line 64, in checked_call
15:13:09     return func(*args, **kwargs)
15:13:09 OSError: [Errno 30] Read-only file system: '/work/.cache/v/cache/lastfailed'
15:13:09 
15:13:09 During handling of the above exception, another exception occurred:
15:13:09 
15:13:09 Traceback (most recent call last):
15:13:09   File "/venv34/bin/py.test", line 11, in <module>
15:13:09     sys.exit(main())
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/config.py", line 48, in main
15:13:09     return config.hook.pytest_cmdline_main(config=config)
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 724, in __call__
15:13:09     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
15:13:09     return self._inner_hookexec(hook, methods, kwargs)
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
15:13:09     _MultiCall(methods, kwargs, hook.spec_opts).execute()
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 596, in execute
15:13:09     res = hook_impl.function(*args)
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/main.py", line 115, in pytest_cmdline_main
15:13:09     return wrap_session(config, _main)
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/main.py", line 110, in wrap_session
15:13:09     exitstatus=session.exitstatus)
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 724, in __call__
15:13:09     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
15:13:09     return self._inner_hookexec(hook, methods, kwargs)
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
15:13:09     _MultiCall(methods, kwargs, hook.spec_opts).execute()
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 595, in execute
15:13:09     return _wrapped_call(hook_impl.function(*args), self.execute)
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 249, in _wrapped_call
15:13:09     wrap_controller.send(call_outcome)
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/terminal.py", line 361, in pytest_sessionfinish
15:13:09     outcome.get_result()
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 278, in get_result
15:13:09     raise ex[1].with_traceback(ex[2])
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 264, in __init__
15:13:09     self.result = func()
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 596, in execute
15:13:09     res = hook_impl.function(*args)
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/cacheprovider.py", line 152, in pytest_sessionfinish
15:13:09     config.cache.set("cache/lastfailed", self.lastfailed)
15:13:09   File "/venv34/lib/python3.4/site-packages/_pytest/cacheprovider.py", line 80, in set
15:13:09     f = path.open('w')
15:13:09   File "/venv34/lib/python3.4/site-packages/py/_path/local.py", line 353, in open
15:13:09     return py.error.checked_call(open, self.strpath, mode)
15:13:09   File "/venv34/lib/python3.4/site-packages/py/_error.py", line 84, in checked_call
15:13:09     raise cls("%s%r" % (func.__name__, args))
15:13:09 py.error.EROFS: [Read-only file system]: open('/work/.cache/v/cache/lastfailed', 'w')
15:13:09 make: *** [itest] Error 1

[The-Compiler]

@asottile I don't think this is a regression - it seems like @nicoddemus' fix handles py.error.ENOTDIR only, but the error you get is something different.

I found it odd that you get that during the f = path.open('w') line though, and not earlier (path.dirpath().ensure_dir()).

[RonnyPfannschmidt]

@asottile can you provided more information on the invocation into docker?
I need a way to reproduce this

[asottile]

Here's a oneliner that reproduces:

rm -rf foo && mkdir foo && touch foo/test.py && docker run -ti -v "$PWD/foo:/code:ro" ubuntu bash -c 'apt-get update && apt-get install -y python-pip && pip install pytest && cd /code && py.test test.py'

The trace above can be reproduced by:

rm -rf foo && mkdir foo && touch foo/test.py && mkdir -p foo/.cache/v/cache && docker run -ti -v "$PWD/foo:/code:ro" ubuntu bash -c 'apt-get update && apt-get install -y python-pip && pip install pytest && cd /code && py.test test.py'

You'll probably have a faster debug loop with a dockerfile that looks something like (not tested):

FROM ubuntu
RUN apt-get update && apt-get install -y python-pip
RUN pip install pytest
CMD cd /code && py.test test.py

[RonnyPfannschmidt]

i created a super-seeding issue for that particular issue to keep track of the exact detail thanks for the debugging instructions

[nicoddemus]

Is this still an issue with 2.8.3?

[asottile]

Yes, pasting either oneliner above still triggers a stacktrace under 2.8.3

[svetlin-mladenov]

pytest creates the .cache directory even when no tests are actually executed and the cache functionality is not used at all. In a sense the user pays for what they are not using. Here is an example:

rm .cache/ -rf
rm empty.py
touch empty.py
py.test empty.py # warns that no tests were run
ls -lad .cache # suprise !

PS. Maybe I should file a new issue for this because although related to this issue it's not limited to read-only filesystems.

[RonnyPfannschmidt]

please a new issue

[mgedmin]

The other day I tried to run our test suite in a bubblewrap sandbox, to make sure our tests don't require network connectivity, and don't write to disk unnecessarily:

bwrap --ro-bind / / --dev-bind /dev /dev --tmpfs /tmp --unshare-net py.test

To my surprise py.test did not show me the tracebacks of the failed tests, but instead crashed trying to write the last-failed cache.

Anyway, just wanted to confirm that this issue is still present in py.test 3.2.3, and mention my use-case.

[RonnyPfannschmidt]

nobody actually went to fix it

[nicoddemus]

@mgedmin thanks. Just want to mention a workaround: -p no:cacheprovider will disable the cache plugin and the problem should go away.

[asottile]

Taking the oneliners above (and adapting them for time changes), I get the following:

rm -rf foo && mkdir foo && echo 'def test(): pass' > foo/test.py && docker run -ti -v "$PWD/foo:/code:ro" ubuntu:xenial bash -exc 'apt-get update && apt-get install -y --no-install-recommends python-pip python-setuptools && pip install pytest && cd /code && py.test test.py'

works fine!

rm -rf foo && mkdir -p foo/.cache/v/cache && echo 'def test(): pass' > foo/test.py && docker run -ti -v "$PWD/foo:/code:ro" ubuntu:xenial bash -exc 'apt-get update && apt-get install -y --no-install-recommends python-pip python-setuptools && pip install pytest && cd /code && py.test test.py'

also works fine!

it appears this was fixed at some point, closing.