Pytest: Unconditional exclude of __pycache__ broke pytest-remove-stale-bytecode

[sallner]

Here's a quick checklist in what to include:

• [X] Include a detailed description of the bug or suggestion

With pytest-dev/pytest#4250 the __pycache__ directories are skipped during the collection phase to improve performance. Since then the plugin pytest-remove-stale-bytecode (history pytest-dev/pytest#590) has not been working anymore.

As the skipping is performed outside the pytest_ignore_collect it is also not easily possible to change the behaviour or to configure it somehow.

The package will be put in inactive mode for now, as this seems to be too much of a change for this plugin. If the interest in this plugin is reasonable high, this can be changed of course.

What is your opinion to that? Do you see any other option to get rid of the cache between test runs?

• [ ] pip list of the virtual environment you are using
  
  
  
  • This is more conceptional and not needed.
  
  

• [X] pytest and operating system versions:

  • Pytest >= 3.10, Linux and Mac

• [X] Minimal example if possible

  • Run the tests of pytest-remove-stale-bytecode at this commit 77bc935b014c

[nicoddemus]

Hi @sallner!

As the skipping is performed outside the pytest_ignore_collect it is also not easily possible to change the behaviour or to configure it somehow.

If we move the skipping code into that hook, it would not really help the plugin, right? After all the hook only allows to skip directories, not un-skip them.

Perhaps the plugin can be changed to use a hoowrapper around pytest_collect_file and remove a .pyc file if found for the path that is about to be collected?

Just took a look at the plugin code (https://bitbucket.org/gocept/pytest-remove-stale-bytecode/src/b1a91f05f9349671968cf88323437e7d5b3e4493/plugin.py?at=default&fileviewer=file-view-default) and now I understand the problem better, sorry.

Not sure if I have another suggestion though...

[blueyed]

As for cleaning, I only do this seldomely, e.g. if there is an issue with different paths when running tests inside/outside of Docker, and have a py-clean alias for that: alias py-clean=find -name "*.pyc" -delete.
I would not like the overhead of an extra plugin running all the time for this - but have not checked it out really.

[nicoddemus]

Can we close this or would you rather leave it open @sallner?

[Zac-HD]

Closing the issue as inactive.

[wosc]

At least for our projects, deleting _all_ pyc files (as has been suggested by several people in the context of this topic) incurs a staggering performance overhead (e.g. the runtime of pytest --collect-only increases by 100%, from 6s to 12s), and thus is not something we want to do all the time. And requiring developers to _think about_ whether they should it definitely is not a feasible solution, either. ;)

Instead I have now proposed an update to the plugin to make it work with current pytest versions. (The performance overhead of walking all pyc files one additional time has been negligible for me.)

[RonnyPfannschmidt]

as far as i can tell the plugin is inactive, im not sure reactivating is acceptable without anyone taking ownership and responsibility

[wosc]

The fine folks at gocept have revived the plugin and released version 4.0

[RonnyPfannschmidt]

They are awesome

[blueyed]

@wosc
A link/reference might be useful - https://bitbucket.org/gocept/pytest-remove-stale-bytecode/pull-requests/3/make-work-with-current-pytest-versions/diff was not updated apparently?

[wosc]

@blueyed https://pypi.org/project/pytest-remove-stale-bytecode/ :)