Pytest: Add something like: "--newest-first"

[jedie]

There is --failed-first ... what's about a --newest-first: Execute the tests from the newest test file first. Sort by file change timestamp.

The idea: If i add a new test, then i would like to see if this new test will be failed first ;)

[pytestbot]

GitMate.io thinks the contributors most likely able to help are @nicoddemus, and @RonnyPfannschmidt.

[nicoddemus]

Thanks @jedie for the suggestion, we appreciate you taking the time to write.

Personally I don't see much value in that option to be honest; when I add a new test file I usually execute it immediately by passing its filename in the command-line, and I never add a bunch of test files and then try to execute the whole suite but would like to see the newly added files first.

This however should be dead-simple to add and is an issue good for a first time contributor, so I don't have anything against adding it to the core. 😉

Perhaps do you have a more specific use case which that option would help that I'm not seeing though?

[RonnyPfannschmidt]

@nicoddemus perhaps this is about adding new items in existing files, its not clear how to implement this then, and certainly not easy

[nicoddemus]

@nicoddemus perhaps this is about adding new items in existing files, its not clear how to implement this then, and certainly not easy

If that's the case then I agree it seems more useful, although the original text is "Execute the tests from the newest test file first". @jedie can you clarify please?

[RonnyPfannschmidt]

as far as i can tell this one can be done as a plugin - so perhaps thats a good place to experiment with different expressions oft he feature

[jedie]

Sorry for the bad English;)

I mean, the latest added/modified tests should be run first.

We can check the file modify date and maybe we can look for new test names: Each pass saves a list of all tests. When new tests are added, they are sorted first.

Maybe the order can look like this:

1. "--failed-first" tests
2. new test names
3. tests from newest modified files
4. all other tests

[blueyed]

btw: in this regard it would be useful if there would be a cache for the collected items (test ids with files and their timestamps).
This could speed up the collection in general (only modified and new files would need to be considered), and could be used by a plugin that implements --newest-first etc then.

[nicoddemus]

Hey @blueyed,

in this regard it would be useful if there would be a cache for the collected items (test ids with files and their timestamps).

You mean add this to the core? I'm not sure, while it would be useful in this special case, I suspect different plugins would have different needs as to what store in the cache (a plugin might be interested to store git hashes instead of timestamps for example). I don't think it is possible to store something that will be useful for plugins in a general manner.

Keep in mind that each plugin can easily save the information on cache themselves, in the exact manner they want.

This could speed up the collection in general

Hmm not sure, why would that speed up the collection? We still need to import the modules and create the items as we can't save those on the cache.

[blueyed]

why would that speed up the collection

Using -k foo might then be able to only import the file defining the matching test items (without collecting everything before) - only checking mtime of all others.

[nicoddemus]

Oh when -k is used, got it, thanks!

[feuillemorte]

I think we can write another plugin like this:

https://github.com/pytest-dev/pytest/blob/3685c1bc019cb119378ebfb31993a1748548e731/_pytest/cacheprovider.py#L101-L103

for new tests. But I can only understand when file was modified and start to run tests first from this file. But I don't know how to check which exactly test was modified. So, if test file contains 50 tests these tests will collect in default order

[nicoddemus]

But I can only understand when file was modified and start to run tests first from this file.

Certainly, but I think that's what the OP has in mind as well.

[feuillemorte]

Ok, then I try to do it :)

[feuillemorte]

Merged