Spyder: runtests.py attempts to collect tests from subrepos if extra arguments are given

[mrclary]

Problem Description

Running pytests with extra arguments results in collection error.
Recommend updating runtests.py to add 'spyder' in front of extra_args

What steps reproduce the problem?

1. run python runtests.py -k 'some_single_test'

What is the expected output? What do you see instead?

Should run single test, but instead attempts to load tests from subrepos in 'external-deps'.

Paste Traceback/Error Below (if applicable)

_________________________________________________ ERROR collecting test session __________________________________________________
Defining 'pytest_plugins' in a non-top-level conftest is no longer supported because it affects the entire directory tree in a non-explicit way.
  /Users/rclary/Documents/Python.git/developer/spyder/external-deps/python-language-server/test/conftest.py
Please move it to a top level conftest file at the rootdir:
  /Users/rclary/Documents/Python.git/developer/spyder
For more information, visit:
  https://docs.pytest.org/en/latest/deprecations.html#pytest-plugins-in-non-top-level-conftest-files
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================== 1 error in 0.38 seconds =====================================================

[goanpeca]

Thanks for the report @mrclary, indeed something we need to address.

[ccordoba12]

Thanks for noticing @mrclary! This is very odd indeed.

@CAM-Gerlach, any ideas on how we could fix this?

[mrclary]

It works fine if I include 'spyder' in the arguments, i.e.

$ python runtests.py 'spyder' -k 'some_single_test'

Could it be as simple as always including 'spyder' in the pytest_args?

def run_pytest(run_slow=False, extra_args=None):
    """Run pytest tests for Spyder."""
    pytest_args = ['-vv', '-rw', '--durations=10']

    if CI:
        # Exit on first failure and show coverage
        pytest_args += ['-x', '--cov=spyder', '--no-cov-on-fail']

        # To display nice tests resume in Azure's web page
        if os.environ.get('AZURE', None) is not None:
            pytest_args += ['--cache-clear', '--junitxml=result.xml']
    if run_slow or RUN_SLOW:
        pytest_args += ['--run-slow']

    pytest_args += ['spyder']  # added line
    # Allow user to pass a custom test path to pytest to e.g. run just one test
    if extra_args:
        pytest_args += extra_args
    # else:
    #     pytest_args += ['spyder']
    print("Pytest Arguments: " + str(pytest_args))
    errno = pytest.main(pytest_args)

    # sys.exit doesn't work here because some things could be running in the
    # background (e.g. closing the main window) when this point is reached.
    # If that's the case, sys.exit doesn't stop the script as you would expect.
    if errno != 0:
        raise SystemExit(errno)

[ccordoba12]

Perhaps we should simply remove the -k flag when passing it to runtests.py?

[CAM-Gerlach]

@mrclary This was the way it was originally, before ##11704, but unless there is something I'm missing (which there might be) this prevents the user from specifying a particular spyder subpackage or module to run tests in (as I often do, when I just want to collect and test the module I'm working on rather than everything). That was one of the motivating reasons for the change.

The presumption here is that if a developer is passing custom pytest args beyond what Spyder defines, they are responsible for passing the appropriate collection directory as well...we could parse the argv and determine if a positional arg is passed and if not insert spyder, but that seems like overkill (and a chance for yet more edge case bugs) just for functionality in a script used by a small proportion of active Spyder developers.

[mrclary]

@CAM-Gerlach , I see. The real issue here is that pytest attempts to collect tests in 'external-deps', which it should not. Is there a way to just exclude that directory? Then 'spyder' wouldn't be needed at all.

[CAM-Gerlach]

Perhaps we should simply remove the -k flag when passing it to runtests.py?

I'm not sure how I understand how stripping the user's explicitly passed arguments is intended to fix the problem, sorry.

[CAM-Gerlach]

@mrclary We could use Pytest's --ignore=path option; see the docs. Does that fix the problem for you?

[mrclary]

Aahh, yes, of course. There is an --ignore=<path>; we can just add that. It can be used multiple time, so if the user supplies one, its okay, but always ignore the 'external-deps' directory.

[mrclary]

Great minds think alike? At the same time?

[ccordoba12]

There is an --ignore=<path>

This is a great idea! We could simply inject that option like this --ignore=./external-deps.

Who's interested in giving us a hand and creating a PR for that?

[mrclary]

I can do that.

[ccordoba12]

Great! Thanks a lot for your help Ryan!

[CAM-Gerlach]

Thanks so much for all your help @mrclary !