Pytest: Prompt user for input during test run

[rogerdahl]

From the docs:

In addition, stdin is set to a “null” object which will fail on attempts to read from it because it is rarely desired to wait for interactive input when running automated tests.

Well, I find myself really desiring the rarely desired ability to wait for interactive input during test runs. I have a pytest based test system that I think is rather nifty, that automatically serializes test output from new tests to files. During regular pytest runs, the test output is compared to the files and tests fail on mismatches. However, I have a mode I can enable with a pytest switch that doesn't immediately fail tests on mismatches. Instead, it pops up a diff and (here's the issue), asks the user if the test should be failed, or if it should just update the reference file to match and keep going.

For my use case, this setup has saved me from writing lots of asserts and made for better tests at the same time. But for now, it only works with --capture=no. I'm aware of the capture fixtures that allow temporarily disabling capturing. Those would probably work for this, but I have maybe 1000 tests that use this sample file based system and I'm looking for a less invasive way. I don't want to add a capture fixture argument to each of the test function, and then having to pass it along into the helper functions that eventually issue the user input prompt.

I think the ideal solution would be to have a context manager that I could import directly from the pytest module and wrap the user input function with. Is anything like that available? If not, any tips on how to get this working would be much appreciated :)

[blueyed]

I don't want to add a capture fixture argument to each of the test function, and then having to pass it along into the helper functions that eventually issue the user input prompt.

You could try using an autouse fixture.

But see also https://github.com/pytest-dev/pytest/issues/4210, which mentions a context manager - but from when I've tried it it did not work at the module level IIRC.

Apart from that I've experimented with disabling capturing automatically when stdin is being read in https://github.com/pytest-dev/pytest/pull/4996 - maybe you want to try that?

[Zac-HD]

I would just design this switch to always overwrite the reference file in question, and then accept or discard each via Git (or your VCS of choice).

[rogerdahl]

I would just design this switch to _always_ overwrite the reference file in question, and then accept or discard each via Git (or your VCS of choice).

@Zac-HD It's an interesting idea, but ultimately, I think it would be less practical for me. I prefer to deal with changes on a one by one basis, which allows me to exit out of the tests right away if there are any unexpected changes, instead of having to later go through and roll back a large block of changes which may be intermixed with expected changes.

[rogerdahl]

@blueyed Thank you for the pointers.

As far as I understand, an autouse fixture wouldn't work for this, since it's not bound to a name that can be referenced in the test or helper modules.

I agree with your conclusion on the context manager mentioned in #4210.

As for disabling the capture automatically when stdin is being read, your PR does seem fairly involved, so I can understand the reluctance of maintainers to merge it (since it could reduce the maintainability of the project as a whole). If it's not going to be merged, my specific use case does not really warrant keeping a fork for it, since having the debug logging scroll by is mainly a cosmetic issue.

One idea that I thought might work but haven't tried yet, is to disable setting the stdin fd to null in a monkey patch from a conftest file and take advantage of pytest's capability of capturing only at log level and not file descriptor level. User prompts that are written to stdout or with print() should then appear, while all the logging is still captured. If that works, maybe the maintainers could be convinced to include a switch that disables setting the stdin fd to null, since that seems like it would be low impact.

Another idea is to spin up another shell for the user prompts, since I think that would get its own working fds for stdin / out. Along those same lines, one might be able to pop up a GUI prompt in X.

[blueyed]

As far as I understand, an autouse fixture wouldn't work for this, since it's not bound to a name that can be referenced in the test or helper modules.

You can get it via request.getfixturevalue() I would assume. But the whole idea would be that you do not really need it maybe (sorry not really up to date with regard to previous comments).

4996

As for disabling the capture automatically when stdin is being read, your PR does seem fairly involved, so I can understand the reluctance of maintainers to merge it …

That wasn't the question, I just wanted your feedback on if it works for your use case, i.e. to try the PR/branch locally. Have you?

include a switch that disables setting the stdin fd to null, since that seems like it would be low impact.

Well, it would make tests hang that request/use stdin - that's the main reason for setting it to null in the first place.

[rogerdahl]

@blueyed I pulled capture-resume-on-stdin-read and tested it locally. It works nicely in my test env, with

'pytest-cov >= 2.6.1',
'pytest-django >= 3.4.8',
'pytest-forked >= 1.0.2',
'pytest-random-order >= 1.0.4',
'pytest-xdist >= 1.28.0',
'freezegun >= 0.3.11',

[rogerdahl]

Did some tests, and my use case is resolved by using --capture=sys and a custom input() function that uses the file descriptors:

def fd_input(prompt):
    with os.fdopen(os.dup(1), "w") as stdout:
        stdout.write("\n{}? ".format(prompt))

    with os.fdopen(os.dup(2), "r") as stdin:
        return stdin.readline()

[b0g3r]

I have the same question, but I solved it in a different manner (via CaptureManager):

capture = self.request.config.pluginmanager.getplugin("capturemanager")
capture.suspend_global_capture(in_=True)
inp = input('whatsup?')
capure.resume_global_capture()

[rogerdahl]

@b0g3r I realized that I didn't need an input prompt. When there is a mismatch, I open a diff tool (I'm currently using meld), showing current result on one side and expected result on the other. If I want to update the expected result, I just do a Merge all from left in meld). I use a global lock on opening meld so that only one instance is opened at a time. Works nicely also when running the tests in parallel with xdist.