Aiohttp: pytest-aiohttp + pytest-xdist cause RuntimeError: set_wakeup_fd only works in main thread

[JoseKilo]

Long story short

Combining pytest-aiohttp and pytest-xdist an error is triggered sometimes from the loop fixture.

Let me know if you think the issue is more on the pytest(-xdist) side.

Expected behaviour

No error.

Actual behaviour

=================================================================================================================== ERRORS ====================================================================================================================
_____________________________________________________________________________________________________ ERROR at setup of test_foo[pyloop] ______________________________________________________________________________________________________
[gw6] linux -- Python 3.7.0 /home/merlin/workspace/pytest-test/.venv/bin/python3.7

loop_factory = <class 'asyncio.unix_events._UnixDefaultEventLoopPolicy'>, fast = False, loop_debug = False

    @pytest.fixture
    def loop(loop_factory, fast, loop_debug):
        """Return an instance of the event loop."""
        policy = loop_factory()
        asyncio.set_event_loop_policy(policy)
>       with loop_context(fast=fast) as _loop:

.venv/lib/python3.7/site-packages/aiohttp/pytest_plugin.py:205:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.7/contextlib.py:112: in __enter__
    return next(self.gen)
.venv/lib/python3.7/site-packages/aiohttp/test_utils.py:404: in loop_context
    loop = setup_test_loop(loop_factory)
.venv/lib/python3.7/site-packages/aiohttp/test_utils.py:427: in setup_test_loop
    watcher.attach_loop(loop)
/usr/local/lib/python3.7/asyncio/unix_events.py:868: in attach_loop
    loop.add_signal_handler(signal.SIGCHLD, self._sig_chld)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_UnixSelectorEventLoop running=False closed=False debug=False>, sig = <Signals.SIGCHLD: 17>, callback = <bound method BaseChildWatcher._sig_chld of <asyncio.unix_events.SafeChildWatcher object at 0x7fbf10c9b048>>, args = ()

    def add_signal_handler(self, sig, callback, *args):
        """Add a handler for a signal.  UNIX only.

        Raise ValueError if the signal number is invalid or uncatchable.
        Raise RuntimeError if there is a problem setting up the handler.
        """
        if (coroutines.iscoroutine(callback) or
                coroutines.iscoroutinefunction(callback)):
            raise TypeError("coroutines cannot be used "
                            "with add_signal_handler()")
        self._check_signal(sig)
        self._check_closed()
        try:
            # set_wakeup_fd() raises ValueError if this is not the
            # main thread.  By calling it early we ensure that an
            # event loop running in another thread cannot add a signal
            # handler.
            signal.set_wakeup_fd(self._csock.fileno())
        except (ValueError, OSError) as exc:
>           raise RuntimeError(str(exc))
E           RuntimeError: set_wakeup_fd only works in main thread

/usr/local/lib/python3.7/asyncio/unix_events.py:94: RuntimeError
===================================================================================================== 10 passed, 1 error in 0.93 seconds ======================================================================================================

Steps to reproduce

I've copied the same test file 10 times (just to have some tests to work with).

async def test_foo(loop):
    assert 42 == 42

I use bash in order to run pytest several times until it fails:

while pytest tests/ -n auto -v; do :; done

Not sure why, but the verbose flag seems to affect it: -v

Your environment

Tested on 3.6 and 3.7.
8 Cores CPU

Python 3.6.7
Python 3.7.0b5

aiohttp==3.4.4
pytest==4.0.1
pytest-aiohttp==0.3.0
pytest-xdist==1.25.0

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are https://github.com/aio-libs/aiohttp/issues/2265 (RuntimeError: set_wakeup_fd only works in main thread), https://github.com/aio-libs/aiohttp/issues/930 (aiohttp.Timeout causes CancelledError), https://github.com/aio-libs/aiohttp/issues/841 (CTRL+C to stop doesn't work in python -m aiohttp.web ), https://github.com/aio-libs/aiohttp/issues/3004 (Ctrl+C not work as excepted for aiohttp on windows), and https://github.com/aio-libs/aiohttp/issues/3175 (aiohttp client timeout does not work).

[asvetlov]

Thanks for the report.
xdist creates several threads to execute tested code, aiohttp wants to subscribe on SIGINT and SIGTEM by default even if the thread is not the main.

We can add skipping signals handling if done from subthread (and raise a warning, sure).
But the real problem is: asyncio semi-working when the thread is not the main.
It works perfectly fine in 98% cases but can raise weird results when used for Unix signals or subproceses.
Now I have no clear vision of how to handle it

[lock[bot]]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a [new issue] for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that [new issue].