Uvloop: Cannot use `sys.stdout` in `create_subprocess_exec`.

[ojii]

• uvloop version: 0.9.1
• Python version: 3.6.5
• Platform: macOS 10.13.3
• Can you reproduce the bug with PYTHONASYNCIODEBUG in env?: Same behavior irregardless of PYTHONASYNCIODEBUG.

Code for reproduction:

import asyncio
import sys

import uvloop


async def run_subprocess():
    proc = await asyncio.create_subprocess_exec(
        sys.executable, '-c', 'print(1)',
        stdout=sys.stdout,
    )
    await proc.communicate()


def run(loop_factory):
    loop = loop_factory()
    asyncio.set_event_loop(loop)
    try:
        loop.run_until_complete(run_subprocess())
    finally:
        loop.close()


def main():
    print('asyncio')
    run(asyncio.new_event_loop)
    print('uvloop')
    run(uvloop.new_event_loop)


if __name__ == '__main__':
    main()

Output:

asyncio
1
uvloop
Traceback (most recent call last):
  File "hoge.py", line 32, in <module>
    main()
  File "hoge.py", line 28, in main
    run(uvloop.new_event_loop)
  File "hoge.py", line 19, in run
    loop.run_until_complete(run_subprocess())
  File "uvloop/loop.pyx", line 1364, in uvloop.loop.Loop.run_until_complete
  File "hoge.py", line 10, in run_subprocess
    stdout=sys.stdout,
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/coroutines.py", line 110, in __next__
    return self.gen.send(None)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/subprocess.py", line 225, in create_subprocess_exec
    stderr=stderr, **kwds)
  File "uvloop/loop.pyx", line 2368, in __subprocess_run
  File "uvloop/handles/process.pyx", line 564, in uvloop.loop.UVProcessTransport.new
  File "uvloop/handles/process.pyx", line 49, in uvloop.loop.UVProcess._init
  File "uvloop/handles/process.pyx", line 38, in uvloop.loop.UVProcess._init
  File "uvloop/handles/process.pyx", line 254, in uvloop.loop.UVProcess._init_options
  File "uvloop/handles/process.pyx", line 456, in uvloop.loop.UVProcessTransport._init_files
ValueError: invalid stdout argument value 1

[achimnol]

I'm facing the same/similar error in my Travis CI environment.
My uvloop version is 0.8.1 and pytest version is 3.5.0, and I'm passing stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL.

        proc = await asyncio.create_subprocess_exec(*[
            'python', '-m', 'ai.backend.agent.stats',
            f'tcp://127.0.0.1:{stats_port}',
            cid,
            '--type', collection_type,
>       ], **output_opts)
tests/test_stats.py:53: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.6/asyncio/subprocess.py:225: in create_subprocess_exec
    stderr=stderr, **kwds)
uvloop/loop.pyx:2164: in __subprocess_run (uvloop/loop.c:39419)
    ???
uvloop/handles/process.pyx:549: in uvloop.loop.UVProcessTransport.new (uvloop/loop.c:93810)
    ???
uvloop/handles/process.pyx:49: in uvloop.loop.UVProcess._init (uvloop/loop.c:84790)
    ???
uvloop/handles/process.pyx:38: in uvloop.loop.UVProcess._init (uvloop/loop.c:84510)
    ???
uvloop/handles/process.pyx:239: in uvloop.loop.UVProcess._init_options (uvloop/loop.c:88007)
    ???
uvloop/handles/process.pyx:415: in uvloop.loop.UVProcessTransport._init_files (uvloop/loop.c:91046)
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <_pytest.capture.DontReadFromInput object at 0x7f4717c82860>
    def fileno(self):
>       raise UnsupportedOperation("redirected stdin is pseudofile, "
                                   "has no fileno()")
E       io.UnsupportedOperation: redirected stdin is pseudofile, has no fileno()
../../../virtualenv/agent/lib/python3.6/site-packages/_pytest/capture.py:583: UnsupportedOperation

[1st1]

Fixed in master. Please test!

[spumer]

In 0.14.0 still present

platform darwin -- Python 3.6.10, pytest-5.4.2, py-1.8.1, pluggy-0.13.1

Code failed with same issue:

    def _parse_list(output, files_patt):
        patt = r'(\d+)\s+.{16}\s+(%s)' % files_patt
        return re.findall(patt, output)

    async def list_files(cls, rar_file: str, files_patt):
        process = await asyncio.create_subprocess_exec(
            'unrar', 'l', rar_file,
            stdout=asyncio.subprocess.PIPE,
        )
        output, _ = await process.communicate()
        if process.returncode != os.EX_OK:
            raise RuntimeError(
                f'unrar failed: return_code={process.returncode!r}')

        return [(int(size), name) for size, name in _parse_list(output.decode(), files_patt)]

csv_in_rar_count = await list_files(rar_path, '(?i).*CSV')

[spumer]

@1st1

    process = await asyncio.create_subprocess_exec(*cmd)
../../../../../.pyenv/versions/3.7.8/lib/python3.7/asyncio/subprocess.py:217: in create_subprocess_exec
    stderr=stderr, **kwds)
uvloop/loop.pyx:2749: in subprocess_exec
    ???
uvloop/loop.pyx:2707: in __subprocess_run
    ???
uvloop/handles/process.pyx:596: in uvloop.loop.UVProcessTransport.new
    ???
uvloop/handles/process.pyx:60: in uvloop.loop.UVProcess._init
    ???
uvloop/handles/process.pyx:49: in uvloop.loop.UVProcess._init
    ???
uvloop/handles/process.pyx:274: in uvloop.loop.UVProcess._init_options
    ???
uvloop/handles/process.pyx:458: in uvloop.loop.UVProcessTransport._init_files
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.capture.DontReadFromInput object at 0x114fc6590>

    def fileno(self) -> int:
>       raise UnsupportedOperation("redirected stdin is pseudofile, has no fileno()")
E       io.UnsupportedOperation: redirected stdin is pseudofile, has no fileno()

../../.venv/lib/python3.7/site-packages/_pytest/capture.py:230: UnsupportedOperation

[blakewatters]

Whatever the underlying problem is here, it definitely has not been closed out. I have found that it can be worked around in tests where it pops up by using the default event loop policy via a pytest mark at the top of my test files that hit the issue. Hopefully, this helps until we can nail down the underlying issue:

# NOTE: These tests are brittle when run under uvloop. We run these under the default
# asyncio event loop policy to avoid exceptions relating to pytest output capture.
# The exception is: `io.UnsupportedOperation: redirected stdin is pseudofile, has no fileno()`
pytestmark = [pytest.mark.asyncio, pytest.mark.event_loop_policy("default")]

[fantix]

redirected stdin is pseudofile, has no fileno()

This error message is from pytest. If you try to read from stdin, it also says:

pytest: reading from stdin while output is captured! Consider using -s.

I'm checking why asyncio doesn't fail this test.