uvloop run_in_executor prohibits coroutines (even if executor supports it) unlike asyncio run_in_executor

[autumnjolitz]

• uvloop version:
  uvloop==0.8.0
• Python version:
  Python 3.6.2
• Platform:
  OSX 16.7.0
• Can you reproduce the bug with PYTHONASYNCIODEBUG in env?:
  Yes.

Existing Asyncio in CPython master (snapshot permalink: https://github.com/python/cpython/blob/b7555babe95ee0db2f1224ec53cfe68a005448a1/Lib/asyncio/base_events.py#L724-L734) does _not_ prohibit coroutines. I make use of this with a custom ProcessPoolExecutor that inits it's own private process event loop and runs coroutines as it encounters them, isolating out actual blocking synchronous tasks to a look aside process (for it to actually block on).

This following part:

https://github.com/MagicStack/uvloop/blob/b5aa30b264ecd602b178245ffae17af19cbde7ca/uvloop/loop.pyx#L2307-L2308

should be moved such that if executor is an instance of cc_ThreadPoolExecutor it raise TypeError("coroutines cannot be used with run_in_executor()") there. That will allow one to preserve safety for cc_ThreadPoolExecutor while allowing custom Executors to handle their support (or lack of) for coroutines.

[1st1]

If you look carefully at the asyncio code, you'll see that run_in_executor calls its _check_callback() method in debug mode:

https://github.com/python/cpython/blob/b7555babe95ee0db2f1224ec53cfe68a005448a1/Lib/asyncio/base_events.py#L726-L727

and _check_callback(), in turn,

https://github.com/python/cpython/blob/b7555babe95ee0db2f1224ec53cfe68a005448a1/Lib/asyncio/base_events.py#L679-L683

checks that the passed function is not a coroutine or a coroutine function. Performance is the only reason why run_in_executor does not always check that.

run_in_executor was never ever supposed to run coroutines and making it doing so isn't a good idea.

Closing this issue as "not a bug".

[autumnjolitz]

Thank you 1st1. You’re correct.

I had not noticed that asyncio is stupidly opinionated in the interface for sharding out tasks to another worker.

[1st1]

Well, asyncio certainly has its fair share of warts, many of which are addressed in Python 3.7 and will be further cleaned up in Python 3.8. Calling it "stupidly opinionated" is a little short-sighted, and I do not appreciate that here.

[autumnjolitz]

That's not short sighted - given the relative stasis that stdlib packages exist in, time-wise, it is quite reasonable to expect that interface restriction (no coroutines) will not change. You're free to be irked over my use of the word "stupid" but I stand by it being opinionated given that it's usually within the purview of an executor instance to reject unusable tasks (submit or an internal method).

Either way, that's a part of the ecosystem that's probably not going to change.

[asvetlov]

@benjolitz your "custom executor" doesn't follow executor's contract.
The executor should accept a function (any callable with __call__ method technically) and args and call it in a separate thread/process.
Calling a coroutine in an executor doesn't make sense.
asyncio was not designed in this way from the very beginning (but had no sanity check for the case).
Eventually, we've added the check (as well as many other sanity checks in asyncio API) to help users avoid silly errors.
In asyncio itself checks are not for free, that's why they are applied in debug mode only very often.

You've tried to break the contract and shoot your leg, that's sad.

[1st1]

You're free to be irked over my use of the word "stupid" but I stand by it being opinionated given that it's usually within the purview of an executor instance to reject unusable tasks (submit or an internal method).

asyncio has existed for many years now, and you're the first one who tries to use run_in_executor this way (and people request new asyncio features all the time). It suggests that what you're asking is either a very niche feature or a misuse of the API. It's your job to convince people otherwise :)

Being opinionated is fine, trying to abuse some API and complain that it didn't work in an offensive way is not. It makes your position weaker and prompts other people to simply stop the conversation.

As for your use case: what stops you from designing your own API/micro-framework/library/helper that does what you want? Why do you want to use run_in_executor() for that? Adding a simple wrapper like this is trivial:

async def my_run_in_executor(func):
    if inspect.iscoroutine(func):
        return await my_special_executor.run(func)
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(func)

Your code will become more readable; another developer will be able to follow what's going on and how it works.

Now it's very unlikely that we will ever change run_in_executor() to accept coroutines. Not because of "stasis" but because we want to design an entirely new API (more high-level and robust) to interact with executors.