Tornado: Wheel release for Python 3.8 on Windows?

[zooba]

I see from other issues that you have resolved all the known problems on Python 3.8, so I'm wondering about the possibility of getting a Windows wheel release some time soon?

For context, I am in control of where the Windows built-in "python.exe" redirector points, and would like to update it to use Python 3.8 rather than 3.7. But Tornado is on my list of critical packages (due to Jupyter's dependency), so I'm holding off until you're ready.

Let me know if there's anything I can do to help out.

[bdarnell]

I see from other issues that you have resolved all the known problems on Python 3.8

Sort of. I've gotten tornado's tests passing on py38+windows, but the application-level backwards-incompatibility remains.

So the short answer is: Now that tornado's tests pass, I can and should do a 6.0.4 release with a py38+windows wheel. Applications that use Tornado will have to apply the WindowsSelectorEventLoopPolicy workaround (#2608). Some (most?) applications in the scientific python community have done so (jupyter and bokeh to name two). Changing python.exe to python 3.8 will break any applications that depend on tornado but have not made this change.

The long answer: Python 3.8 made a backwards-incompatible change by switching the default event loop to one that doesn't support the add_reader family of methods (which tornado depends on, and this is very unlikely to change). The workaround, setting the process-wide event loop policy to WindowsSelectorEventLoopPolicy, isn't really usable from a library because it needs to be done before anything else might have touched the event loop, so I have to punt it out to each application that is affected.

I'm working on a solution that runs a SelectorEventLoop in a separate thread if the default event loop is a proactor one; this should be a viable non-intrusive workaround that could be shipped in tornado 6.1. And I'd like to submit something like it to python 3.9, schedule permitting.

[zooba]

Thanks for the additional info!

I'm not familiar enough to know whether Tornado is a "once-per-application" kind of framework (that is, you can't mix it with other asyncio frameworks), in which case failing with an informative message is probably the best you can do. Trying to run a second loop on a new thread is going to cause you added pain (I know... I've done it...).

And if you might run multiple frameworks in an application, then starting a new thread unconditionally probably isn't the worst idea. This seems a bit like COM, where your application chooses thread affinity at startup and libraries just fail if you chose the "wrong" one.

What would it take to switch away from add_reader? If it's just time or expertise, I may be able to find/provide some, but if it's all just built around that design then yeah, I get it.

Jupyter having the workaround is enough for me, though. So whenever you're happy to put out a release, I'm good to call Python 3.8 "ready" :)

[bdarnell]

I'm not familiar enough to know whether Tornado is a "once-per-application" kind of framework (that is, you can't mix it with other asyncio frameworks)

Tornado can be combined with other asyncio frameworks/libraries (here's an example that combines tornado's server components with the client components of aiohttp). That's why this is tricky; we can't just put the process in a tornado-only mode, or fail if an asyncio-native library started the event loop first.

Trying to run a second loop on a new thread is going to cause you added pain (I know... I've done it...).

Yeah, I've done a lot of multi-thread event loop trickery too. My plan is to bury the extra thread under the event loop abstraction: when add_reader is called, spin up an extra thread to run the selector, but hand all callbacks back to the main event loop's thread so the threading isn't exposed to the application.

What would it take to switch away from add_reader? If it's just time or expertise, I may be able to find/provide some, but if it's all just built around that design then yeah, I get it.

Tornado exposes unix-style file descriptor stuff in some of its public interfaces, so it would require a significant deprecate-and-replace cycle to get away from it. (There's also a lot of performance-sensitive code that would need reworking)

[zooba]

Okay, yeah, using a separate thread makes the most sense there. ISTR it was discussed at one point for another reason as well, so having it integrated into the core asyncio library would probably be appreciated.

Feel free to close this issue (unless you want to use it to catch other queries like this until you can get a 3.8 release out).

[bdarnell]

6.0.4 is out now with wheels for python 3.8.