Mypy: Add support for mypy daemon on Windows

[ilevkivskyi]

There is a major limitation for mypy daemon and fine-grained incremental mode: mypy daemon doesn't work on Windows, see http://mypy.readthedocs.io/en/latest/mypy_daemon.html#limitations

In particular, dmypy_server.daemonize uses some functionalities not available on Windows (such as os.fork). It would be great to remove this limitation (and update Appveyor to test the support).

@ethanhs It looks like you have most experience on Windows among all of us, would you like to take this task?

[ethanhs]

@ilevkivskyi I have a few more things to follow up on for PEP 561, but after that I can take this.

[ethanhs]

Mostly for myself, but also those who are interested:

After looking at the implementation a bit and talking with Steve Dower (who was very helpful!) I think the best plan is to use named pipes (which act esesentially like Unix sockets on Windows) for communication and a detached process (using creation flags of Popen). This should be enough to approximate the current UNIX API used by the mypy daemon. We will end up using _winapi.

[ethanhs]

As an update, I have finally figured out the Windows API incantations needed to get a server/client setup working over named pipes, and can spawn a detached process which together essentially emulate much of the fork/Unix pipe aspects of the classic Unix daemon. I will start working on the actually mypy daemon now I know how I will implement it :)

[ilevkivskyi]

@ethanhs Sorry for disturbing you, just wanted to check if there are any updates on this. This problem just appeared again in other context.

Thanks for helping with this!

[ethanhs]

@ilevkivskyi no problem! I've been a bit side-tracked with some PEP 561 stuff, so nothing new to report sadly. I will get back to this soon hopefully.

[ethanhs]

Unfortunately, I do not think I will have time to work on this any time soon (at least until December most likely). I am going to un-assign myself so someone else can pick this up if they are interested, but if not, I will come back to this in December.

[JukkaL]

@ethanhs Do you happen to have any prototype/work-in-progress code you can share? It's okay if it's not very polished.

[ethanhs]

@JukkaL I have some experiments with named pipes and detached processes I could upload. Nothing really mypy specific, mostly experiments to understand the APIs and how a daemon would work.

[JukkaL]

I have some experiments with named pipes and detached processes I could upload.

Those could be quite helpful, and I'd appreciate if you can share them! Our plan is to add Windows support in the next few weeks, as part of making mypy daemon a more fully supported feature.

[ethanhs]

@JukkaL That is good to know. If it is such high priority I would be happy to help how I can. I originally put this off because I felt other things were higher priority (with many things "high-priority" it sometimes can be hard to tell how to prioritize 😄 ).

Anyway, here is the experimental code I had: https://github.com/ethanhs/windows-daemon-python

It doesn't exactly work correctly, but it covers a rough sketch of how the named pipes code is supposed to be used. For the DETACHED_PROCESS flag, you basically need to just add creationflags=0x00000008 to your subprocess call (0x00000008 being the DETACHED_PROCESS flag).

[ethanhs]

Okay, well I got somewhat nerd-sniped (perhaps I self-sniped) :) I have a branch with some work I just did that daemonizes the server with a detached process, passing arguments via a temporary pickle'd file. I'm still working on getting named pipes to work for IPC, but hopefully I can get something working today or tomorrow.

The branch is available here: https://github.com/ethanhs/mypy/tree/windmypy

[JukkaL]

@ethanhs Thanks! It looks like you are most of the way there already.

[ethanhs]

@JukkaL yes, if you want, I can probably get a prototype working this week.

[JukkaL]

@ethanhs That would be great! But I can also finish it up if you are busy.

[ethanhs]

@JukkaL I will have time to work on it this week. If I get stuck I will let you know.