Flask-socketio: Re-using socketio with multiprocessing process (thread) in module

[kaaetech]

1. Spawn socket (works)
2. Emit through socket in main module (works)
3. Import module for the thread
4. Start new thread and pass socket through args() with multiprocessing.Process (works)
5. Emit from new thread created (NOT working .. )

This worked perfectly fine with threading.Thread, but after changing to multiprocessing to obtain other functionality; SocketIO emits doesn't reach frontend. Why is that?

Tried:
async_mode: Eventlet, Gevent and Threading
from flask_socketio import emit
construct new socket (port in use obviously)

[miguelgrinberg]

Eventlet and gevent are unlikely to work with multiprocessing. They are not compatible. You may have a chance in threading mode, but this mode does not support websocket, so even if it is doable, it isn't a practical implementation.

What I recommend that you do is use the officially supported method of emitting from an external process. See the docs.

[kaaetech]

Thank you a lot for getting back

For my project it is not possible to have a Redis server or so running, even though it would enable it to work externally.

What is your suggestion for a non-practical implementation with threading mode? Threading mode was what I used before I implemented multiprocessing, and that worked out well. I was able to emit to the client by passing the socketio object as an argument using threading.Thread.

If it makes any difference, I don't need any callbacks. I only need to be able to emit from my Javascript SocketIO on Frontend (works now, as it goes to the main thread) and then where the problem lies: _being able to emit from subprocesses spawned by multiprocessing._

[kaaetech]

Update:
After enabling further logging I can see that the multiprocessing threads DOES emit, but they do not reach the client.

Example log of emit from main that reaches client:

[2019-05-05 13:16:05,241] emitting event "updateEmail" to all [/]
[2019-05-05 13:16:05,246] 30c2f2352541457cbd1f09dd3dfa9213: Sending packet MESSAGE data 2["updateEmail",{"email":"[email protected]","taskID":0}]

Example log of emit that does not reach client:

[2019-05-05 13:16:05,264] emitting event "updateTask" to all [/]
[2019-05-05 13:16:05,266] a4938da844cb4fe38e85113cb6ebf62d: Sending packet MESSAGE data 2["updateTask",{"status":"Scraping data!","taskID":0}]

These log examples confuses me, because they're identical and none of them outputs an error; but the second one (done from thread) doesn't reach the client

[miguelgrinberg]

Yeah, I'm not sure if this can be made to work in this way, never attempted something like this.

The problem is that this server is stateful, all the information about connected clients is stored in the memory of the process. For the subprocess to be able to emit you would need all that plus all the active file descriptors for the connections to be inherited by the subprocess.

Regarding the two log snippets that you pasted above, are they from the same session or different? The session IDs are different, that could be a side effect of spawning the subprocess maybe.

[kaaetech]

Apparently I just had to tabs open, I tried again ensuring only 1 socket connection was open and the session IDs came out identical in the main module and in the subprocess. So it seems to successfully share those values with the processes, but apparently it could be missing descriptors for the connections; hence the emit is not received on client side?

Do you think that I would I have to come up with a whole other solution than sockets to complete my purpose?

Thank you

[miguelgrinberg]

@kaaetech my opinion without actually debugging what happens is that this approach is not a good one, it's bound to have problems. If it can be made to work it's going to be by taking advantage of how your operating system spawns subprocesses. I prefer not to rely on OS specifics.

[kaaetech]

If that’s the case then you’re right, my program can’t rely on OS species either. Thanks a ton.