Julia: Remotecall and @spawnat for threads?

[ChrisRackauckas]

Is there a good answer to this SO question? Distributed parallelism (multiprocessing) has @spawnat and remotecall. Could similar functions be available for spawning a function call on a thread?

[oxinabox]

Transcribing this from other places. (SO, Gitter)

I spent a while digging around in the internals, so see if I could make this work as a hack.

• Use async to run a function on all threads using jl_threading_run,
• Yield then terminate (return) if the current thread is thread 1. (to free the async call from waiting for the ccall of jl_threading_run from finishing)
• If you are otherwise not the tartget thread, just terminate (return)
• if you are the target thread, run that actual function, storing the result in a Future

See https://gist.github.com/oxinabox/d0eb8a126948a77680fd6ba08582eca8

But I couldn't quiet get it to work. Not entirely sure why.
It is not a great solution anyway, since it fires work on to threads it doesn't want to use anyway.

I believe the hold up is https://github.com/JuliaLang/julia/blob/b80dad40ab36bcd7e11859d36b51c6afcf008148/src/threading.c#L699
which forces the main thread, to wait for all the other threads to complete, with the threadjoin.
and in doing so holds Locks, stopping any other coroutines from triggering a fork.
Thus no new work can be added.

[kpamnany]

The current parallel loop infrastructure is not suitable for this; we plan to introduce an @spawn macro in the future. Much work needs to happen in simplifying and unifying coroutines, threads, locks, workers...

In the meantime, http://docs.julialang.org/en/latest/manual/parallel-computing/#threadcall-experimental may suit your needs.

[tknopp]

This would be great to have since it makes asynchronous programming in GUI applications (e.g. Gtk.jl) easier. Here I kind of like the BackgroundWorker model from C# where after the thread has finished a callback in the main thread is executed.

[ancapdev]

Adding another use case to express interest in this.

I'm embedding Julia in a soft-realtime event processing system, where from time to time the system is parameterized in a new way and we take advantage of Julia's code generation facilities to generate event handlers specific to the new parameterization. These event handlers are later invoked for the time critical processing. The parameterization doesn't change terribly often, but the code generation and JIT is something I'd much rather be able to push to a background thread, so the main thread can continue servicing events using the old parameters.

Even some guidelines for how to hack something together with the current threading functionality would be very helpful. With the parallel fork/join pattern that's currently available, it looks like some of the fundamentals (like concurrent GC and JIT) are all available, or maybe I'm missing something?

[yuyichao]

it looks like some of the fundamentals (like concurrent GC and JIT) are all available

None of the two are available.

[ancapdev]

I see. So is GC suspended until the threads are joined?

[yuyichao]

No. All threads will wait for the GC to finish if it needs to run.

[vitreo12]

Are there any news about this? Is this set to be released on a specific future Julia version (1.5, 2.x, 3.x....)?

[oxinabox]

1.x after sometime after the first PATR PR is merged. Or as part of that.
(Or so I believe)