Lwt: Lwt_main.run hanging (threads and integration with external event loop)

[andreas]

I'm working on OCaml client bindings for FoundationDB, ocaml-fdb. As detailed below, I've run into a case where Lwt_main.run hangs, presumably due to some bad interaction Lwt, threads and libfdb. I haven't been able to pinpoint the issue, and it might not even be a bug in Lwt, but about the my assumptions on Lwt and threads. I'd appreciate any help in getting closer to the root of this issue 🙏

ocaml-fdb is based on the C library, libfdb, which implements its own event loop, similar to Lwt or Async, with a concept of "futures". Most library functions return futures rather than immediate values, and callback functions can then be bound to these futures that will be invoked when they are resolved.

The FDB event loop is started by calling fdb_run_network, and stopped with fdb_stop_network. fdb_run_network does not return until fdb_stop_network is called, and must thus be invoked on a separate thread. fdb_stop_network can be called from any thread. Once the event loop is stopped it cannot be restarted during the lifetime of the running program!

Integrating the FDB event loop with Lwt works quite well for the most part. An FDB Future, fdb_future_t, can be convert to a Lwt.t by creating a new Lwt promise and resolve it by binding a callback to the fdb_future_t -- this callback will be invoked by thread running the FDB event loop. Then Lwt can take over and it's business as usual.

Simplified code:

(* future_to_lwt : fdb_future_t -> fdb_future_t Lwt.t *)
let future_to_lwt future =
  let promise, resolver = Lwt.wait () in
  fdb_future_set_callback future (fun future ->
    Lwt.wakeup_later resolver future
  );
  promise

I considered two options for running the FDB event loop:

1. Using the Thread library (Thread.create),
2. Using something specific to Lwt (Lwt_preemtive.detach).

Seeing as the FDB event loop may outlive a single execution of Lwt_main.run, which happens in tests for example, I figured using Thread.create would be better.

Further, to make the API for ocaml-fdb simple, I would hide starting and stopping the FDB event loop: starting would be done automatically in a lazy fashion, and stopping would happen using an at_exit-hook. This approach is also used in FDB bindings for other languages.

Simplified code:

let fdb_started = ref false

let start_fdb_event_loop () =
  if not !fdb_started then begin
    let fdb_thread = Thread.create fdb_run_network () in
    fdb_started := true;
    at_exit (fun () ->
      fdb_stop_network ();
      Thread.join fdb_thread
    )
  end

(note that fdb_run_network releases the runtime lock using release_runtime_lock in ctypes)

I've run into a tricky problem though: Lwt_main.run p does not return when p uses the FDB event loop. All the actions in p happen, but at the very end, run simply hangs. Consider the following code:

let () =
  start_fdb_event_loop ();
  let p = (* interact with FDB *) in
  Lwt_main.run (
    p >|= fun _ ->
    print_string "1"
  );
  print_string "2"

This example will happily interact with the FDB, getting and setting values, then print 1, but never actually return and print 2. Explicitly stopping the FDB event loop with fdb_stop_network inside p does not work either -- for some reason the call to fdb_run_network never returns.

Running the process with dtruss on OSX, I see the following repeated indefinitely:

select(0x0, 0x0, 0x0, 0x0, 0x700000103ED8)       = 0 0

Is Lwt waiting for some promise to be resolved?

To debug the issue, I tried approach 2 instead, i.e. using Lwt_preemptive.detach instead of Thread.create. With this approach Lwt_main.run p returns correctly, but only if I stop the FDB event loop thread before p is resolved:

let () =
  let fdb_thread = Lwt_preemptive.detach fdb_run_network () in
  let p = (* interact with FDB *) in
  Lwt_main.run (
    p >>= fun _ ->
    fdb_stop_network ();
    fdb_thread >|= fun () ->
    print_string "1"
  );
  print_string "2"

The above terminates correctly and prints both 1 and 2, but it means I cannot start the event loop again 😢

To recap, both approaches (Thread.create and Lwt_preemptive.detach) have caveats and require the FDB event loop to be terminated before Lwt_main.run can return correctly. It appears there's some kind of bad interaction between threads, Lwt and the FDB event loop that I'm not sure of. I've tried to educate myself on these topics, but I haven't been able to determine the root cause yet.

[raphael-proust]

I don't know exactly what's happening in your case, but here are a few things to consider.

• Lwt provides its own at_exit in the Lwt_main module. I don't know if it'd fix your specific problem because I don't understand exactly what it is doing. Note that it was introduced to fix a dead-lock so maybe it's useful.

• Lwt lets you make custom engines. This might be a cleaner approach to binding with a future library but it would also probably be more complicated. If someone else has more info/example of use for engines could weigh in…

[aantron]

I'd need instructions for setting up the code to debug this and say what is really happening. However, some general notes follow:

I considered two options for running the FDB event loop:

1. Using the Thread library (Thread.create),
2. Using something specific to Lwt (Lwt_preemtive.detach).

They should be equivalent, unless you are using extra functions from Lwt_preemptive. I suggest to use Thread, however, since you don't need to represent the event loop thread as a promise.

p uses the FDB event loop

p can't use an event loop, it's a promise, so roughly an

[ `Resolved of 'a | `Pending of ('a -> unit) list ] ref

i.e. an inert piece of data, or a fancy kind of ref. So, in this code,

let p = (* interact with FDB *) in

it's the commented out part that "uses" FDB, so that's the interesting part!

Explicitly stopping the FDB event loop with fdb_stop_network inside p

We would need to see the computations that would resolve p, which I guess is what you mean by "inside" p.

Is Lwt waiting for some promise to be resolved?

Yes, p :) In order for Lwt_main.run to stop, two things need to happen:

1. p must become resolved (or rejected)
2. Lwt_main.run must be "awakened" if it is asleep inside a select (or equivalent polling system call). Otherwise, the loop won't have a chance to check if p is resolved.

Is this select being called repeatedly after p should have been resolved? Or is it one call, from which control has not yet returned when p is resolved, and does not return?

This is the most likely culprit IMO at this point: that Lwt_main.run is blocked (as normal) in a select, and there is nothing waking it up. You should be able to work around that by creating a fake pipe, whose only purpose is to force Lwt_main.run out of select, and sending a byte on it (from the FDB thread to the main thread). You'll probably have to ask users to call a helper that waits for this byte, and this helper will return p.

• Lwt provides its own at_exit in the Lwt_main module.

This probably won't help. It just allows asynchronous at_exit functions.

• Lwt lets you make custom engines.

This normally would be the right way to go. For an application, you could use FDB's event loop as the Lwt engine (main loop implementation). But when writing a library (such as a binding), you probably can't make that decision for your users (and FDB probably shouldn't have made that decision in C).

---

Separately from the above, I'm concerned about communication from the FDB thread to the main thread, where Lwt should be running. Does FDB call its callbacks on the same thread that its event loop is running on? (I would assume so).

If so, this code

let future_to_lwt future =
  let promise, resolver = Lwt.wait () in
  fdb_future_set_callback future (fun future ->
    Lwt.wakeup_later resolver future
  );
  promise

will trigger Lwt callbacks on the FDB thread, which will probably eventually violate assumptions made by Lwt (and most similar libraries).

[aantron]

IIRC Lwt uses something similar to this pipe internally for the same purpose, for notifying the main thread when events occur, in case those events aren't tied to a timeout or to an fd that is passed to select. An example would be running stat in a worker thread. There is no way to directly connect this stat or the worker thread to the select that is called by Lwt_main.run and blocking the main thread. So, when the stat task completes, it writes to a pipe. The corresponding read end is always passed by Lwt_main into select, so when the write occurs, the select returns, "waking up" the main thread. At that point, promises are resolved, then Lwt_main checks whether p was resolved, and exits if so.

I guess this means you don't have to make p explicitly depend on your pipe using a helper, but only do something like

let read_end, write_end = Lwt_unix.pipe in

let rec read_from_fdb_completion_fd () =
  Lwt_unix.read read_end (* other args *) >>= fun _ ->
  read_from_fdb_completion_fd ()
in

Lwt_async read_from_fdb_completion_fd;
(* The Lwt_unix.read will continually add the pipe to the list of fds that have to
   be monitored by select. *)

as part of your FDB thread launching code, and in your FDB thread, when the FDB event loop exits,

Lwt_unix.write write_end (* other args *)

[andreas]

@aantron, thanks a lot for the quick and detailed response! 💯

p uses the FDB event loop

p can't use an event loop

I guess what I was trying to was is that resolving p depends on the FDB event loop, i.e. happens in a FDB callback.

it's the commented out part that "uses" FDB, so that's the interesting part!

I'll construct and share a minimal, but complete reproduction example. I tried to omit unnecessary detail, but it's always a balance 😄

Is Lwt waiting for some promise to be resolved?

Yes, p :)

Given that 1 is being printed on stdout in the given example, it seems to me that p is actually resolved?

Is this select being called repeatedly after p should have been resolved? Or is it one call, from which control has not yet returned when p is resolved, and does not return?

select is being called repeatedly after p appears to have been resolved, i.e. after 1 is printed on stdout.

You should be able to work around that by creating a fake pipe [...]

I'll have to digest this fake pipe idea 🤔

Does FDB call its callbacks on the same thread that its event loop is running on?

In short, yes (if the future is already resolved, it happens on the thread calling fdb_future_set_callback, otherwise on the thread running the event loop). Should future_to_lwt happen on the Lwt thread, e.g. using Lwt_preemptive.run_in_main? Or some other more disciplined way of communicating between the two threads.

---

Thanks again for your help!! I'll share a reproduction example as soon as possible.

[andreas]

Here's a gist with a quite small repro case:

https://gist.github.com/andreas/f8c5eeb278d866e528f1b261c84eb535

To run this, you need to download and install libfdb: https://www.foundationdb.org/download/

For convenience:

git clone https://gist.github.com/f8c5eeb278d866e528f1b261c84eb535.git fdb-repro
cd fdb-repro
dune build repro.exe && ./_build/default/repro.exe

When running the above on my machine, I get the following output:

Start of program...
Creating cluster...
Start of network thread
Created cluster
Destroyed cluster
Called fdb_stop_network -- joining thread...

[aantron]

it seems to me that p _is_ actually resolved?

Try using Lwt.poll or Lwt.on_success to examine the true state of p. on_success will tell you if callbacks are also called.

Also, I suggest pinning Lwt and inserting a print before this line, https://github.com/ocsigen/lwt/blob/2fbc9982d05470e3c4905aaad8d984258ec88be4/src/unix/lwt_main.ml#L26, so you can be sure that it is being executed again repeatedly after p is resolved.

Should future_to_lwt happen on the Lwt thread, e.g. using Lwt_preemptive.run_in_main?

Generally, yes, because resolving the Lwt promise can cause callbacks that are attached to it to run immediately, which will make them run in the FDB thread. This violates the assumption of Lwt users that everything runs in one thread.

You probably don't need or want run_in_main for this, though, because it (1) allows running a promise-valued computation, but resolving a promise is not that, and (2) blocks the FDB thread (i.e., is a two-way communication, since a message is passed back to the FDB thread when the computation in the main thread completes). You don't need to block the FDB thread, nor the overhead of a two-way communication, you just need the FDB thread to produce resolution messages, and the main thread to consume them. That will fit nicely with the pipe that you may need to wake up select.

I suggest considering making changes along the lines above first, especially the prints, which will let you see better what is really going on. After that, I will take a look directly :)

[andreas]

I pinned Lwt and inserted print statements as suggested. The output indicated that Lwt is blocked on this line: https://github.com/ocsigen/lwt/blob/2fbc9982d05470e3c4905aaad8d984258ec88be4/src/unix/lwt_main.ml#L33

The expression Lwt.paused_count () = 0 && Lwt_sequence.is_empty yielded is true, which fits the picture that Lwt_engine.iter ends up blocking the thread.

Inspecting the promise p with Lwt.poll revealed that it is indeed resolved, but as this is done by the FDB event loop, the Lwt event loop seems to not know about it (that's my assumption anyway).

Based on your advice, and after looking at the implementation of Mwt, I ended up looking into notifications from Lwt_unix. Do I understand correctly that they could be a suitable tool for solving this problem? They appear to use pipes under the hood.

I'll try use notifications to implement future_to_lwt and see if that solves the issue. Thanks again for all your help so far.

[aantron]

That sounds reasonable, and the notifications APIs should work if the problem is what I/we think it is (right now).

[andreas]

I implemented the suggested fix and saw some progress, but then another hang appeared. As such, I wasn't sure whether using notifications had really helped or not. It turns out the new hang is due to something else (more at the bottom). In case it's helpful to anyone else, the new code looks something like this (simplified, again):

let future_to_lwt future =
  let promise, resolver = Lwt.wait () in
  let result = ref (Error -1) in
  let notification = Lwt_unix.make_notification ~once:true (fun () ->
    Lwt.wakeup_later resolver !result
  ) in
  fdb_future_set_callback future (fun future ->
    result := future_to_result future;
    Lwt_unix.send_notification notification
  );
  promise

Thanks a lot for you help, @aantron!!

(I think the new hang is due to a deadlock from acquiring the runtime lock from the main thread, since fdb_future_set_callback may invoke the callback immediately if the future has already been resolved. Opened an issue with Ctypes: https://github.com/ocamllabs/ocaml-ctypes/issues/592)

[aantron]

Great :)

The only notes I have are:

1. I imagined multiplexing all the completions into one notification, and using a queue of some kind. That would probably require more code from you, but I'm not sure if it has any other implications, such as improved performance. Not saying you should do this, just something to consider.
2. I didn't look at the deadlock issue in detail, but it seems like you would solve this by asking Ctypes to generate a binding that releases the lock around the implementation of fdb_future_set_callback.

[andreas]

1. I imagined multiplexing all the completions into one notification [...]

Ah, I see -- I'll give that approach a try.

1. I didn't look at the deadlock issue in detail, but it seems like you would solve this by asking Ctypes to generate a binding that releases the lock around the implementation of fdb_future_set_callback.

Oh man... I feel stupid that the solution was so simple and right in front of me. It works! Now I owe you two beers! 😉💯