Lwt: Lwt_main.run should fail with prejudice if already running

[fare]

[Original issue title: Lwt_mvar.take fails to be woken up by put]

Using lwt 3.3.0 on ocaml 4.06.1 from opam, I have one thread that does:

        Lwt_io.printf "Waiting for DB message on %d...\n%!"
          (2 * Obj.magic db_mailbox)
        >>= fun () ->
        Lwt_mvar.take db_mailbox
        >>= fun msg ->
        Printf.printf "RECEIVED DB MESSAGE\n%!";

while the other thread does:

  Lwt_io.printf "Sending DB message to %d: %s\n%!"
    (2 * Obj.magic db_mailbox)
    (string_of_yojson (yojson_of_db_request r))
  >>= fun () ->
  Lwt_mvar.put db_mailbox r

and a third watching thread in a loop does:

         Lwt_io.printf "The mailbox at %d is %sempty\n%!"
           (2 * Obj.magic db_mailbox)
           (if Lwt_mvar.is_empty db_mailbox then "" else "not ")
         >>= fun () ->
         Lwt_unix.sleep 1.0

The first (server) thread hangs after it prints its message.
The second (client) thread prints and queues three messages.
The third (watcher) thread tells me that the mvar is not empty.
The first thread looks like it's never woken up.

The whole thing works when I run my unit tests, that use the very same setup. Then when I try to compile the actual executable server, it hangs. I'm a bit at a loss regarding what I may be doing wrong, and would appreciate guidance in debugging, whether the bug is mine (likely) or yours (unlikely, but possible?).

[aantron]

Thanks for the report.

I think the first order of business is to reduce this code as much as possible to a minimal example, but given it is working in one context and not another, I suspect that won't necessarily reveal the problem.

I'll take a close look in some hours.

Is it necessary to queue three messages to observe the problem? What happens if only one message is queued?

Keep in mind that Lwt_mvar.put "blocks" if there is already a message in the mvar, so it's possible you may have some kind of deadlock.

Are all the log messages from each "thread" being successfully printed?

This is probably a stupid question, but have you confirmed you are calling Lwt_main.run in your executable? :)

[fare]

Dear @aantron, thanks for the prompt reply!

I'm trying to reduce the context non-trivially without making the problem disappear, but so far without success. I also tried to get to link my application to a locally modified lwt, but so far only ended in opam dependency hell, with old opam 1.2.2 refusing to build old dependencies that use jbuilder or to build lwt from source, new opam 2.0.0rc4 failing to build ocaml 4.06.1 itself, and having no working build environment at all except some old docker images.

I believe the 3 put's that I see should be from the "same" thread (what do I know, I only wrote that program, but I may have inadvertently failed to include necessary fun () -> protections against early evaluation), which is disturbing if the first one ought to have blocked while the server didn't process any take.

And yes, the Lwt_main.run and database server startup happen in the same function Db.run.

[fare]

BTW, I can confirm that two Lwt_mvar.put indeed succeed before the third one blocks, but the Lwt_mvar.take is never woken up. I don't know whether a queue of depth 3 is the intended behavior or not for the mvar.

[fare]

I just managed to compile with lwt 4.1.0 from source, and the problem is still here. But at least now I can modify the source to introspect the internals and see what's broken...

[fare]

The mistake was that I was calling Lwt_main.run in my main file's let _ =, which looks like it was causing a collision with some Lwt_main.run already called by some support library, probably eliom.server.

Could Lwt_main.run use some kind of lock-protected mutable flag to detect right at the start that it's already running, and then raise a condition?

[aantron]

Yes, that should be easy to do. It doesn't even need to be lock-protected, as Lwt_main.run is basically always called from the main thread. I'll open a new easy issue about it.

Thanks for debugging it and reporting what it was!

[aantron]

Actually, can you check where the other Lwt_main.run call is being done?

If you were calling Lwt_main.run under a top-level let _ =, it seems to me the other Lwt_main.run must have already returned, so the two calls shouldn't be interfering with each other. I'm probably missing some detail on what is actually happening.

[fare]

I don't explicitly call Lwt_main.run either directly or indirectly, but I am using the ocsigenserver infrastructure that presumably does it on its own.

It shouldn't be too expensive to grab a lock around checking the "is it running?" flag, and failing with prejudice if it's running already. Maybe also add a label argument so we can provide __LOC__ or some other arbitrary string to Lwt_main.run, so people who disagree whose responsibility it was to start Lwt can easily identify each other and negotiate a proper protocol.

[aantron]

@fare I am just not sure of the diagnosis, as if Lwt_main.run had been called somewhere, it would not have returned control to you to call Lwt_main.run at a top-level binding again.

Was your let _ = Lwt_main.run ... at (non-local) module level, or inside some expression?

[bobbypriambodo]

@fare to try to clarify what @aantron is saying (as I understand it, sorry if this is already obvious to you), it should not be possible for two top-level Lwt_main.runs to be executed at a single point of time.

Lwt_main.run will block the calling thread until the promise passed to it is resolved, and in case of servers, it means it is never resolved (except maybe if there's unhandled exception). OCaml threads provide concurrency, but not parallelism, so only one thread can run OCaml code at one time. Hence, if there are two calls to Lwt_main.run at the top-level, it is guaranteed that the first call should already return when the second one is started.

However, problems may arise if inside the promise passed to Lwt_main.run you have another call to Lwt_main.run again, which might be the case in your issue. CMIIW, Anton.

[fare]

My current understanding is that ocsigenserver had already started a Lwt_main.run loop before it was dynamically loading my code that itself included a let _ = Lwt_main.run ....

What I would like is for Lwt_main to have a runner_name : string option ref itself protected by a mutex. When run is called with an additional optional parameter ?(name="(Lwt_main runner)"), it grabs the mutex, and

match (!runner_name) with
| Some existing -> raise (Lwt_main_already_running
                              (Printf.sprintf "Lwt_main.run is already being called by %s, cannot give control to new claimant %s" existing name))
| None -> runner_name := name ; ...proceed as usual...

Then, the code in Lwt_preemptive.run_in_main would check that the reference is Some owner, save the owner, set the reference to None, and at the end do the reverse.

[aantron]

Ah.

Lwt_main.run always runs in the main thread (or is supposed to). Was yours running in the main thread? This is why it shouldn't be necessary to use a mutex.

@Drup Do you know enough about ocsigenserver to comment on this dynamic loading?

[fare]

There is a tiny cost to check for a mutex just at the beginning of Lwt_main.run, and it will literally save DAYS of debugging to your users. It certainly cost me and my coworkers more than a day of despair.

[aantron]

We already agree on adding a flag. I am saying it doesn't need mutex protection, because Lwt_main.run should be running in the main thread in both your code and ocsigenserver.

Is that not the case? Can you confirm the two calls to Lwt_main.run are in different system threads? I suspect they are in the same thread, they are just nested.

[fare]

By definition, if someone gets it wrong, he may get it very wrong. There's also no cost to a mutex to protect that flag. You run get it twice every time you run Lwt_main.run and twice every time you use Lwt_pervasives.run_in_main — I don't see on what grounds you object to that mutex.

[aantron]

I don't think we actually object to the mutex, we just wanted to figure out/understand what is really going on. We will add the flag with the mutex.

[Drup]

@aantron No. You should ask @vasilisp