Lwt: Lwt_unix calls get stuck after Lwt_unix.fork

[ygrek]

Consider

let work () = Lwt.map ignore Lwt_unix.(getprotobyname "udp")

let () =
  Lwt_main.run @@ work ();
  match Lwt_unix.fork () with
  | 0 ->
    let n = 10 in
    Lwt_main.run @@ for%lwt i = 0 to n do
      let () = Printf.printf "loop %d of %d\n%!" i n in
      let%lwt () = work () in
      Lwt.return ()
    done;
    exit 0
  | _ -> exit 0

with lwt from git master and glibc 2.28 (current debian stable, problem doesn't reproduce on glibc 2.24 from previous stable) the process gets stuck in the middle of the loop :

$ ./main.native 
loop 0 of 10
loop 1 of 10
loop 2 of 10

(gdb) thread apply all bt

Thread 2 (Thread 0x7f0411faa700 (LWP 16866)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x557d5f3dddac <pool_condition+44>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x557d5f3ddd40 <pool_mutex>, cond=0x557d5f3ddd80 <pool_condition>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=cond@entry=0x557d5f3ddd80 <pool_condition>, mutex=mutex@entry=0x557d5f3ddd40 <pool_mutex>) at pthread_cond_wait.c:655
#3  0x0000557d5f337795 in lwt_unix_condition_wait (condition=condition@entry=0x557d5f3ddd80 <pool_condition>, mutex=mutex@entry=0x557d5f3ddd40 <pool_mutex>) at lwt_unix_stubs.c:225
#4  0x0000557d5f337bd6 in worker_loop (data=0x557d60a30b60) at lwt_unix_stubs.c:1129
#5  0x00007f04129c4fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f041276d4cf in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 1 (Thread 0x7f0412672b80 (LWP 16865)):
#0  0x00007f0412765037 in select () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000557d5f34298b in unix_select ()
#2  0x0000557d5f2da0c3 in camlLwt_engine__fun_3597 () at src/unix/lwt_engine.ml:403
#3  0x0000557d5f2d9ca6 in camlLwt_engine__fun_3524 () at src/unix/lwt_engine.ml:344
#4  0x0000557d5f2dc38c in camlLwt_main__run_1134 () at src/unix/lwt_main.ml:33
#5  0x0000557d5f2d8912 in camlMain__entry ()
#6  0x0000557d5f2d5b39 in caml_program ()
#7  0x0000557d5f360980 in caml_start_program ()
#8  0x0000557d5f360d15 in caml_startup_common (argv=0x7ffcb9bcddd8, pooling=<optimized out>, pooling@entry=0) at startup.c:157
#9  0x0000557d5f360d6b in caml_startup_exn (argv=<optimized out>) at startup.c:162
#10 caml_startup (argv=<optimized out>) at startup.c:167
#11 0x0000557d5f2d576c in main (argc=<optimized out>, argv=<optimized out>) at main.c:44

The following diff fixes it :

diff --git a/src/unix/lwt_unix_stubs.c b/src/unix/lwt_unix_stubs.c
index 6737095a..9786b594 100644
--- a/src/unix/lwt_unix_stubs.c
+++ b/src/unix/lwt_unix_stubs.c
@@ -1444,6 +1444,8 @@ CAMLprim value lwt_unix_reset_after_fork(value Unit) {

     /* Empty the queue. */
     pool_queue = NULL;
+
+    threading_initialized = 0;
   }

   return Val_unit;

PS the only commit in glibc that touches condition variables between those two versions is https://github.com/lattera/glibc/commit/ed19993b5b0d05d62cc883571519a67dae481a14 but I cannot understand if it explains

[aantron]

Thanks for the report!

[aantron]

@ygrek, I haven't been able to reproduce this so far.

I created a fresh VM, installed Debian stable, and have

$ /lib/x86_64-linux-gnu/libc.so.6
GNU C Library (Debian GLIBC 2.28-10) stable release version 2.28.
[snip]

I checked out this repo and installed packages lwt and lwt_ppx from master (9f02543). I pasted the code you have provided and built it with ocamlfind opt. I ran it many times, and it always runs to completion, until loop 10 of 10, and the exit status is zero.

[aantron]

This was with opam 2.0.5 from the GitHub release binaries, and OCaml 4.08.1 compiled by opam. I didn't use the system packages.

[ygrek]

ftr this was reproduced with ocaml 4.08.1 independently. I will try to find more deterministic repro

[aantron]

If it can help find the difference, I had

$ uname -a
Linux debian 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5+deb10u2 (2019-08-08) x86_64 GNU/Linux

[aantron]

Ah, the SMP in the output made me check the processor count. I can reproduce it with multiple CPUs (4). I had only 1 CPU allocated to the VM.

[aantron]

I've applied your patch to fix this (confirmed in VM). I'll include it in a release after also addressing #714.

I tried bisecting glibc to figure out exactly what is causing this, but it seems that this will take more time than I have available, including because I have to do it in a VM due to my setup not supporting glibc development. I was hoping to find the glibc bug and report it upstream. There doesn't seem to be anything in the glibc bugzilla about this.

Your patch looks safe, so let's just use it as a workaround (or maybe a fix, if this new glibc behavior is not incorrect).

[aantron]

Again, thanks!

[ygrek]

iiuc what happens the condition variables are in undefined state after fork. I would not claim this is incorrect in face of all the fork+pthreads interaction caveats and guidelines.