Lwt: Missing or improper flush on "%!"

[vog]

The %! element in format strings doesn't produce a proper flush on the underlying channel. It is not clear to me whether the flush is just plain out not happening, or whether the flush happens just in an improper way.

The easiest way to observe this issue is to output to stdout, then flush, then output to stderr. Assume the following test.ml file:

let () =
  Lwt_main.run @@
  let%lwt () = Lwt_io.printf "Foo\n%!" in
  let%lwt () = Lwt_io.eprintf "Bar\n%!" in
  Lwt.return ()

If you run that multiple times in a row,

$ ./test; echo ---; ./test; echo ---; ./test

you get a mixed output similar to this:

Bar
---
Bar
Foo
---
Foo
Bar

If you don't use %!, but call instead the flush by hand, everything is fine:

let () =
  Lwt_main.run @@
  let%lwt () = Lwt_io.printf "Foo\n" in
  let%lwt () = Lwt_io.flush Lwt_io.stdout in
  let%lwt () = Lwt_io.eprintf "Bar\n" in
  let%lwt () = Lwt_io.flush Lwt_io.stderr in
  Lwt.return ()

$ ./test; echo ---; ./test; echo ---; ./test

Bar
---
Foo
Bar
---
Foo
Bar

[Drup]

Yeah, it's due to the fact that the implementation of the Lwt_io.*printf* family of function is fundamentally broken.

Can you try with Lwt_fmt instead ? It shouldn't have the issue.

[vog]

This works perfectly, thanks for the hint!

May I suggest to mark all Lwt_io.*printf* as deprecated and buggy, with instructions to use Lwt_fmt instead?

Currently, the Lwt_io documentation doesn't have a single hint to Lwt_fmt.

[vog]

(Moreover, I wonder why Lwt_io.printf, etc. don't just call their Lwt_fmt.printf, etc. counterparts directly. It seems that those are perfectly working drop-in replacements.)

[raphael-proust]

I wonder why Lwt_io.printf, etc. don't just call their Lwt_fmt.printf, etc. counterparts directly.

Anyone knows why? @aantron @Drup?
AFAICT, Lwt_io calls Printf whereas Lwt_fmt calls Format. Does it have an impact on compatibility with older OCaml versions? impact on performance? impact on some other such thing?

Otherwise, I'm tempted to implement this.

[vog]

As far as I remember, the OCaml manual recommends against mixing such modules, as they use separate buffers, so interleaving their output would not work as expected. If that is true, this would be yet another reason to stick to one side (preferably Format) and to abandon the other side (or at least also implement it using Format, to enable mix and match without introducing bugs).

[Drup]

The types are similar, but not straight up compatible. It can be seen in particular when using %a in particular. Replacing one by the other would be a breaking change.

Another issue is that, ideally, Lwt_fmt should not depend on lwt_unix so it could be used in more context.

Those two issues are why we didn't make more drastic changes to the APIs at the time. It definitely should be done eventually though ...

[raphael-proust]

Maybe Lwt_fmt could have a similar fate as Lwt_log where we deprecated it and recommend a separate package instead.

[Drup]

Maybe Lwt_fmt could have a similar fate as Lwt_log where we deprecated it and recommend a separate package instead.

I don't really see how that solves any of our problems ? If anything, it's the functions in Lwt_io that should be deprecated and removed in favor of the one in Lwt_fmt.

[raphael-proust]

That solves the other thing you mentioned:

ideally, Lwt_fmt should not depend on lwt_unix so it could be used in more context.

[Drup]

That solves the other thing you mentioned:

ideally, Lwt_fmt should not depend on lwt_unix so it could be used in more context.

No, the goal to not depend on lwt_unix is to be able to put it in lwt_core !