Lwt: More general join and choose operators

[yallop]

The <&> operator has the following type:

val ( <&> ) : unit t -> unit t -> unit t

It'd be useful to have a variant that works for any promise type, returning both results, with the following type:

val ( <&> ) : 'a t -> 'b t -> ('a * 'b) t

that behaved something like this:

let (<&>) l r =
   let lv = ref None and rv = ref None in
   (    (l >>= fun v -> lv := Some v; return_unit)
    <&> (r >>= fun v -> rv := Some v; return_unit)) >|= fun () ->
   match !lv, !rv with
   | Some lv, Some rv  -> (lv, rv)
   | _ -> assert false

Similarly, the <?> operator has the following type:

val ( <?> ) : 'a t -> 'a t -> 'a t

It'd be useful to have a variant that doesn't constrain the result types to be the same, and that preserves the information about which promise delivered the result, with the following type:

val ( <?> ) : 'a t -> 'b t -> ('a, 'b) either t

that behaved something like this:

let (<?>) l r =
     map (fun l -> Left l) l
 <?> map (fun r -> Right r) r

[c-cube]

This seems interesting to me, but I'm wondering if it's worth breaking compatibility. Maybe providing both versions would help, e.g. with <??> and <&&> as the generalized versions?

[mfp]

On Fri, Mar 10, 2017 at 04:04:52AM -0800, Simon Cruanes wrote:

This seems interesting to me, but I'm wondering if it's worth breaking
compatibility. Maybe providing both versions would help, e.g. with <??>
and <&&> as the generalized versions?

My gut reaction is a very strong "don't break compatibility over this".
Typing <&&> is just a tiny little bit less convenient than <&>, and this
improved convenience in new code that only targets newer Lwt is minuscule
compared to the major inconvenience of fixing older code, or rewriting
altogether (with let%lwt a = a and b = b in, thus defeating the very point:
succinctness) if you want the code to compile against both old and newer Lwt
(so the end result is you cannot use the operator for a while).

<??> looks very useful to me, but in the several cases I can remember where
I could have used it, I actually needed Lwt.pick semantics. It's too late to
change <?> now, so for the sake of coherence <??> should also use
Lwt.choose. Maybe a Lwt.pick_either (like <??> but using pick) would
be in order?

--
Mauricio Fernández

[yallop]

Despite my reusing the existing names in the description, I agree with @c-cube and @mfp: it's better to add new operators, not to change the types of the existing ones.

@mfp: let%lwt ... and ... has slightly different behaviour to <&>. I don't know if it's deliberate. For example, if we define

let fail_after n msg = sleep n >>= fun () -> fail_with msg

then the following code fails with "a":

  let%lwt a = fail_after 1.0 "a"
      and b = fail_after 0.1 "b"

whereas the similar code written using <&> fails with "b":

   fail_after 1.0 "a" <&> fail_after 0.1 "b"

It would be very nice not to have these subtleties, and to be able to define everything in terms of the syntax (and the syntax itself in terms of a tiny core).

[Drup]

Actually, that might be considered a bug. I should take a look at it.

[talex5]

@yallop : what's the difference between your code:

let (<&>) l r =
   let lv = ref None and rv = ref None in
   (    (l >>= fun v -> lv := Some v; return_unit)
    <&> (r >>= fun v -> rv := Some v; return_unit)) >|= fun () ->
   match !lv, !rv with
   | Some lv, Some rv  -> (lv, rv)
   | _ -> assert false

and the apparently simpler:

let (<&>) l r =
  l >>= fun lv ->
  r >|= fun rv ->
  (lv, rv)

?

[yallop]

@talex5: the program

   fail_after 1.0 "a" <&> fail_after 0.1 "b"

fails with Failure "a" (i.e. the leftmost failure) with the simpler version, but Failure "b" (i.e. the temporally first failure) with the more convoluted version.

[talex5]

OK, but that seems like a separate issue (just reverse the order of the arguments if you want that). I mean, why are the refs needed?

[yallop]

Reversing the order of the arguments doesn't give the right behaviour.

If both promises fail, the behaviour should be to raise the error from the one that failed first.

[talex5]

Ah, so it does. I'd assumed join waited for all the threads to finish so that it could be deterministic and always report the left-most error, but it is indeed documented to return the first one to fail:

join l waits for all threads in l to terminate. If one of the threads fails, then join l will fails with the same exception as the first one to terminate.

[aantron]

Probably best re-considered and done after #354 (new lwt.ml) is in.

[aantron]

<&&> has just been added under the name Lwt.both in #668 . See https://github.com/ocsigen/lwt/pull/668#issuecomment-475424431.

[yallop]

Thanks, @aantron!

[aantron]

We can also add the actual <&&> operator, if there is still demand for it, or when there is again. As for <??>, I think we need a good choice of an either type. I think the constructors of result are too opinionated in naming. Perhaps [ `Left of 'a | `Right of 'b ] would work. Again, though, if there is no immediate demand for it at this point, I'd like to just leave this as a note for the future.

[Drup]

I don't think having the either version is useful. You almost never actually want to use either: a specific datatype is almost always better, and once you have the appropriate constructor, you can just do

     job1 >|= foo
 <?> job2 >|= bar
 <?> job3 >|= baz

which is much better.

[aantron]

I agree with that. E.g., when considering the either type, I was wondering about choosing [ `Left | `Right ] forcing extra allocations and syntax from mapping if one actually does want result.

If we all agree on this, I suggest to close this issue. Now that both exists, we can add <&&> in a quick PR on demand at any time.

[yallop]

Closing this is fine with me.