Async-std: Add Future::join

[yoshuawuyts]

In the 2016 futures announcement post, the join combinator was shown as something that would make choosing between several futures easy.

In https://github.com/rust-lang-nursery/futures-rs/issues/1215 and beyond this was changed to several methods: join, join1, join2, join3, and join!, try_join macros (proc macros since 0.3.0-alpha.18 so statements can be written inline).

Future::join

It still seems incredibly useful to be able to join multiple futures together, and having a single combinator to do that seems like the simplest API, even if resulting code might not look completely symmetrical. I propose we add Future::join:

use async_std::future;

let a = future::ready(1);
let b = future::ready(2);
let pair = a.join(b);

assert_eq!(pair.await, (1, 2));

Future::try_join

The futures-preview library also exposes a try_join method. This is useful when you want to unwrap two results. Internally it uses TryFuture as a reference, which means this method should only exist on futures where Output = Result<T, E>, and I'm not entirely sure if that's feasible. However if it is it might be convenient to also expose:

use async_std::future;

let a = future::ready(Ok::<i32, i32>(1));
let b = future::ready(Ok::<i32, i32>(2));
let pair = a.try_join(b);

assert_eq!(pair.await, Ok((1, 2)));

Future::join_all

The third join combinator present is Future::join_all. The docs don't make a big sell on them (inefficient, set can't be modified after polling started, prefer futures_unordered), but it's probably still worth mentioning. I don't think we should add this combinator, but instead point people to use fold instead:

__don't do this__

use async_std::future::join_all;

async fn foo(i: u32) -> u32 { i }
let futures = vec![foo(1), foo(2), foo(3)];

assert_eq!(join_all(futures).await, [1, 2, 3]);

__do this instead__

let futures = vec![foo(1), foo(2), foo(3)];
let futures = futures.fold(|p, n| p.join(n))
assert_eq!(futures.await, [1, 2, 3]);

_note: not tested this, but in general I don't think we need to worry about this case too much as handling the unordered case seems much more important and would cover this too._

[ghost]

This is somewhat tangential, but I wonder if the behavior of try_join!() is what it should be.

Currently, try_join!(a, b) will evaluate to Result<(A, B), E>.

But consider the fact that the now soft-deprecated macro try!() is equivalent to ?, so perhaps the try_ prefix should have the same effect as a ?.

What if try_join!(a, b) evaluated to (A, B) instead and returned the error if there was one? That would also mean one doesn't have to write ? after try_join!() because the ? is implied by the try_ part.

[yoshuawuyts]

Wrote more thoughts here: https://paper.dropbox.com/doc/2019-08-13-Fallible-Futures-Combinators--Aiz0YPhNHzntVEmJj~xUII2fAQ-001Bi7zqOMp6o5cyGuifL -- think this should cover most (and includes e.g. select).

[dignifiedquire]

is there a reason that join! can't handle both

let res: (_, _, _) = join!(ok(1), err(2), ok(3)).await;
let res: Vec<_> = join!(vec![ok(1), err(2), ok(3)]).await;

Where both would behave exactly the same, polling all features in parallel.

[skade]

Macros run before type checking, so this would be hard to to do in a way that allows for:

let vec = vec![ok(1), err(2), ok(3)];
join!(v);

[dignifiedquire]

so this would be hard to to do

you did not say impossible :)

[skade]

It's pretty much impossible.

[taiki-e]

This is somewhat tangential, but I wonder if the behavior of try_join!() is what it should be.

Currently, try_join!(a, b) will evaluate to Result<(A, B), E>.

But consider the fact that the now soft-deprecated macro try!() is equivalent to ?, so perhaps the try_ prefix should have the same effect as a ?.

What if try_join!(a, b) evaluated to (A, B) instead and returned the error if there was one? That would also mean one doesn't have to write ? after try_join!() because the ? is implied by the try_ part.

I think current behavior of try_join! is near to try_blocks.

[taiki-e]

Also, we (futures team) have designed these proc-macros to that they can be easily re-exported, so async-std can easily reuse them.

[skade]

We don't want to re-export futures types.

[taiki-e]

It is internal implementation of proc-macro, not a type.

[taiki-e]

It would be confusing if they use a different syntax than futures's select! macro because async-book also uses it.

[skade]

Using it would be an attempt at de-facto stabilizing it, though, as we're committing to our interface. Don't get me wrong, I don't think the futures-rs implementation is bad, just not committed stable

[skade]

FWIW, I see it as reasonable to recommend using the select macro for now (and potentially forever), but just not export it. We just don't have to commit to it now if we can't come up with a reliable design on our own.

[skade]

Coming back to join, I always found it jarring that the interface to join is a.join(b) and not (a,b).join(), is there a particular reason for this?

[yoshuawuyts]

Coming back to join, I always found it jarring that the interface to join is a.join(b) and not (a,b).join(), is there a particular reason for this?

This feels alien to me, and I don't know of any precedent in stdlib for this.

[vertexclique]

@skade (Mostly) to not confuse which one is the joining thread.

[skade]

@vertexclique That's a good point, though, with futures, it's more like A and B get joined to (A,B) by the joiner?

[vertexclique]

@skade Truly, in many other library designs, this is a freestanding function. But I prefer having

• For join => sequence
• For try_join => absolve
• For select and try_select they can stay same. (though I am still thinking this might also create confusion and I don't know why is it called select in the first place especially for futures.)

For iterator based approach instead inheriting fold it might be better to use a different naming.
For this using traverse is actually will be much more clear. Why? Because people might start to use par_iter(from rayon or so) and some others in combination with this library. So it might end up with weird errors lying around.

---

About join_all. It would be seriously nice to have it as sequence. Not as a tuple taking but using sequence which takes Vec<A>.
The unordered case can be solved by using semaphore at the thread_local where every future decrease the count by one when completes. Then if the implementation of sequence logic is like that people can create futures inside futures and juggle them around.

---

In my opinion, especially join and try_join should take collection. Which might be a better choice for all. Since at the end they are going to return either nested Vec<T> inside Result or something similar or pure Vec<Result<T, E>> with everything contained inside.

---

[yoshuawuyts]

Since at the end they are going to return either nested Vec<T> inside Result or something similar or pure Vec<Result<T, E>> with everything contained inside.

This is inaccurate. The return signature of join is (x, y). The return signature of try_join is Result<(x, y)>. Not making these APIs allocate is a requirement.

---

• For join => sequence
• For try_join => absolve

The naming of sequence is confusing to me. The operations happen in parallel, so parallel may be better, though I still strongly prefer join. I'm also confused about absolve; I don't understand the reasoning?

[vertexclique]

This is inaccurate. The return signature of join is (x, y). The return signature of try_join is Result<(x, y)>. Not making these APIs allocate is a requirement.

My point is we shouldn't write join and try_join as to how it is proposed. Why?

• Firstly, this is not referentially transparent at any given point of time.

let futures = vec![foo(1), foo(2), foo(3)];
let futures = futures.fold(|p, n| p.join(n))

You don't know the upcoming futures before the middle of the execution point. There shouldn't be something called FuturesUnordered in the universe. This usage is quite a bit interesting.

• Writing macros and not convenience function over the abstractions. If we don't want to have allocations this will be the way to go. Definitely, this will take a variable-length set of futures then return one. So no allocations requirement can be satisfied.

• Moreover, I was looking to futures and I saw something at join macro of futures where it tells about how it works. So let's take a look at the documentation of futures::join

While join!(a, b) is similar to (a.await, b.await), join! polls both futures concurrently and therefore is more efficient.

Unfortunately, it is not polling both futures concurrently. We should share a part of the thread pool explicitly for polling purposes to our own task mechanisms' management. If we don't do that writing abstractions will be cumbersome over time. You can take a look at the code of join macro here to understand me better.

---

If you want to write and it is a strict requirement for #[no_std] or some other core usage. Then I will suggest an API surface. Rewording what I am saying:

• Write future::sequence macro to resolve all futures and return back from it.
• Write future::absolve macro to return from the first error from a variable number of futures.

Inside the macro trying not to do what futures crate did (what I mentioned in the first part).

---

The naming of sequence is confusing to me. The operations happen in parallel, so parallel may be better, though I still strongly prefer join. I'm also confused about absolve; I don't understand the reasoning?

Naming is neither related to execution nor related to syntax. It is based on how other programming languages approach to the topic. And how it will be cleaner.
sequence stands for taking a sequence and resolve it all.
absolve (can be called emap or error map) which has nothing to do with mapping of collections in Rust context. Returns from the first error in a sequence (not in an iterative way but in a control yielding way – what I tried to describe above).
So I think it is better to call it absolve or if it feels weird traverse.

---

Edit: One more thing that I've found. In higher abstraction than this level (I assume this will be the task level.) This https://github.com/stjepang/async-std/issues/20 is what I am also thinking (just names are different, and at task level traits can give these options)

[skade]

@vertexclique the final point of expressing these concepts less abstract over tasks is indeed very interesting.