Tokio: rfc: reducing "runtime not found" confusion

[carllerche]

Context

Tokio's resource types (TcpStream, time::delay, ...) require a runtime to function. Users interact with the resource types by awaiting on results. The runtime receives events from the operating system related to these resources and dispatches the events to the appropriate waiting task.

Tokio does not implicitly spawn runtimes. It is the user's responsibility to ensure a runtime is running. This is usually done using #[tokio::main] but creating a runtime manually is also possible.

Multiple runtime flavors are provided. There is a multi-threaded, work-stealing, runtime. This runtime spawns multiple threads. It is the recommended runtime for a number of cases, including network server applications. There is also a single-threaded "in-place" runtime. This runtime spawns no threads and is useful for cases like implementing a blocking interface for an async client. This is how the reqwest crate's blocking module works.

Additionally, there are cases in which it is useful to have a single process start multiple runtimes of the same flavor. For example, linkerd currently uses a pair of single-threaded runtimes, one for data-plane forwarding work and the other for control-plane tasks (serving metrics, driving service discovery lookups, and so on). This helps keep the two workloads isolated so neither can starve the other, and allows the use of !Sync futures that avoid the overhead of synchronization, without making the entire application single-threaded. The ability to run multiple runtimes in the same process is an important feature for many use-cases.

The problem

When using a TcpStream, the resource type must reference a runtime in order to function. Given that Tokio may have any number of runtimes running in the current process, the resource type must have some strategy by which it can select the correct runtime.

Currently, this is done by using a thread-local to track the current Runtime. In many cases, a process only includes a single runtime. A problem arises when attempting to use a resource from outside of a runtime. In this case, it is unclear which runtime the resource type should use and Tokio will panic!.

use tokio::net::TcpStream;

fn main() {
    // Boom, no runtime.
    let future = TcpStream::connect("www.example.com");
}

The strategy for fixing this is to enter a runtime context using a runtime::Handle.

use tokio::net::TcpStream;
use tokio::runtime::Handle;

fn do_some_work(handle: &Handle) {
    handle.enter(|| {
      let future = TcpStream::connect("www.example.com");
    });
}

This panic is the source of confusion for users who are not aware of how Tokio searches for the runtime.

History

In the very early days, tokio-core always required an explicit Handle. There was no context, thread-local or global, that stored the "current reactor". This resulted in Handle being a field on virtually every single type or an argument to every single function. This was tedious given that, in most cases, there was only ever a single tokio-core reactor in the process. In some cases, it resulted in measurable performance degradation as the Handle field increased struct size.

Because of this, tokio-core started providing a static runtime. Resource types would default to using this static runtime. Resource types also included method variants that took an explicit &Handle, allowing the user to specify a custom runtime.

The primary problem with a static runtime is that it cannot easily be configured. However, all users who ended up configuring their runtime were forced to use the more verbose APIs with an explicit &Handle argument. Additionally, some libraries did not provide methods with an explicit &Handle argument, preventing them from being used with custom runtimes.

To solve these problems, Tokio added a thread-local tracking the "current" runtime. Now, resources would first check the thread-local and if it was not set, it would use the global runtime. This introduced a new problem. Users that intended to use a custom runtime would accidentally use their resource types from outside of their custom runtime, which would start the global runtime and shift their parts application to the global runtime. The worst part of this is everything "seemed to work" but was not doing what the user intended. Half the application ran on a static runtime with default configuration and the other half on the configured runtime. Usually, nothing was noticed until poor performance was noticed in production.

The final iteration, resulting in the Tokio of today, was to remove the concept of the global runtime in favor of the thread-local context. This prevents users from accidentally being shifted to the global runtime and things appear to work, but are in a degraded state. The consequence of this change is that attempting to use Tokio's resource types from outside of a runtime results in a panic.

Options

There are a few ways forward from here. These options are not mutually exclusive. This issue is to discuss ways forward. Feel free to propose alternate strategies as well.

Change no behavior and improve the panic message

The thread-local context logic can remain unchanged. Instead, the panic message is improved to include more context about the problem and some options for fixing it.

Re-introduce a static runtime using a feature flag

In this case, a static runtime is re-introduced. However, it is guarded by a feature flag: rt-global. rt-global would also be included in the full meta feature.

When rt-global is enabled, Tokio resources would first check the thread-local context. If it is set, the current runtime is used. If there isn't one set, then the global runtime is used.

The primary danger here is silently ending up with the "split application". Since feature flags are additive, a library or component may include the rt-global feature flag and the application does not know it is accidentally using the global runtime.

Provide a separate tokio-global-runtimecrate

The specific name of the crate would need to be massaged. The idea is to have a separate crate define the static variable. In this case, users who wish to use a statically defined runtime would depend on this crate.

The main downside as I see it to this is that it makes the global runtime less discoverable.

Re-introduce &Handle method variants

In this case, all async methods include a variant that takes an explicit &runtime::Handle. Users who want to ensure a runtime exists and it is the correct runtime may opt to be explicit about the runtime used by specifying it.

This doesn't really solve the problem that users are confused when calling TcpStream::connect panics as it requires them knowing they must call the &runtime::Handle variant. However, improvements to the panic message would include mention of this strategy.

[dcarosone]

Re-introduce a static runtime using a feature flag
[..]
The primary danger here is silently ending up with the "split application". Since feature flags are additive, a library or component may include the rt-global feature flag and the application does not know it is accidentally using the global runtime.

So one mitigation here is to address the "silent" aspect (emphasis added).

The global runtime could announce its creation (via log, stderr, etc - details important, but for later) unless suppressed explicitly by the developer because they want the behaviour. Suppression markers could include:

• that the feature was defined in the top-level binary crate Cargo.toml, rather than a dependency library (if it's possible to tell, somehow?)
• when it's started via #[tokio::main]
• when it's started by explicit manual configuration

So, basically, squeal rather than panic!

Finally, general observability and debugging aids (so I can see what runtimes are instantiated, and which tasks are using them). Again this might be addressed by simply changing some log levels and documentation/guidance on things to check in logs for your debug builds and test runs.

[hawkw]

• that the feature was defined in the top-level binary crate Cargo.toml, rather than a dependency library (if it's possible to tell, somehow?)

I don't think there's really any way to determine this, unfortunately.

[djc]

From these options, I (as someone who has definitely hit this a few times and remembering being pretty confused) feel that the current situation is the most attractive if the error message was sufficiently good. Maybe we could even include a link to a documentation page that lays out (a) ways to fix it and (b) background why this is the case (as laid out here -- thanks for writing it up so clearly!).

[najamelan]

I think a good error message combined with good documentation can alleviate this problem. Looking at the docs for the runtime module, I feel it's not as clear as to what is going on under the hood as reading this issue.

• The first phrase there says "the following runtime services are necessary", which is not really true. It depends on what one wants to do, but it doesn't explain that certain types in the tokio library will require certain components to be turned on in a global (or TLS) way. And it doesn't make it clear which parts of tokio depend on what.
• The example of using a closure with enter as showed above doesn't appear on that page either.
• Further confusion (for me that was) comes from the fact that even though it describes the basic scheduler, it doesn't explain why this doesn't support !Send tasks and doesn't mention LocalSet, nor how it can work together with it.
• Lastly, tokio Runtime uses AssertUnwindSafe on user's code, but doesn't mention Unwind safety in the docs.
• From my testing it seemed that Handle will silently drop tasks if the Runtime is no longer running, but that isn't mentioned. The general docs for Handle are one phrase, and don't really mention why it exists and when (not) to use it.

All in all I think the OP makes it clear there is no magical solution for this. The robust type system way is to pass a Handle around, but it's understandable that is not necessarily convenient and comes at a perf cost. Thus I think solid docs and error messages are paramount.

Alternatively a guide level documentation on tokio.rs and a link in the API docs that really walks through the reasoning behind the Runtime design might be a good solution.

[hdevalence]

Hi there, as a Tokio user I don't have a particular positive preference about which strategy to use, but I have a very strong negative preference against adding a feature flag, because of the "spooky action at a distance" disadvantage mentioned above.

Tokio's features are already very complicated and it's easy to end up with things only working accidentally (your code uses some part of Tokio that's feature-gated without setting the feature, but it works anyways because some Tokio-using library in your workspace set the feature for you). Although I am sympathetic to the concerns that led to this design, as a user I find it extremely confusing and I think that adding new features will make the problem worse.

[carllerche]

@hdevalence

Tokio's features are already very complicated and it's easy to end up with things only working accidentally (your code uses some part of Tokio that's feature-gated without setting the feature, but it works anyways because some Tokio-using library in your workspace set the feature for you).

One option would be to enable full in your app and just assume everything is present.

Regarding adding a feature flag, what if the feature flag didn't enable automatic starting of the static runtime. Instead, a user that wanted the static runtime would need to specifically start one (maybe w/ a runtime::Builder option). This would reduce the chance of something happening accidentally.

[hawkw]

Instead, a user that wanted the static runtime would need to specifically start one (maybe w/ a runtime::Builder option).

How is this noticeably different than #[tokio::main]?

My understanding is that the general source of confusion we're trying to solve here is "panics occur when users don't take explicit action to create a runtime". Providing a static runtime to be started via a builder option or something could be useful, but I don't it really solves that problem?

[carllerche]

My guess, for #[tokio::main] is that it could be an option:

#[tokio::main(static)]
async fn main() {
}

or something...

[carllerche]

In general, I think we can agree that feature flags should not change behavior (I am aware I failed some on this front in 0.2).

[jebrosen]

A few questions and opinions on this topic, not especially organized:

• The panic message should be more detailed if it's kept in. It can happen a lot of different ways, which may make it difficult to explain well without confusing the reader with irrelevant information:

  • Using a non-tokio runtime
  • Using a Runtime, but trying to do some setup work outside the runtime first (e.g. as seen in seanmonstar/reqwest#778)
  • Using a Runtime, but left it - I'm not sure exactly which of spawn_blocking, block_in_place, etc. "leave" which parts of the runtime. Clearly thread::spawn does.

• What is the consequence of polling a runtime-bound resource from a different runtime after it's already been created?

  • Is it less performant (and why?)
  • Does it panic?
  • How much of this should be considered implementation details vs public API? Is it a breaking change to add or remove the "runtime-boundness" of a resource?

• Having both implicit and explicit constructors available, or multiple equivalent code paths that do the same thing, leads to confusion.

  • For example, TcpListener should either take an explicit Handle or use the thread-local - but not have one constructor for each option.
  • Since Handle is not a method parameter, it is not clear which methods or modules require a tokio runtime and which do not unless you try one and it panics. For this reason I have a slight preference toward explicit Handle, but I also agree it's probably too verbose for most use cases and everyone will just use Handle::current() most of the time.

• tokio should not start a background runtime automatically. This seems like too big of a footgun, such as a library or application working fine in isolation but breaking or performing badly when combined with other applications/libraries when multiple runtimes compete against each other for CPU time.

[najamelan]

Hope it's not to off topic, but I just remembered other gotcha's I ran into:

• block_on within a task spawned on a runtime panics
• runtimes shouldn't be dropped from async context.

[shuoli84]

Is there a way to prevent tasks or resources cross runtime boundary? E.g, resource is owned by this runtime, if you need to await it on another runtime, then an explicit bridge must be created. Or any sharing between runtime must through some runtime(or even crate) independent channel.

[Darksonn]

I will try to answer some questions that have been asked here.

What is the consequence of polling a runtime-bound resource from a different runtime after it's already been created?

The main issue is not that it is polled from a different runtime — rather the issue is that you now have two runtimes. A Tokio runtime spawns a thread for every cpu core, so if you spawn two runtimes, you have more threads than you have cpu cores, which can lead to inefficiencies.

There's also the fact that it guarantees that your IO wont be handled on the same thread as where the future is polled. It is more efficient not to cross a thread boundary. Of course, this doesn't always happen with a single runtime either unless you use the single-threaded scheduler, but it does happen for some of the tasks.

Is there a way to prevent tasks or resources cross runtime boundary?

Not really.

I'm not sure exactly which of spawn_blocking, block_in_place, etc. "leave" which parts of the runtime. Clearly thread::spawn does.

All threads managed by Tokio are inside the runtime context, so this includes spawn_blocking and block_in_place. Calling tokio::spawn and friends from inside them should work.

Threads spawned by other means are not inside the context unless they explicitly enter it with enter.

Hope it's not to off topic, but I just remembered other gotcha's I ran into:

• block_on within a task spawned on a runtime panics
• runtimes shouldn't be dropped from async context.

Yeah well this is just the sort of thing you fundamentally have to avoid in async/await, because tasks must regularly yield control back to the executor to allow other tasks to run, and block_on doesn't do that. There is block_in_place, but it's a pretty big footgun.

[carllerche]

I added a new option:

Provide a separate tokio-global-runtime crate

The specific name of the crate would need to be massaged. The idea is to have a separate crate define the static variable. In this case, users who wish to use a statically defined runtime would depend on this crate.

The main downside as I see it to this is that it makes the global runtime less discoverable.