Rustup: Add prelimenary telemetry [easy]

[brson]

We have always wanted better insight into what people are doing with Rust. Knowing things like which errors people hit the most will help us know what areas of development to focus on. rustup, with it's compiler proxies is a perfect place to add instrumentation to observe what the compiler is doing.

This is a sensitive subject: data gathering is always controversial so we have to be completely upfront about what we collect, when and why, and we have to legitimately not collect personally-identifying information.

Additionally, every bit of code on the proxy path reduces the compiler's performance slightly. Whatever we do can't degrade performance significantly.

Here's how I'd like to start prototyping:

Define several events that we're interested in, as an enum. To start with:

• RustcRun - The duration of a rustc invocation, it's exit code, and any error numbers produced.
• ToolchainUpdate - The normalized name of a toolchain being updated, and whether it succeeded.
• TargetAdd - The normalized name of a toolchain being added to, the target triple, and whether it succeeded.

Every event is paired with a UTC timestamp.

Before rustup exits, serialize each event to json and emit it as a new line in a log file. Add two new commands: rustup telemetry on, and rustup telemetry off. By default it should be off.

We'll at some point need to discuss with fx developers how they do telemetry, since they've had a lot of experience.

[mitsuhiko]

Please, please, please make this opt-in with any flow going to "no" by default. You can encourage people with various means to opt into it at a later point easily. For instance the compiler could spit out periodically that telemetry is disabled but enabling it might improve your experience going forward, something like this.

[peschkaj]

@brson I'd like to take this on. Is there a set deliverable date or process?

[jdm]

Consider making use of https://github.com/Yoric/telemetry.rs which is written based on Firefox's experiences with telemetry.

[SimonSapin]

Consider making use of https://github.com/Yoric/telemetry.rs which is written based on Firefox's experiences with telemetry.

CC @Yoric

[brson]

Please, please, please make this opt-in with any flow going to "no" by default. You can encourage people with various means to opt into it at a later point easily. For instance the compiler could spit out periodically that telemetry is disabled but enabling it might improve your experience going forward, something like this.

It'll be opt-in. Right now I'm thinking we'll wait for a week to see if you are a 'serious' Rust user then print a hint.

[brson]

@brson I'd like to take this on. Is there a set deliverable date or process?

@peschkaj Please do. I'd like to have this in by the time we announce rustup for wider use so that we firmly establish the precedent that rustup does telemetry from the start. Right now I'm looking at the end of April for that.

[brson]

Consider making use of https://github.com/Yoric/telemetry.rs which is written based on Firefox's experiences with telemetry.

Nice! @peschkaj this is pure-Rust. Looks very promising.

[brson]

Also, to start let's not worry about reporting the telemetry back to a server - just store and rotate logs.

[peschkaj]

to start let's not worry about reporting the telemetry back to a server - just store and rotate logs.

:+1: I was thinking of rotating logs daily by writing into a file named telemetry-YYYY-MM-dd. I figured that pushing logs to a server would come later (especially since you didn't mention it in the initial issue).

I currently have a scaffold in place and was working on getting the JSON serialization going. I'll take a look again at Yoric/telemetry.rs to see if it meets our needs. On my first glance, it looked like it was geared toward creating a histogram from a single run, but that was only a quick glance.

[brson]

I am super eager to hear @Yoric's opinions on Rust telemetry.

[Yoric]

The parts of https://github.com/Yoric/telemetry.rs that have landed are indeed targeted towards creating a histogram from a single run. The objective was to provide the core data structures, and then to add additional layers (most likely application-specific) to implement disk storage and server upload.

I haven't had time to do this work on higher layers, unfortunately, but I'd be glad to give a hand if this is useful.

[peschkaj]

@Yoric Appreciated. I've run into one blocker.

The first is I've found is that our data structures don't seem to match up with what telemetry.rs is expecting. As best as I can tell, telemetry.rs is looking for either a function to measure multiple times (e.g. the fibonacci example), or else it's recording a count/flag of the use of something.

In this use case, we're looking to measure the execution time of rustc as executed through rustup's proxying. It's easy enough to wrap timers around it, but it seems more difficult to wrap the telemetry library around it to give us what we want.

[Diggsey]

This exists - lets have separate issues for further development.