Rust: Indicate compilation progress in a form useful to end users

[kmcallister]

Was #22227, closed due to vandalism.

@sanxiyn said:

I think we should have --progress option. This could show phases, and for long phases like type checking, this could even show the name of the function being checked.

To which I replied:

+1. We can use control codes to update parts of the terminal without spewing output.

I will think about how to integrate this with --color and the drawing characters.
I'm imagining that rustc in whiz-bang fancy terminal mode would display progress on a single line, updated at each stage:

[/] Parsing
[-] Expanding macros
[\] Resolving names
[|] Checking types
[/] Checking lifetimes
[-] Checking
[+] All checks passed; generating LLVM
[+] All checks passed; optimizing
[+] All checks passed; generating machine code
[+] All checks passed; linking
[+] Compilation successful (in 0m 18s)
your-prompt$

And just that last "success" message will remain in your terminal buffer.

The spinner has a constant speed in wall-clock terms. Long-running parts of rustc will call sess.update_spinner() now and then to keep it going.

After we pass the "rest easy" point, the spinner turns into a calming + and the first part of the message turns green.

[blaenk]

Yes this would be so cool.

[asolove]

I'm brand new to contributing, but I'd like to take a crack at this one. I'm pretty sure I can do the pretty terminal bits, but I would probably need help keeping the spinner running.

[asolove]

I don't understand the relationship between the term repo and the libterm folder in this repo. But over in the term repo, I am working on an already-extant issue to add more features to the Terminal interface. Presumably if that lands there, it could get merged in here and be available for this PR.

[blaenk]

I'm not so sure myself, but my understanding is that since rustc doesn't use cargo, they embed the crate directly into this source tree.

[asolove]

Quick update: the needed terminal stuff landed in the standalone term crate, but there is some work needed to get the version in libterm here to match that, because the term crate now uses winapi bindings that aren't yet available here. I don't fully understand how to make that happen, and if adding all that stuff to this repo is worth it over this little progress animation.

In the mean time, I also started working on the substance of this issue in my progress branch. I think I can finish off the pretty-printing stuff, but I will still need helping figuring out how to get the spinner animation to work while other long-running compile phases are going on.

[retep998]

@asolove You should be able to just add the required definitions to the liblibc in here.

[dcampbell24]

What happens when things other than the progress notifications are printed to the screen? I like how apt-get handles this. The bottom line displays progress while all the other output gets printed above it.

[|] Checking

warning: this is a warning
[/] Checking

warning: this is a warning
warning: some other warning
[-] Checking

warning: this is a warning
warning: some other warning
another message from rustc
[\] Checking

[asolove]

@dcampbell24 Yup, that's how this will work too. I think the practical way to get that behavior is to print out and then do a carriage return to move back to the beginning of the line you just printed to. Other output will overwrite it and add a newline and then we'll print the next status update to the bottom of the output.

[retep998]

@asolove Don't forget that you have to delete the entire line before writing more output, because if the live message is shorter than the line that is printed out, some will remain.

For example ideally:

[-] Checking

Hi!
[\] Checking

And if you forget to delete the line first

[-] Checking

Hi! Checking
[\] Checking

[asolove]

Yup, left that out. My current plan is: delete the current line, write a line of status, then carriage return to end up at the beginning of the line again.

In that scheme, only the last line of status output is ever displayed in the console. I don't understand how apt manages to leave behind its old status lines when other parts of the program write output.

[kmcallister]

What about multiple rustc instances running in parallel?

[asolove]

Had not thought about that at all. What do you think is the desired behavior there?

[kmcallister]

What I really want is an option for cargo build that shows progress bars for every build in progress, as well as the list of crates still in the queue. Mostly because I think it would be super fun to watch.

[asolove]

That sounds super cool and I have no idea how to do it...

(Spends a few minutes looking through the cargo source.)

It looks like right now we're shelling out to separate rustc processes that don't know about each other and would have a hard time collaborating in this kind of animation. Each process could potentially save the cursor position as of when it started and always write to that line. But any other output from the compiler would potentially overwrite other lines of the animation until they redraw.

Alternately, we could redirect the output of those processes somewhere else, parse it, and have the actual final animation driven inside cargo.

[richo]

My 2c (with really no basis)

What you probably want is a --status-pipe or somesuch, that causes rustc to
emit machine readable status data instead of the spinner, and then have
cargo parse that and draw the same thing in both places. Then ideally
extract that actual render-y code out into a new crate that both cargo and
rustc can consume the same way (modulo that rustc's printer reads it from a
channel instead of a pipe).

I'm happy to chat more about this approach, although I'm not convinced it's
correct, it was just the most obvious thing to me

On Tue, May 5, 2015 at 11:23 AM, Adam Solove [email protected]
wrote:

That sounds super cool and I have no idea how to do it...

(Spends a few minutes looking through the cargo source.)

It looks like right now we're shelling out to separate rustc processes
that don't know about each other and would have a hard time collaborating
in this kind of animation. Each process could potentially save the cursor
position as of when it started and always write to that line. But any other
output from the compiler would potentially overwrite other lines of the
animation until they redraw.

—
Reply to this email directly or view it on GitHub
https://github.com/rust-lang/rust/issues/24335#issuecomment-99166211.

[asolove]

Spent some time making a prototype to see what the end result could look like: see the video or try the code

I assume if we got error output from any of the compilations, we'd mark that in red and show the errors underneath.

[tamird]

@asolove that's dope.

[pcn]

@asolove: woah. That is some quality console eye candy.

[mkpankov]

@asolove holy cat! :+1: Very cool.

As a side note, little things like these probably indicate how badly rustc needs to be optimized to speed up compilation :(

[retep998]

@asolove Maybe you could try actually using the functions you added to term instead of using ansi escape codes directly? :stuck_out_tongue:

[asolove]

@retep998 the latest release on crates doesn't have them yet and I wanted others to be able to run the code directly. If there's a way to configure it to use master on that repo, I'd love to know.

[retep998]

@asolove

[dependencies.term]
git = "https://github.com/rust-lang/term.git"

[asolove]

Aha, yay for learning things.

[kmcallister]

Jeez, how big are the test crates? I've never seen "Parsing" take multiple seconds even on a huge crate.

[asolove]

Yeah, this is a simulation! I'm just walking things through the states at random intervals.

[kmcallister]

Ah, okay :) The output looks really nice!

[Nashenas88]

@asolove That animation is really cool looking. When you do get it working, I suggest trying to build servo with it to see how that outputs. On my 4 year old macbook pro, it takes about 10 minutes. (To build with multiple threads, pass -j # where the # sign is the number of threads you want to run)

[xaviershay]

Thread is looking a little dead, so going to put in a vote of support for anyone still interested in working on this. I'm not familiar enough with rust yet, but my compile times are really long and I have no insight into why. This would really help!

[xaviershay]

I don't even need pretty animations. _Any_ kind of indication of what's going on would be appreciated.

[xaviershay]

I discovered the following which gives me what I need: cargo rustc --lib -- -Z time-llvm-passes -Z time-passes

[ticki]

This would be very useful. I guess this goes hand-in-hand with #3533 and #29989.

[mikosdikos]

This could have given me a clue earlier!
https://github.com/rust-lang/rust/issues/31164

[retep998]

Doesn't rustc now have an unstable option to emit errors as json? That should probably help with this somewhat.

[frewsxcv]

Errors don't indicate compilation progress though, unless I'm misunderstanding something...

[alexcrichton]

I was playing around a week or so with an idea that I had, and I came up with a cargo fancy subcommand. It's certainly stable by no means nor will it be shipping with Cargo any time soon, but I had some fun working with it and others may find it interesting as well!

[brson]

Still a worthy goal, but not "easy" since there is no agreed upon design. Accounting for the entire cargo pipeline like @alexcrichton did will be important for real use.

[Mark-Simulacrum]

So I'm not entirely sure that this is a good idea for rustc itself, especially since it seems that tools like cargo fancy are feasible. Should we be tracking "compilation progress in a fancy way" here? -Ztime-passes works, though is unstable and not at all pretty.

[kennytm]

cargo fancy relies on -Z time-passes to work, and -Z time-passes itself is not suitable for printing progress since it shows the message when a step is finished, but for progress reporting it is better we get the message when a step starts.

I agree rustc itself shouldn't do the animation, and @richo's idea of "printing to a pipe" is better. Writing to stdout or stderr will cause bugs for tools which have strict requirement on the output format of rustc.

Probably related to #42678.

[steveklabnik]

Triage: cargo now has a progress bar to show progress, but at the granularity of x/y crates, not for the compilation process of each crate.

[Mark-Simulacrum]

This seems hard to do today and even harder to do over time as we're slowly moving towards not having "phases" of compilation. We do emit a notification when we move to codegen today and there's an issue on Cargo https://github.com/rust-lang/cargo/issues/7265 about actually displaying that. I think further improvements here are likely of the scope that prior discussion on internals and maybe even a compiler team planning meeting are necessary (i.e., how to architect useful progress in a query-based compiler). As such I'm going to close this issue.