Julia: Generalized progress API (for download etc.)

[ihnorton]

From https://github.com/JuliaLang/IJulia.jl/pull/211:

@stevengj

I wish that the download function had an option to display some kind of progress indicator, but I agree that at a minimum we should print out some status updates for a long build.

@StefanKarpinski

In general, I think we need a standard pattern for functions that might or might not want to present updates. One thing that comes to mind is having a status::Callable=(args...)->nothing keyword argument that defaults to ignoring status updates.

@vtjnash

@StefanKarpinski that sounds a lot like the Julia-side of the libuv-API (esp. for Process) pre-Jeff's cleanup to use Tasks rather than status callback functions :)

[ihnorton]

My $.02: I like the iterator-style design of @timholy's https://github.com/timholy/ProgressMeter.jl ... I think bringing that into base with a generalization to a display-style stack would be ideal, so that IJulia and all the rest can provide hooks as appropriate.

[tknopp]

Yes ProgressMeter.jl is really nice. Was so happy when I first discovered it. Would be great to make the architecture general enough that one potentially could plug a Gtk progress bar to it when programming a GUI with Gtk.jl.

[ivarne]

Why not just let IJulia depend on ProgressMeter.jl, and help make it better where it is?

[timholy]

Why not just let IJulia depend on ProgressMeter.jl, and help make it better where it is?

Base's download function would have to depend on it. Or of course IJulia could rewrite the download function.

I think bringing that into base with a generalization to a display-style stack would be ideal, so that IJulia and all the rest can provide hooks as appropriate.

I'm fine either way. Will submit a PR if it seems there is sufficient interest, but it does seem worth exploring whether it would be usable to leave it as a package.

[stevengj]

For something like this to work in IPython, I think it would have to hook into the new interactive-widget stuff (cc @shashi).

[shashi]

Here is the code you can use to show a progress bar in IJulia:

using React, Interact
i = Input(0)
@lift Progress(value=i) # you can specify range::Range kwarg for a range. It's 0:100 by default
push!(i, 20) # Set the progress to 20%. Should probably do this with @async inside an event loop.

i could be substituted with any Signal of Integers. E.g.

using React, Interact
s = Slider(0:100)
@lift Progress(value=signal(s))

This will give you a slider and sync its value with that of the progress bar. One should be able to write counterparts to Interact widgets (Progess is one of them) in Gtk, and have something similar.

[StefanKarpinski]

I kind of like that line of thought but I'm not sure we really want to be updating on every iteration.

[shashi]

Ah, I don't know the entire context here, but calling @async push!(i, <progress>) whenever you _want_ to update the progress would do that. You can use @lift Progress(value=droprepeats(i)) to only update the progress widget if i has changed since the last update. This is cheap. In fact updating the progress is cheap too. The progress widget is not actually redrawn, just updated.

[timholy]

Perhaps we should see if we can integrate this with ProgressMeter, which is smart enough (1) to update no more frequently than some specified temporal interval, and (2) not display anything at all if the calculation finishes before the first interval expires, but (3) doesn't force the user to think about any of this stuff (you notify ProgressMeter of incremental progress, and it decides whether to update the display).

[shashi]

I just tried out ProgressMeter. That's very useful!!

I tried making my React example more feature rich, I like how it reads :D :

using React, React.Timing
function progress(n, dt)
   t0 = time()
   ticks = every(dt)
   input = Input(nothing)
   count = foldl((a, b) -> a+1, 0, input)
   completed = @lift droprepeats(sampleon(ticks, count)) / n * 100
   eta = @lift (ticks - t0) * (100 - completed) / 100
   input, completed, eta
   # you can push!(input, nothing) to increment count
   # completed holds the % completed
   # eta is the ETA
end

This has a few issues: 1) ticks never stops updating (React lacks this feature now. I'll fix that.) 2) I don't know how I could not update anything at all if the calculation finishes within dt, but could find a way.

ProgressMeter looks self sufficient though. OTOH, I will do a PR integrating ProgressMeter with Interact's progress bar so that ProgressMeter is prettier on IJulia.

[StefanKarpinski]

The algorithm side should also not need to care whether there is a) no way to output any progress, b) only a text terminal, or c) it's in IJulia or some other rich environment. It would also be great to be able to retroactively decide that you want to connect to something that's ongoing and see its progress – how many times haven't we all regretted not turning on progress output when trying to figure out if some long-running computation is hung or just taking a _really_ long time.

[timholy]

The retroactive thing would be great, but honestly I have no idea how to pull that off. Perhaps by disassembling the code_native output it would be possible, but I think that officially counts as heroics.

[Keno]

Adding it to the TODO list ;)

[timholy]

I just checked my English-to-Julian dictionary, and it turns out that "Keno" actually means "heroic." Will coincidences never cease...

[Keno]

Seriously though, this will be pretty easy to do with the debugging stuff I'm working on.

[timholy]

That will be amazing.

[c42f]

I just came across this old issue. I'm thinking about the best way to build this into MicroLogging, as I think progress reporting is not very different from other types of logging. The current idea is that it's just a progress key in a standard structured log record:

for i=1:100
    sleep(0.01)
    @info "Algorithm 1" progress=i/100
end

This way the user specified logger gets to decide how to display progress messages - or filter them as desired - and they can interact properly with other non-progress log messages. It's a proof of concept still, but seems to be turning out quite naturally.

[pfitzseb]

Juno will have something like this on 0.7 (see here); should be rather easy to have ProgressMeter.jl conform to a simliar API.

[cossio]

Any updates on this? Particularly for Base.download(...), is there a way to print progress (in Juno, IJulia, or the REPL)?

[StefanKarpinski]

No, since it uses various external programs which don’t have a consistent way of reporting progress. However, we would like to replace that approach with a native Julia implementation which could report progress.

[c42f]

The logging-based progress APIs have been improving lately.

With https://github.com/JunoLab/ProgressLogging.jl for the frontend and https://github.com/c42f/TerminalLoggers.jl for the in-terminal formatting I think we've getting close to a slick solution (see demo at https://github.com/c42f/TerminalLoggers.jl/pull/13)