Dvc: pull/fetch: progress bar is all over the console output

[nategadzhi]

DVC version:

DVC version: 0.91.1
Python version: 3.7.7
Platform: Darwin-19.4.0-x86_64-i386-64bit
Binary: False
Package: brew
Supported remotes: azure, gdrive, gs, http, https, s3, ssh, oss
Cache: reflink - supported, hardlink - supported, symlink - supported
Repo: dvc, git

When I run dvc pull, dvc tries to show the progress bar for downloading data files, but it looks like there's a bug in rendering it.

Here's the video: https://share.getcloudapp.com/jkuKG6rp

Please let me know if I can provide more details on this or help in any way. I know this reproduces for one of my teammates, too, on Ubuntu with DVC 0.91.0.

[skshetry]

Hi, @xnutsive. Thanks for opening this issue.

The files that you are downloading are small, and hence looks flickering, and is a problem on the
fetch part.

3500 set a threshold for showing maximum number of bars and hides them (it was more crazier before), but we still need to unify multiple bars into a single bar. This was done on upstream tqdm/tqdm#924.

Related/part-of #2905.

@casperdcl, I assume this is going to be a bit nicer with tqdm/tqdm#930. I notice that the bars are getting placed randomly on each render making it look flickery?

[casperdcl]

The ordering isn't random - the screen is actually "full" of bars "in order." To maintain the order, the whole screen below a bar is cleared when said bar completes, and the cleared bars are moved upwards. However, some (most) are randomly skipping display for speed purposes.

https://github.com/tqdm/tqdm/issues/930 will actually fix this by changing to random ordering. This means finished bars get replaced without clearing others.

(disclaimer: this is a simplified explanation; in practice it's a little more complicated because concurrency + UI + cross-platform support + speed optimisation = 4 x hell more fun)

[nategadzhi]

Experience-wise (please take this with a grain of salt, I'm not by any means a serious ML practitioner), a single pbar would be nicer — I don't really care how much of a specific image / text is downloaded, I just care about the whole data downloading / uploading process.

I've read another issue where you discuss this and I imagine doing a single pbar is a ton of work though ;)

Let me know if I can be useful in the meantime!

On Tue, Mar 31, 2020 at 11:52 PM, Casper da Costa-Luis < [email protected] > wrote:

The ordering isn't random - the screen is actually "full" of bars "in
order." To maintain the order, the whole screen below a bar is cleared
when said bar completes, and the cleared bars are moved upwards. However,
some (most) are randomly skipping display for speed purposes.

tqdm/ tqdm#930 ( https://github.com/tqdm/tqdm/pull/930 ) will actually fix
this by changing to random ordering. This means finished bars get replaced
without clearing others.

(disclaimer: this is a simplified explanation; in practice it's a little
more complicated because concurrency + UI + cross-platform support + speed
optimisation = 4 x hell hard)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub (
https://github.com/iterative/dvc/issues/3565#issuecomment-607068855 ) , or
unsubscribe (
https://github.com/notifications/unsubscribe-auth/AAAKU4IJX6UYETDADQ3QVLDRKLQDXANCNFSM4LYGK3HA
).

[casperdcl]

3583 should mostly fix this issue.

@xnutsive I agree with your last comment - #3452 should be fixed by adding missing "main bars." The nested ones are relatively minor by comparison.