Dvc: pseudo-tty verbosity

[casperdcl]

continuing discussion from #2659,

Envs like gitlab don't support carriage return \r so progressbar updates get printed on new lines (https://github.com/tqdm/tqdm#faq-and-known-issues).

possible solutions

To disable and/or reduce progress frequency:

• check os.environ for gitlab (and other CI?) env vars
• support a new dvc env var
• add another dvc log level
• document redirecting stderr to file (dvc ... 2>/tmp/dvc-stderr.log)
• document redirecting stderr to tee (dvc ... 2>&1 | tee)

related to #2329, #2332

[efiop]

I wonder how people using gitlab are dealing with stuff like wget and other tools that have progress bars? I have run into similar issues previously and using -q with other tools was suitable for me. Or redirecting to a log file and then doing tail log.log when the process is done.

[efiop]

CC @BernMcCarty

[efiop]

Looks like the issue was that -q is not working https://github.com/iterative/dvc/issues/2665 , so there is no need for any of those possible solutions.

[BernMcCarty]

It appears gitlab has known about their \r problem for two years.

[casperdcl]

As @BernMcCarty mentions, -q suppresses (or is meant to) more than just progress, which may not be desirable

[BernMcCarty]

Redirecting stderr to a file in order to get the progress to turn itself off worries me because if there is one thing that I really need to see in my CI logs it is errors. I'd rather manually turn progress off with something like a --no-progress option.

[efiop]

@BernMcCarty I usually do something like:

mycmd &> log.log || tail log.log

that way if mycmd fails, I get the last part of the log with the errors.

[casperdcl]

doesn't dvc ... 2>&1 | tee pretty much fix this?

[BernMcCarty]

@casperdcl I think you are right. Seems to work. Clever.

[casperdcl]

much hack such fun