dvc: invisible stderr ERROR message in some terminals?

[jorgeorpinel]

UPDATE: I'm using Git Bash (via std. Cmder) on Windows
UPDATE: Go to https://github.com/iterative/dvc/issues/3902#issuecomment-636106003

Bug Report

λ dvc run -n test2 -d output.data -o out-short.data head output.data > out-short.data
ERROR

The error is that I'm not wrapping the command in quotes but the output is just ERROR.

Please provide information about your setup

Output of dvc version:

λ dvc version
DVC version: 1.0.0a6+283718
Python version: 3.8.2
Platform: Windows-10-10.0.18362-SP0
Binary: False
Package: None
Supported remotes: azure, gdrive, gs, hdfs, http, https, s3, ssh, oss
Cache: reflink - not supported, hardlink - supported, symlink - supported
Filesystem type (cache directory): ('NTFS', 'C:\\')
Repo: dvc, git
Filesystem type (workspace): ('NTFS', 'C:\\')

Additional Information (if any):

If applicable, please also provide a --verbose output of the command, eg: dvc add --verbose.

[skshetry]

Not much dvc can do here.

Unless quoted, shell will assume > is a redirection and dvc will only get upto head output.data and the stdout will be redirected to out-short.data (here, other INFO level messages got redirected) and you should be getting:

ERROR: output 'out-short.data' does not exist

[efiop]

@skshetry But error should go to stderror and not to stdout. Why are we only getting the prefix?

[skshetry]

@efiop, I am not able to reproduce that. I am getting the error message, so, I am waiting for @jorgeorpinel to respond with more information.

[jorgeorpinel]

What is the question for me, @skshetry? This is still happening to me.

λ dvc run -n badout -o out.txt echo 'out' > out.txt
ERROR

[jorgeorpinel]

Wait... the stderr is there! But I have to select it:

so the original bug report is invalid.

However maybe there's a problem with the output coloring?

[skshetry]

@jorgeorpinel, can you try the following snippet and see if it's visible:

$ echo "hello world" >&2

Just want to rule out if it's the command prompt at fault here.

Also, it would be interesting to see output of the following:

$ dvc run -n test2 -d output.data -o out-short.data head output.data 3> out-short.data

[jorgeorpinel]

Sorry for the delay @skshetry. Both outputs are visible:

[jorgeorpinel]

p.s. now I'm getting a longer invisible output with the reported command on 1.0.0a6+283718 dev version:

[skshetry]

I think the issue is more or less with Maximus5/ConEmu here.

Sorry for the delay @skshetry. Both outputs are visible:

First one demonstrates that stderr are correctly shown by term.
Second one demonstrates that stderr is correctly prepared by dvc, and supported by terminal.

Looks like ANSI Escape sequences + stream redirection is buggy on the terminal.

[jorgeorpinel]

OK np should we open an issue in their repo and close this?

[skshetry]

OK np should we open an issue in their repo and close this?

I'd leave that up to you. ;)

[majidaldo]

This isn't with conemu. Happens in cmd.exe too. In my case it happens with the git hooks.
I changed the hooks to redirect stderr like exec dvc git-hook pre-commit $@ 2>%1 so now it's visible.

Oddly, if my terminal widh is 80 chars or less, I can still see it.

ref https://conemu.github.io/en/MicrosoftBugs.html#ErasedWindows10

[efiop]

@majidaldo Btw, do you use https://pre-commit.com/ ? If so, could you give it a shot with dvc install --use-pre-commit-tool and see if that helps?

[majidaldo]

@majidaldo Btw, do you use https://pre-commit.com/ ? If so, could you give it a shot with dvc install --use-pre-commit-tool and see if that helps?

Well, I can tell you that you have to have a valid .pre-commit-config.yaml (already) before you can use --use-pre-commit-tool. It should error out with something like: .pre-commit-config.yaml not found.

[efiop]

@majidaldo Yes, it is our assumption that people already use it for something else, hence why I've asked you if you use it :slightly_smiling_face:

[efiop]

Closing since there doesn't seem like there is much we can do on our side :slightly_frowning_face: If anyone is running into this as well, please leave a comment to let us know.

[majidaldo]

@majidaldo Btw, do you use https://pre-commit.com/ ? If so, could you give it a shot with dvc install --use-pre-commit-tool and see if that helps?

Well, I can tell you that you have to have a valid .pre-commit-config.yaml (already) before you can use --use-pre-commit-tool. It should error out with something like: .pre-commit-config.yaml not found.

starting to use it. seems to be working. works even in my interoperating conda envs (though i know what i'm doing).