Azure-pipelines-tasks: return the stderr stream with "Bash wrote one or more lines to the standard error stream." for troubleshooting

[bgrounds]

Troubleshooting

Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting

Environment

• Server - VSTS or TFS on-premises?

  • If using VSTS, provide the account name, team project name, build definition name/build number:
    
    
    
    • VSTS
    
    
    • account name: watchmixer
    
    

• Agent - Hosted or Private:

  • If using private agent, provide the OS of the machine running the agent and the agent version:
    
    
    
    • private
    
    
    • CentOS Linux release 7.4.1708 (Core)
    
    
    • agent version: 2.127.0
    
    

Issue Description

When a release task is set to fail on stderr, not enough feedback is provided on failure to troubleshoot the problem.

The only feedback is "Bash wrote one or more lines to the standard error stream."
This makes it very difficult to troubleshoot those errors. It would be nice to have the stderr output.

Right here, it looks like you could accumulate the data (stderr stream) into a string, then pass it along with the error message.

https://github.com/Microsoft/vsts-tasks/blob/b6ca8a23cbdc5f825ebe3d260664073e079f1f1e/Tasks/CmdLine/cmdline.ts#L50-L52

I'm currently looking for a workaround to get those error messages, as they aren't being printed from my scripts.

[ericsciple]

it should be in the log, correct?

[ericsciple]

queue: hosted linux preview
steps:
- bash: echo hello from stderr 1>&2
  failOnStderr: true

[bryanmacfarlane]

Yes, it writes it to the log

https://github.com/Microsoft/vsts-tasks/blob/b6ca8a23cbdc5f825ebe3d260664073e079f1f1e/Tasks/CmdLine/cmdline.ts#L42

Reactivate if you're not seeing it in the log

[farrukh-normuradov]

@bryanmacfarlane, We are not seeing the error in the logs, please, reactivate.

[ericsciple]

@farrukh-normuradov can you download the logs zip and check there? Sometimes the streaming console can miss a web event. But I would expect to see the error in the downloaded log.

[farrukh-normuradov]

@ericsciple, I did that. The logs file contains the same logs that are presented through the UI. I also turned on debug mode and agent diagnostics.

The only error containing phrase is this:

I search for "err" in the logs and do not find an error.

I ran build multiple times.

Some extra info:

1. It is npm install that fails
2. I load packages from private registry
3. When I remove private registry package, the issue persists
4. Only when I remove most of the dependencies, and leave lodash I get a green build

[VelizarVESSELINOV]

It is very difficult to differentiate standard output and error stream, please improve your output log, for easy identification of the error in a long output of mixed streams.

Suggestion add a prefix:

out> bla-bla
err> focus

[keikumata]

@ericsciple

I am also not seeing any useful logs. Any chance we can see what happened?

[rishibamba]

+1 on the issue. Unable to find the errors. It is an "ant" command that is run inside the bash task. Build is successful but bash complains of errors.

Error: #[error]Bash wrote one or more lines to the standard error stream.

Task : Bash
Description : Run a Bash script on macOS, Linux, or Windows
Version : 3.148.2

Workaround: works fine if "Fail on Standard Error" this is not checked.

[benbuzbee]

We have 5000 lines of output in a build that succeeds completely, but that prints this error and marks the build as failed. It is impossible to figure out which of the 5000 lines was written to stderr. This is not a usable feature without a way to distinguish stderr output from stdout output.

If the UI could just color stderr output red, or call it out separately, or write it to a stdout.log file, or any of a number of things we could be easily unblocked.

As a workaround we have added this to the end of our script run. It will print "STDERROR" before the lines that were printed to stderr, and the extra >&2 make sure the result is still actually printed to stderror so the build will fail.
2> >(while read line; do (>&2 echo "STDERROR: $line"); done)

This is a pretty silly workaround, there should be a first class way of debugging this.

[bramvdklinkenberg]

Same issue. Running a shell script that completes and exits with 0 but the job/task fails with:
Error: #[error]Bash wrote one or more lines to the standard error stream.

But I run this script on multiple environment and 50% fails and the other 50% succeeds.

[joyl1216]

Having the same error. Any idea?
```Having the same error here.

==> Downloading from https://github-production-release-asset-2e65be.s3.amazonaws.com/45861018/d7233980-b740-11e8-8a89-b30a1e0ac118?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20191025%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191025T032427Z&X-Amz-Expires=300&X-Amz-Signature=3b8eb6339119ea8413ec2b27a14273171621c38b47d64e9fa132b7e185313db8&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dgit-credential-manager-2.0.4.jar&response-content-type=application%2Foctet-stream
🍺 /usr/local/Cellar/git-credential-manager/2.0.4: 4 files, 2.4MB, built in 4 seconds
Cloning spec repo msblox from [email protected]:v3/msblox/CocoaPods/CocoaPods

[error]Bash wrote one or more lines to the standard error stream.```

[denravonska]

Same problem here. Maybe it does indeed redirect the error to stdout, but since it's indistinguishable from the regular output there's no way of knowing which line that's the culprit.

[benbuzbee]

Same problem here. Maybe it does indeed redirect the error to stdout, but since it's indistinguishable from the regular output there's no way of knowing which line that's the culprit.

Yes @denravonska that is correct.

The script I pasted above is intended to allow you to annotate the output of bash with STDERROR for anything that goes there, for example

bbuzbee@bbuzbee-dev:/$ (ls broken || ls .) 2> >(while read line; do (>&2 echo "STDERROR: $line"); done)

Will print

STDERROR: ls: cannot access 'broken': No such file or directory
bin  boot  cdrom  dev  etc  home  initrd.img  initrd.img.old  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  snap  srv  swapfile  sys  tmp  usr  var  vmlinuz  vmlinuz.old

This has been invaluable for my team to diagnose what actually failed when shelling our to a sufficiently complex command.

I suggest you folks bang on PR #9391 since @damccorm made a simple fix for this a year ago that is being soundly ignored and simply needs to be looked at by an area owner at Microsoft like @stephenmichaelf @jtpetty.

[denravonska]

@benbuzbee Thanks, I'll try it out. I just assumed that failure triggers would verbosely be printed in the logs. Edit: like all other CI systems.

[denravonska]

@benbuzbee Your suggestion worked perfectly, without it I would have had no idea how to track down the issue. It goes to show how important such a trivial feature is to the users.