Azure-pipelines-tasks: helm3 - setting system.debug causes "helm upgrade" pipeline to be marked as fail even though it succeeded

[hhellbusch]

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug/Feature

Enter Task Name: helm

Environment

• Server - Azure Pipelines or TFS on-premises?
  Azure DevOps / Pipelines

Issue Description

The debug output of helm3 is being treated as an error message. This causes the stage to be marked as failed

To reproduce, set the pipeline variable system.debug to true.

2019-09-19T17:42:18.7081006Z ##[debug]execResult: {"code":0,"stdout":"NAME: project-dev\nLAST DEPLOYED: 2019-09-19 17:42:15.684362147 +0000 UTC m=+0.537851092\nNAMESPACE: project-dev\nSTATUS: deployed\nRelease \"project-dev\" has been upgraded. Happy Helming!\nNAME: project-dev\nLAST DEPLOYED: 2019-09-19 17:42:15.684362147 +0000 UTC\nNAMESPACE: project-dev\nSTATUS: deployed\n","stderr":"history.go:52: [debug] getting history for release project-dev\nupgrade.go:79: [debug] preparing upgrade for project-dev\nupgrade.go:88: [debug] creating upgraded release for project-dev\nupgrade.go:94: [debug] performing update for project-dev\nclient.go:140: [debug] checking 4 resources for changes\nwait.go:50: [debug] beginning wait for 4 resources with timeout of 5m0s\nupgrade.go:101: [debug] updating status for upgraded release for project-dev\n"}
2019-09-19T17:42:18.7082076Z ##[debug]task result: Failed
2019-09-19T17:42:18.7138882Z ##[error]history.go:52: [debug] getting history for release project-dev

posted at https://developercommunity.visualstudio.com/content/problem/740540/helm3-setting-systemdebug-causes-helm-upgrade-pipe.html and they suggest updating the task to handle this situation.

[vithati]

@hhellbusch , which task are you using? can you share task log?

[hhellbusch]

@vithati

2019-09-19T18:17:16.0771338Z ==============================================================================
2019-09-19T18:17:16.0771489Z Task         : Package and deploy Helm charts
2019-09-19T18:17:16.0771632Z Description  : Deploy, configure, update a Kubernetes cluster in Azure Container Service by running helm commands
2019-09-19T18:17:16.0771783Z Version      : 0.156.0
2019-09-19T18:17:16.0771895Z Author       : Microsoft Corporation
2019-09-19T18:17:16.0772031Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/helm-deploy
2019-09-19T18:17:16.0772347Z ==============================================================================

See attached for the task log.
tasklog_9.log

Let me know if there's any other information that would be useful.
Cheers!

[Anumita]

@hhellbusch , thanks for reporting this issue. I have checked in the fix for this(https://github.com/microsoft/azure-pipelines-tasks/pull/11473) and the fix should reach you in 2-3 weekes. Till then you can remove the --debug option as a workaround.

[hhellbusch]

That's great! Thanks for fixing @Anumita. I'll test on my end once the fix has been rolled out.

[jcoeltjen]

Still an issue with task version 0.159.3

2019-11-29T20:34:29.3012051Z ==============================================================================
2019-11-29T20:34:29.3012198Z Task         : Package and deploy Helm charts
2019-11-29T20:34:29.3012315Z Description  : Deploy, configure, update a Kubernetes cluster in Azure Container Service by running helm commands
2019-11-29T20:34:29.3012419Z Version      : 0.159.3
2019-11-29T20:34:29.3012527Z Author       : Microsoft Corporation
2019-11-29T20:34:29.3012596Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/helm-deploy
2019-11-29T20:34:29.3012707Z ==============================================================================

Looking forward to @Anumita changes to be available!

[sam-cogan]

Yep, still seeing this

[ankrause]

Still seeing this as well. Based on the commit it looks like the change was slated for version 0.159.4, but in ADO I still see it failing under version 0.161.0. Is it possible to get an update on when this fix will be available? Thanks!

[vithati]

Helm 3 writing some of the logs to stderr. If stderr contains messages the task will fail (. https://github.com/microsoft/azure-pipelines-tasks/issues/11935). This is issue with helm3. If your helm chart works with helm2. Can you install helm2 version using "Helm tool installer", untick 'Check for latest version of Helm' in task and try again.

[sam-cogan]

I know Helm 2 works OK as I've been using that for some time. This only occurs with Helm 3.

[pkleimann]

There are some issues in the helm project, which are related to this issue.
https://github.com/helm/helm/issues/6009
https://github.com/helm/helm/issues/6461

I think this should either be fixed in the Azure DevOps task or, as a compromise, you can add an option to ignore STDERR as it is done in the Azure CLI task (see option "Fail on Standard Error").
Nevertheless i will create another issue in the helmet project. Somewhere it should be solved.

[hhellbusch]

The changes at https://github.com/helm/helm/pull/7409 should resolve this issue. I haven't personally tested this yet, but v3.1.0-rc.1 or newer of Helm should work as desired.

[vithati]

We have added option in task https://github.com/microsoft/azure-pipelines-tasks/pull/12211 to decide fail on stderr or not. It will take couple of days to deploy this change,

[vcorr]

Still happening it seems, Azure DevOps...
Found tool in cache: helm 3.2.4 x64
##[error]history.go:52: [debug] getting history for release xyz

[akazemis]

Still happening it seems, Azure DevOps...
Found tool in cache: helm 3.2.4 x64
##[error]history.go:52: [debug] getting history for release xyz

Same here, seems like a bug in Helm3. That line has both [error] and [debug] tags on it

[robinkb]

@vithati this is absolutely _not_ an issue with Helm 3. Plenty of CLI tools (like curl, for example) write their debug output to STDERR to make it easier to differentiate between normal output and debug output. The failOnStderr option should not be on by default, because Helm will exit with a non-zero exit code when it fails, like any well-behaved CLI tool. You should only need this option with CLI tools that do not follow this convention.

[joostvanvucht]

still happening. as soon as we add
arguments: --debug
to the HelmDeploy@0 task
the build pipeline fails with

[error]history.go:53: [debug] getting history for release sample-application

@vithati please re-open

[ds-ms]

@shigupt202 could you take a look

[shigupt202]

The default value of failOnStderr has been set to false in this PR. The deployment is already in progress and should have reached most of the customers. If the customers are seeing HelmDeployV0 task version as 0.177.0 and still facing this problem, we'll reopen this issue (given that they have not explicitly changed the value of failOnStderr to true in the pipeline).

[GokGokalp]

I'm also getting this error with helm 3.2.4 x64 and also tried other versions.

Any updates?

[shigupt202]

@GokGokalp Can you share the HelmDeploy task version? You can find it in the pipeline logs.

[GokGokalp]

@GokGokalp Can you share the HelmDeploy task version? You can find it in the pipeline logs.

This is the version: Version : 0.177.0

As temporary solution, I've disabled the "Fail on Standard Error" property.

[shigupt202]

That's what the solution is. We have made the default value of failOnStderr to false now. If it was explicitly set to true in the pipeline earlier, you need to make it false.