Azure-cli: az acr helm commands not working

[WilldotNET]

Yesterday my pipelines were working.

Today, they fail with :
This command is implicitly deprecated because command group 'acr helm' is deprecated and will be removed in a future release. Use 'helm v3' instead.

The error is caused by a bash script step in the pipeline :
bash: |
echo attempting push
chartPackage=$(ls $(projectName)/$(imageName)-$(helmChartVersion).tgz)
az acr helm push \
-n $(registryName) \
-u $(registryLogin) \
-p '$(registryPassword)' \
$chartPackage
echo $(jq -n --arg version "$(helmChartVersion)" '{helmChartVersion: $version}') > $(build.artifactStagingDirectory)/variables.json
failOnStderr: true
name: helmPush
condition: and(succeeded(), eq(variables['build.sourceBranch'], 'refs/heads/theNameOfTheBranch'))
displayName: 'az acr helm push'

I am specifying Helm version 3.3.0-rc.1

Up until today, this was a warning, but today it's a pipeline stopping error, docs seem to suggest upgrading to

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 2bed94c3-b7a7-40d4-0102-8c9b1f1145ca
• Version Independent ID: 2d4ac8bd-dc7a-236c-56db-7f0e1ce0fd05
• Content: az acr helm
• Content Source: src/azure-cli/azure/cli/command_modules/acr/_help.py
• Service: container-registry
• GitHub Login: @rloutlaw
• Microsoft Alias: routlaw

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @toddysm, @yugangw-MSFT.

[yonzhan]

acr

[yungezz]

hi @WilldotNET could you pls share error message, repro steps to help investigation? thanks.

[WilldotNET]

hi @WilldotNET could you pls share error message, repro steps to help investigation? thanks.

Error Message:

[error]Bash wrote one or more lines to the standard error stream.

[error]This command is implicitly deprecated because command group 'acr helm' is deprecated and will be removed in a future release. Use 'helm v3' instead.

Reproduce with:

Create the following bash task as a step in the azure pipeline YAML:

• bash: |

  chartPackage=$(ls $(projectName)/$(imageName)-$(helmChartVersion).tgz)
  az acr helm push \
  -n $(registryName) \
  -u $(registryLogin) \
  -p '$(registryPassword)' \
  $chartPackage
  echo $(jq -n --arg version "$(helmChartVersion)" '{helmChartVersion: $version}') > $(build.artifactStagingDirectory)/variables.json
  failOnStderr: true
  name: helmPush
  condition: and(succeeded(), eq(variables['build.sourceBranch'], 'refs/heads/master'))
  displayName: 'az acr helm push'

(Note - real branch name changed to master for public forum)

[WilldotNET]

Also tried with helm version 3.2.4 but I get the same issue.

[WilldotNET]

Here is the log for this task from the same pipeline yesterday if it helps:

2020-07-21T14:04:05.6034671Z Task : Bash
2020-07-21T14:04:05.6034914Z Description : Run a Bash script on macOS, Linux, or Windows
2020-07-21T14:04:05.6035141Z Version : 3.171.1
2020-07-21T14:04:05.6035334Z Author : Microsoft Corporation
2020-07-21T14:04:05.6035638Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
2020-07-21T14:04:05.6035970Z ==============================================================================
2020-07-21T14:04:05.7481419Z Generating script.
2020-07-21T14:04:05.7510965Z ========================== Starting Command Output ===========================
2020-07-21T14:04:05.7537327Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/74de69e3-7f01-4ff2-b550-9771aa53dfdf.sh
2020-07-21T14:04:05.7629652Z attempting push
2020-07-21T14:04:09.1209741Z {
2020-07-21T14:04:09.1210521Z "saved": true
2020-07-21T14:04:09.1210766Z }
2020-07-21T14:04:09.2537389Z
2020-07-21T14:04:09.2620374Z ##[section]Finishing: az acr helm push
2020-07-21T14:04:09.2671638Z ##[section]Starting: PublishPipelineArtifact

[yugangw-msft]

@WilldotNET, any chance you can get rid of "failOnStderr: true"? Warning always goes to stderr, but doesn't mean the command failed.
Helm2 is deprecating. On how to use helm3, please check out: https://docs.microsoft.com/en-us/azure/container-registry/container-registry-helm-repos

[WilldotNET]

@WilldotNET, any chance you can get rid of "failOnStderr: true"? Warning always goes to stderr, but doesn't mean the command failed.
Helm2 is deprecating. On how to use helm3, please check out: https://docs.microsoft.com/en-us/azure/container-registry/container-registry-helm-repos

Thanks @yugangw-msft - this stopped the warning breaking the build. I'll check out moving to Helm3 - but I thought I was already using version 3 - the error message seemed to state the az acr helm push command was being deprecated. Is az acr helm push being removed do you know?

[yugangw-msft]

az acr helm is based on helm2. We won't pull the plug right now, but we don't make further change on it. Helm3 will be a standalone command; and "az acr helm" command will just be removed when the time finally comes

[WilldotNET]

Just as additional info on this, I'm struggling to replicate what az acr helm push did with helm push on v3 -

Based on -- https://github.com/Azure/phippyandfriends/blob/master/common/ci-steps-template.yml

`- bash: |
export HELM_EXPERIMENTAL_OCI=1
chartPackage=$(ls $(projectName)/$(imageName)-$(helmChartVersion).tgz)
echo chartPackage is $chartPackage
helm chart save $chartPackage $(registryServerName)/$(imageName)
failOnStderr: true
displayName: 'helm save'

• bash: |
  export HELM_EXPERIMENTAL_OCI=1
  echo $(registryPassword) | helm registry login $(registryServerName) \
  --username $(registryLogin) \
  --password-stdin
  failOnStderr: true
  displayName: 'helm login to acr'
• bash : |
  export HELM_EXPERIMENTAL_OCI=1
  echo attempting push
  ls DotNet/MyIdentityCloud/Web/ServiceManager/src
  helm chart push $(registryServerName)/$(imageName):$(helmChartVersion)
  echo $(jq -n --arg version "$(helmChartVersion)" '{helmChartVersion: $version}') > $(build.artifactStagingDirectory)/variables.json
  failOnStderr: true
  displayName: 'helm chart push to acr'`

I believe the problem is with the push, as when I use az acr helm push, the cd pipeline part of the example project I've linked to works fine. But when I use v3 of helm push - the helm install in the CD pipeline says it can't download the chart. Just wondered if I was missing something with the manual helm push?

[yugangw-msft]

@shizhMSFT do you have any ideas?

[meappy]

I've given up with making helm v3.4.0 authenticating with ACR

Downgraded to helm v3.3.3 and proceeded to authenticate/add the ACR repo with the following command

az acr helm repo add --name <containerregistry>

I'm hoping something similar for helm v3.4.0 will be possible with az acr helm3 or similar in future.

[stephenmoloney]

Requiring people to use HELM_EXPERIMENTAL_OCI=1 does not seem like a good idea to me.

While I am a believer and supporter of the OCI concept and would like things to go that way, there are various tooling out there which will not support this right now.

With HELM_EXPERIMENTAL_OCI=1, the word EXPERIMENTAL alone is enough to put off many people and organisations.

My suggestion is that this deprecation should wait for HELM_EXPERIMENTAL_OCI flag to be no longer required and part of standard behaviour.

Ref https://github.com/Azure/azure-cli/issues/14498

[andyliddle]

Also at the moment, it seems that OCI support for helm dependency URLs is still not implemented https://github.com/helm/helm/issues/6593. There would be no way for us to reference dependant charts in ACR without the v2 support.

[yugangw-msft]

@sajayantony @SteveLasker to comment

[SteveLasker]

Hi folks,
We appreciate it's a bit confusing as there are too many moving parts at various stages of development.
I'll writeup a blog post that explains this a bit further. Here are some rough notes to fill the gap:

To Yugangs point, we will not deprecate the helm repo ... feature supported by az acr helm until helm registry is elevated from experimental. We understand the limits the "experimental" label entails. And, the helm team is just not ready to remove experimental as they believe helm registry is not at an MVP level. It does look like they're getting close, and there was a good discussion on what was required as an MVP.
However, this is not directly correlated with helm 2 or helm 3. Helm 3 supports helm repo and helm registry. The helm team has deprecated helm 2.

helm repo is supported with helm 3. However, the helm repo feature simply doesn't scale with the dependency on index.yaml to represent all helm charts in a repository and doesn't have full feature support in registries such content listing.

To assure all registry features apply to helm charts, we need to move to the OCI implementation of helm registry.
ACR has GA support of OCI Artifacts, which supports helm registry. We will continue evolving the spec for OCI Artifacts, now supported by all major registries.

There is work left on the helm team to complete the helm registry features.

So, to summarize:

• az acr helm will continue until helm registry is lifted from experimental
• users can use helm repo and az acr helm ... with helm 3 charts
• moving to helm registry will support all OCI compliant registires, and provide the scalability of automated build scenarios
• acr has full GA support of OCI Artifacts, which includes helm registry support
• the helm community owns the completion of helm registry features, and we'll support all OCI Artifact enhancements required.

Steve

[stephenmoloney]

Hi folks,
So, to summarize:

• az acr helm will continue until helm registry is lifted from experimental
• users can use helm repo and az acr helm ... with helm 3 charts
• moving to helm registry will support all OCI compliant registires, and provide the scalability of automated build scenarios
• acr has full GA support of OCI Artifacts, which includes helm registry support
• the helm community owns the completion of helm registry features, and we'll support all OCI Artifact enhancements required.

Steve

Sounds good to me!