Helm-operator: Helm Operator 1.4.0 SIGSEGV

[SovakPaleny]

Describe the bug

Upgrading from 1.2.0 to 1.4.0, fresh 1.4.0 (1.3.0 as well) helm install, old version removed, CRD also removed.

To Reproduce

Steps to reproduce the behaviour:

1. Provide the Helm Operator install instructions
   
   
   
   • helm-operator installed using helm.
   
   
   • values:
   
   

podLabels:
  aadpodidbinding: helm-operator
helm:
  versions: v3
kube:
  config: null
configureRepositories:
  enable: false
prometheus:
  enabled: true
  serviceMonitor:
    # Enables ServiceMonitor creation for the Prometheus Operator
    create: true
dashboards:
  enabled: true
  namespace: prometheus
  nameprefix: helm-operator
securityContext:
  fsGroup: 1000
  runAsUser: 1000
  runAsGroup: 1000
containerSecurityContext:
  helmOperator:
    runAsUser: 1000
    runAsNonRoot: true
initContainers:
  - name: azure-plugin-install
    image: mcr.microsoft.com/azure-cli:2.23.0
    command: ['sh', '-c', '--']
    args:
      - |
        mkdir -p /share-plugins/azure-plugin
        cp -R -L /azure-plugin/* /share-plugins/azure-plugin
        chmod +x /share-plugins/azure-plugin/*.sh
    securityContext:
      runAsUser: 1000
      runAsNonRoot: true
    volumeMounts:
    - name: plugins-cache
      mountPath: /share-plugins
      subPath: v3-config
    - name: azure-plugin
      mountPath: /azure-plugin
  - name: azcli-acr-login
    image: mcr.microsoft.com/azure-cli:2.23.0
    command: [ "/bin/bash", "-c", "--" ]
    args:
      - |
        set -euo pipefail
        mkdir -p "/tmp/azcli-bin"
        export PATH="/tmp/azcli-bin:$PATH"
        echo "az login: "
        az login --identity --scope '${data.azurerm_container_registry.acr.id}';
        cat << EOF > /tmp/azcli-bin/helm
        #!/bin/sh
        echo "\$8" > /tmp/token
        EOF
        chmod +x /tmp/azcli-bin/helm;
        az acr helm repo add -n '${data.azurerm_container_registry.acr.name}'
        cp /tmp/token /data/token
        TOKEN_VALIDITY=$(cat /data/token | jq -R 'gsub("-";"+") | gsub("_";"/") | split(".") | .[1] | @base64d | fromjson | (.exp - .nbf)/60')
        TOKEN_NBF=$(cat /data/token | jq -R 'gsub("-";"+") | gsub("_";"/") | split(".") | .[1] | @base64d | fromjson | .nbf')
        echo "token file created $(date) with the token nbf $(date -d "@$TOKEN_NBF"). It's valid for $TOKEN_VALIDITY minutes"
    securityContext:
      runAsUser: 1000
      runAsNonRoot: true
    volumeMounts:
      - name: azcli-work
        mountPath: /.azure
      - name: acr-data
        mountPath: /data
sidecarContainers:
  - name: azcli-acr-login-refresh
    image: ${var.azure_cli_image}
    command: [ "/bin/bash", "-c", "--" ]
    securityContext:
      runAsUser: 1000
      runAsNonRoot: true
    args:
      - |
        set -euo pipefail
        mkdir -p "/tmp/azcli-bin"
        export PATH="/tmp/azcli-bin:$PATH"
        trap : TERM INT;
        (while true; do
          echo "Refreshing token, az login: "
          az login --identity --scope '${data.azurerm_container_registry.acr.id}';
          cat << EOF > /tmp/azcli-bin/helm
        #!/bin/sh
        echo "\$8" > /tmp/token
        EOF
          chmod +x /tmp/azcli-bin/helm;
          az acr helm repo add -n '${data.azurerm_container_registry.acr.name}'
          cp /tmp/token /data/token
          TOKEN_VALIDITY=$(cat /data/token | jq -R 'gsub("-";"+") | gsub("_";"/") | split(".") | .[1] | @base64d | fromjson | (.exp - .nbf)/60')
          TOKEN_NBF=$(cat /data/token | jq -R 'gsub("-";"+") | gsub("_";"/") | split(".") | .[1] | @base64d | fromjson | .nbf')
          SLEEP_SECONDS=$((TOKEN_VALIDITY/3*60))
          echo "token file created $(date) with the token nbf $(date -d "@$TOKEN_NBF"). It's valid for $TOKEN_VALIDITY minutes, refreshing after $SLEEP_SECONDS seconds."
          sleep "$SLEEP_SECONDS";
        done;
        ) & wait
    volumeMounts:
      - name: azcli-work
        mountPath: /.azure
      - name: acr-data
        mountPath: /data
extraVolumes:
  - name: azcli-work
    emptyDir: {}
  - name: acr-data
    emptyDir: {}
  - name: plugins-cache
    emptyDir: {}
  - name: repository-cache
    emptyDir: {}
  - name: azure-plugin
    configMap:
      name: helm-operator-azure-plugin
      defaultMode: 0400
extraVolumeMounts:
  - name: acr-data
    mountPath: /acr
    readOnly: true
  - name: plugins-cache
    mountPath: /.cache/helm/plugins
    subPath: v3
  - name: plugins-cache
    mountPath: /.local/share/helm/plugins
    subPath: v3-config
  - name: repository-cache
    mountPath: /.cache/helm/repository
extraEnvs:
  - name: AZURE_TOKEN_PATH
    value: "/acr/token"

1. Provide a HelmRelease example

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: k8s-controller-sidecars
  namespace: kube-system
  annotations:
    fluxcd.io/automated: 'true'
spec:
  releaseName: k8s-controller-sidecars
  resetValues: true
  skipCRDs: true
  chart:
    repository: azure+https://xxxx.azurecr.io/helm/v1/repo
    name: k8s-controller-sidecars
    version: 0.1.0
  values:
    test: xx

1. Post the HelmRelease status, you can get this by running kubectl describe helmrelease <name>
   no release installed.

Expected behavior

A clear and concise description of what you expected to happen.

Logs

E0714 14:00:59.833522       7 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 18 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1a9f400, 0x2ed54f0)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x92
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
panic(0x1a9f400, 0x2ed54f0)
    /usr/local/go/src/runtime/panic.go:969 +0x166
k8s.io/cli-runtime/pkg/genericclioptions.(*ConfigFlags).ToRawKubeConfigLoader(0x0, 0x7de72989c868250d, 0x1)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/k8s.io/cli-runtime/pkg/genericclioptions/config_flags.go:122 +0x22
helm.sh/helm/v3/pkg/cli.(*EnvSettings).Namespace(0xc0000e82a0, 0xc00029d5c0, 0x1d0b1a9)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/cli/environment.go:177 +0x2f
helm.sh/helm/v3/pkg/cli.(*EnvSettings).EnvVars(0xc0000e82a0, 0x160)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/cli/environment.go:158 +0x5a1
helm.sh/helm/v3/pkg/plugin.SetupPluginEnv(0xc0000e82a0, 0xc000059e70, 0xa, 0xc000056680, 0x27)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/plugin/plugin.go:276 +0x40
helm.sh/helm/v3/pkg/getter.(*pluginGetter).Get(0xc0003c3790, 0xc000604180, 0x3f, 0xc00060cc00, 0x4, 0x4, 0xc00030fac0, 0xc00032a500, 0x50)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/getter/plugingetter.go:73 +0x290
helm.sh/helm/v3/pkg/repo.(*ChartRepository).DownloadIndexFile(0xc00032a500, 0xc0005a0c00, 0x3, 0x4, 0xc00032a500)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/repo/chartrepo.go:127 +0x330
helm.sh/helm/v3/pkg/repo.FindChartInAuthAndTLSRepoURL(0xc00016f500, 0x34, 0x0, 0x0, 0x0, 0x0, 0xc000753010, 0x8, 0xc0004fefc0, 0x12, ...)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/repo/chartrepo.go:240 +0x2cd
helm.sh/helm/v3/pkg/repo.FindChartInAuthRepoURL(...)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/repo/chartrepo.go:212
helm.sh/helm/v3/pkg/repo.FindChartInRepoURL(0xc00016f500, 0x34, 0xc000753010, 0x8, 0xc0004fefc0, 0x12, 0x0, 0x0, 0x0, 0x0, ...)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/repo/chartrepo.go:205 +0x114
github.com/fluxcd/helm-operator/pkg/helm/v3.(*HelmV3).PullWithRepoURL(0xc00000cb40, 0xc00016f500, 0x34, 0xc000753010, 0x8, 0xc0004fefc0, 0x12, 0xc0000360f0, 0x50, 0x0, ...)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/helm/v3/pull.go:68 +0x374
github.com/fluxcd/helm-operator/pkg/chartsync.downloadChart(...)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/chartsync/download.go:57
github.com/fluxcd/helm-operator/pkg/chartsync.EnsureChartFetched(0x2051b60, 0xc00000cb40, 0x1cee55d, 0x4, 0xc00030e480, 0xc000518930, 0xc00072b3d8, 0x5519c3, 0x1fddd40, 0xc0005603e0)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/chartsync/download.go:26 +0x3c8
github.com/fluxcd/helm-operator/pkg/release.(*Release).prepareChart(0xc0005434a0, 0x2051b60, 0xc00000cb40, 0xc00072da90, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000669710, ...)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/release/release.go:161 +0x163
github.com/fluxcd/helm-operator/pkg/release.(*Release).Sync(0xc0005434a0, 0xc00072da90, 0x0, 0x0)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/release/release.go:81 +0x4a7
github.com/fluxcd/helm-operator/pkg/operator.(*Controller).syncHandler(0xc000312720, 0xc0006fe000, 0xe, 0x0, 0x0)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/operator/operator.go:231 +0x5d2
github.com/fluxcd/helm-operator/pkg/operator.(*Controller).processNextWorkItem.func1(0xc000312720, 0x19ff8a0, 0xc000900000, 0x0, 0x0)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/operator/operator.go:185 +0xd7
github.com/fluxcd/helm-operator/pkg/operator.(*Controller).processNextWorkItem(0xc000312720, 0x203000)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/operator/operator.go:192 +0x87
github.com/fluxcd/helm-operator/pkg/operator.(*Controller).runWorker(0xc000312720)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/operator/operator.go:142 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00025e030)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00025e030, 0x1fdfa40, 0xc0005186f0, 0x1, 0xc000096420)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00025e030, 0x3b9aca00, 0x0, 0x1de4e01, 0xc000096420)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc00025e030, 0x3b9aca00, 0xc000096420)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/fluxcd/helm-operator/pkg/operator.(*Controller).Run
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/operator/operator.go:128 +0x188
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xa8 pc=0x16340a2]

goroutine 18 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
panic(0x1a9f400, 0x2ed54f0)
    /usr/local/go/src/runtime/panic.go:969 +0x166
k8s.io/cli-runtime/pkg/genericclioptions.(*ConfigFlags).ToRawKubeConfigLoader(0x0, 0x7de72989c868250d, 0x1)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/k8s.io/cli-runtime/pkg/genericclioptions/config_flags.go:122 +0x22
helm.sh/helm/v3/pkg/cli.(*EnvSettings).Namespace(0xc0000e82a0, 0xc00029d5c0, 0x1d0b1a9)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/cli/environment.go:177 +0x2f
helm.sh/helm/v3/pkg/cli.(*EnvSettings).EnvVars(0xc0000e82a0, 0x160)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/cli/environment.go:158 +0x5a1
helm.sh/helm/v3/pkg/plugin.SetupPluginEnv(0xc0000e82a0, 0xc000059e70, 0xa, 0xc000056680, 0x27)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/plugin/plugin.go:276 +0x40
helm.sh/helm/v3/pkg/getter.(*pluginGetter).Get(0xc0003c3790, 0xc000604180, 0x3f, 0xc00060cc00, 0x4, 0x4, 0xc00030fac0, 0xc00032a500, 0x50)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/getter/plugingetter.go:73 +0x290
helm.sh/helm/v3/pkg/repo.(*ChartRepository).DownloadIndexFile(0xc00032a500, 0xc0005a0c00, 0x3, 0x4, 0xc00032a500)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/repo/chartrepo.go:127 +0x330
helm.sh/helm/v3/pkg/repo.FindChartInAuthAndTLSRepoURL(0xc00016f500, 0x34, 0x0, 0x0, 0x0, 0x0, 0xc000753010, 0x8, 0xc0004fefc0, 0x12, ...)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/repo/chartrepo.go:240 +0x2cd
helm.sh/helm/v3/pkg/repo.FindChartInAuthRepoURL(...)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/repo/chartrepo.go:212
helm.sh/helm/v3/pkg/repo.FindChartInRepoURL(0xc00016f500, 0x34, 0xc000753010, 0x8, 0xc0004fefc0, 0x12, 0x0, 0x0, 0x0, 0x0, ...)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/helm.sh/helm/v3/pkg/repo/chartrepo.go:205 +0x114
github.com/fluxcd/helm-operator/pkg/helm/v3.(*HelmV3).PullWithRepoURL(0xc00000cb40, 0xc00016f500, 0x34, 0xc000753010, 0x8, 0xc0004fefc0, 0x12, 0xc0000360f0, 0x50, 0x0, ...)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/helm/v3/pull.go:68 +0x374
github.com/fluxcd/helm-operator/pkg/chartsync.downloadChart(...)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/chartsync/download.go:57
github.com/fluxcd/helm-operator/pkg/chartsync.EnsureChartFetched(0x2051b60, 0xc00000cb40, 0x1cee55d, 0x4, 0xc00030e480, 0xc000518930, 0xc00072b3d8, 0x5519c3, 0x1fddd40, 0xc0005603e0)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/chartsync/download.go:26 +0x3c8
github.com/fluxcd/helm-operator/pkg/release.(*Release).prepareChart(0xc0005434a0, 0x2051b60, 0xc00000cb40, 0xc00072da90, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000669710, ...)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/release/release.go:161 +0x163
github.com/fluxcd/helm-operator/pkg/release.(*Release).Sync(0xc0005434a0, 0xc00072da90, 0x0, 0x0)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/release/release.go:81 +0x4a7
github.com/fluxcd/helm-operator/pkg/operator.(*Controller).syncHandler(0xc000312720, 0xc0006fe000, 0xe, 0x0, 0x0)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/operator/operator.go:231 +0x5d2
github.com/fluxcd/helm-operator/pkg/operator.(*Controller).processNextWorkItem.func1(0xc000312720, 0x19ff8a0, 0xc000900000, 0x0, 0x0)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/operator/operator.go:185 +0xd7
github.com/fluxcd/helm-operator/pkg/operator.(*Controller).processNextWorkItem(0xc000312720, 0x203000)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/operator/operator.go:192 +0x87
github.com/fluxcd/helm-operator/pkg/operator.(*Controller).runWorker(0xc000312720)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/operator/operator.go:142 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00025e030)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00025e030, 0x1fdfa40, 0xc0005186f0, 0x1, 0xc000096420)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00025e030, 0x3b9aca00, 0x0, 0x1de4e01, 0xc000096420)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc00025e030, 0x3b9aca00, 0xc000096420)
    /home/circleci/go/src/github.com/fluxcd/helm-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/fluxcd/helm-operator/pkg/operator.(*Controller).Run
    /home/circleci/go/src/github.com/fluxcd/helm-operator/pkg/operator/operator.go:128 +0x188

Additional context

• Helm Operator version:
• Kubernetes version: 1.19.11 Azure AKS
• Git provider:
• Helm repository provider: custom azure+https provider:

#!/bin/bash
set -euo pipefail

certFile=$1
keyFile=$2
caFile=$3
URL=$4

HOST_PATH=${URL#*://}
ACR_NAME=${HOST_PATH%%.*}
AZURE_URL=${URL#*+}

TOKEN=$(cat "${AZURE_TOKEN_PATH}")

USERNAME=00000000-0000-0000-0000-000000000000
curl -sSL -u "${USERNAME}:${TOKEN}" "${AZURE_URL}"

name: "helm-azure"
version: "0.0.1"
description: |-
  Support helm acr repository with token refresh
downloaders:
  - command: "helm-azure.sh"
    protocols:
      - "azure+https"

[kingdonb]

Thank you for the report! Is this a Helm Plugin issue? Looks like the report in #626

We have an idea of how to restore the capability to use plugins, which was broken by a Helm upgrade. Very sorry for the inconvenience. (We're pretty certain the breakage is also present in 1.3.0, FYI.)

[SovakPaleny]

@kingdonb thank you! It's the same for 1.3.0, I just tried it.

[kingdonb]

I think we definitely want to fix this, but to be clear Helm Operator is not planned to be supported forever, and Helm Controller does not (and does not plan to) support Helm plugins, as I understand it. I personally don't have a lot of details or context about this position and it's not my decision, but from what I understand it's not very likely to change going forward.

So while there is a path to restore this functionality in Helm Operator, I'm inclined to try and help you migrate away from using a Helm plugin if possible, or better understand why it is not possible and what use cases this will break. We really don't want for there to be a substantial number of users to be stuck on Helm Operator when support is ultimately ended in favor of Helm Controller.

Can you elaborate on what this plugin is for, and how you're using it, so we can think and talk about how this could perhaps be solved without a Helm plugin (and so that we can help you consider migrating to the new Helm Controller, if possible?)

[SovakPaleny]

We are on Azure, we used to use a service principal to access azure container services (with fixed passwords) and moved to user-assigned identities with short-lived passwords. So the plugin is a tool to work with short (around 1 hour) passwords.