Helm-operator: Helm Release failing on ChartFetched stage with inconclusive error logs

[OliverLeighC]

I'm using flux with the helm-operator and I'm running into some issues that I think are related to the helm v3 upgrade.
Basically the helm release is failing, and it seems to be failing on the 'ChartFetched' stage, but the error logs are not super clear

output from kubectl describe hr graphql

Name:         graphql
Namespace:    default
Labels:       fluxcd.io/sync-gc-mark=sha256.ycVMSXoZ-_BFlG-vBhkO8RbNCbaMmHRXkdcu0ZOIAho
Annotations:  flux.weave.works/automated: true
              flux.weave.works/tag.chart-image: glob:dev-*
              fluxcd.io/sync-checksum: 57756c08e5644a1b5562752ccc965c23e4d25b93
              kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"helm.fluxcd.io/v1","kind":"HelmRelease","metadata":{"annotations":{"flux.weave.works/automated":"true","flux.weave.works/ta...
API Version:  helm.fluxcd.io/v1
Kind:         HelmRelease
Metadata:
  Creation Timestamp:  2020-01-02T18:26:11Z
  Generation:          1
  Resource Version:    231235
  Self Link:           /apis/helm.fluxcd.io/v1/namespaces/default/helmreleases/graphql
  UID:                 5a5a240b-2d8d-11ea-bcd5-022b9c111279
Spec:
  Chart:
    Git:         [email protected]:<---repo>.git
    Path:        charts/graphql
    Ref:         dev-experiment
  Helm Version:  v3
  Release Name:  graphql
  Values:
    Image:  ----image is here 
    Tag:    dev-*
Status:
  Conditions:
    Last Transition Time:  2020-01-02T18:30:23Z
    Last Update Time:      2020-01-02T18:42:22Z
    Message:               successfully cloned chart revision: 7cedb2f90288c7bcfee8644b90b92588a56b2c2d
    Reason:                GitRepoCloned
    Status:                True
    Type:                  ChartFetched
    Last Transition Time:  2020-01-02T18:26:22Z
    Last Update Time:      2020-01-02T18:42:22Z
    Message:               failed to upgrade chart for release [graphql]: an error occurred while uninstalling the release. original install error: no objects visited: uninstallation completed with 1 error(s): object not found, skipping delete
    Reason:                HelmInstallFailed
    Status:                False
    Type:                  Released
  Observed Generation:     1
  Release Name:            
  Release Status:          
  Values Checksum:         ba605d4f35decc1d51f49063276dc9fbe247b459528ceb7d362d03c506bf8a25
Events:
  Type    Reason       Age                From           Message
  ----    ------       ----               ----           -------
  Normal  ChartSynced  15m (x3 over 17m)  helm-operator  Chart managed by HelmRelease processed
  Normal  ChartSynced  78s (x6 over 13m)  helm-operator  Chart managed by HelmRelease processed

the kubectl pod logs for the helm operator pod, looks like it fails on the first release and then the uninstall fails because the release doesn't exist yet (since the initial install is failing), but I can't figure out why it's failing

W0102 18:30:22.552506       7 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
ts=2020-01-02T18:30:22.553730479Z caller=operator.go:99 component=operator info="setting up event handlers"
ts=2020-01-02T18:30:22.553769779Z caller=operator.go:119 component=operator info="event handlers set up"
ts=2020-01-02T18:30:22.553785183Z caller=main.go:279 component=helm-operator info="waiting for informer caches to sync"
ts=2020-01-02T18:30:22.653965914Z caller=main.go:284 component=helm-operator info="informer caches synced"
ts=2020-01-02T18:30:22.654035675Z caller=operator.go:131 component=operator info="starting operator"
ts=2020-01-02T18:30:22.654066913Z caller=operator.go:133 component=operator info="starting workers"
ts=2020-01-02T18:30:22.6552347Z caller=git.go:94 component=gitchartsync info="starting sync of git chart sources"
ts=2020-01-02T18:30:22.655549167Z caller=git.go:289 component=gitchartsync info="started mirroring new remote" [email protected]:RockCreekGroup/kubernetes-gitops.git [email protected]:RockCreekGroup/kubernetes-gitops.git
ts=2020-01-02T18:30:22.656003949Z caller=server.go:42 component=daemonhttp info="starting HTTP server on :3030"
ts=2020-01-02T18:30:22.683972999Z caller=release.go:112 component=release release=graphql targetNamespace=default resource=default:helmrelease/graphql helmVersion=v3 error="chart not ready: no existing git mirror found"
ts=2020-01-02T18:30:23.007642025Z caller=checkpoint.go:24 component=checkpoint msg="up to date" latest=0.10.1
ts=2020-01-02T18:30:23.130131518Z caller=release.go:311 component=release release=graphql targetNamespace=default resource=default:helmrelease/graphql helmVersion=v3 info="no existing release" action=install
I0102 18:30:23.371730       7 client.go:87] creating 0 resource(s)
I0102 18:30:23.371751       7 install.go:312] Install failed and atomic is set, uninstalling release
I0102 18:30:23.376185       7 uninstall.go:91] uninstall: Deleting graphql
I0102 18:30:23.381430       7 uninstall.go:124] purge requested for graphql
ts=2020-01-02T18:30:23.40108288Z caller=release.go:217 component=release release=graphql targetNamespace=default resource=default:helmrelease/graphql helmVersion=v3 error="Helm release failed" revision=7cedb2f90288c7bcfee8644b90b92588a56b2c2d err="failed to upgrade chart for release [graphql]: an error occurred while uninstalling the release. original install error: no objects visited: uninstallation completed with 1 error(s): object not found, skipping delete"
ts=2020-01-02T18:33:22.560570417Z caller=operator.go:309 component=operator info="enqueuing release" resource=default:helmrelease/graphql

when I list the workloads with fluxctl list-workloads --k8s-fwd-ns fluxcd

I get:

WORKLOAD                     CONTAINER  IMAGE  RELEASE  POLICY
default:helmrelease/graphql                             automated

so no container, image, or release but the right workload and policy (automated)

our helm release looks like this:

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: graphql
  namespace: default
  annotations:
    flux.weave.works/automated: "true"
    flux.weave.works/tag.chart-image: glob:dev-*
spec:
  releaseName: graphql
  helmVersion: v3
  chart:
    git: [email protected]:<git repo>.git
    path: charts/graphql
    ref: dev-experiment
  values:
    tag: dev-*
    image: <image here>

this scripts we were using to set up flux and the helm operator are:

helm repo add fluxcd https://charts.fluxcd.io

kubectl create ns fluxcd

# https://github.com/fluxcd/flux/blob/master/chart/flux/README.md
helm upgrade -i flux fluxcd/flux --wait \
  --namespace fluxcd \
  --set [email protected]:<git repo> \
  --set git.branch="dev-experiment" \
  --set git.ssh.secretName=flux-git-deploy \
  --debug \
  --set-file ssh.known_hosts=$KNOWNHOSTSPATH

kubectl apply -f https://raw.githubusercontent.com/fluxcd/helm-operator/master/deploy/flux-helm-release-crd.yaml

helm upgrade -i helm-operator fluxcd/helm-operator --wait \
  --namespace fluxcd \
  --set git.ssh.secretName=flux-git-deploy \
  --set helm.versions=v3 \
  --debug \
  --set-file git.ssh.known_hosts=$KNOWNHOSTSPATH


## Add deploy key to git repo
FluxPub=$(fluxctl identity --k8s-fwd-ns fluxcd)

curl --user $GITCREDENTIALS -d "{\"title\":\"flux-$ENV\", \"key\":\"$FluxPub\"}" \
     -X POST https://api.github.com/repos/<git repo>/keys

Is there something we are doing wrong? Or configuring incorrectly?
Because of the helm v3 update there is some out of date documentation around so I want to make sure we aren't missing something obvious. I considered posting this on the flux slack channel but because of the update, I figured this would be better in case other people are running into this same issue, since I'm not sure if this is a bug or if it's an error on my part.

Note: We are using ssh.known_hosts because our repo is private.

[OliverLeighC]

The helm operator version is: 1.0.0-rc5

NAME            NAMESPACE   REVISION    UPDATED                                 STATUS      CHART               APP VERSION
flux            fluxcd      1           2020-01-02 13:21:29.717522 -0500 EST    deployed    flux-1.0.0          1.17.0     
helm-operator   fluxcd      2           2020-01-02 13:30:20.346706 -0500 EST    deployed    helm-operator-0.4.0 1.0.0-rc5  

[stefanprodan]

Can you please confirm that:

• the chart can be installed with Helm v3 CLI
• execing into the helm-op container you can clone the repo

Thanks

[OliverLeighC]

I can clone the repo within the helm-op container:

kubectl exec -it helm-operator-8684975ccb-p42j6 -n fluxcd -- /bin/sh
/home/flux # git clone [email protected]:RockCreekGroup/kubernetes-gitops.git
Cloning into 'kubernetes-gitops'...
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 242 (delta 0), reused 1 (delta 0), pack-reused 241
Receiving objects: 100% (242/242), 40.08 KiB | 10.02 MiB/s, done.
Resolving deltas: 100% (97/97), done.
/home/flux # 

Confirming that it works with the helm v3 CLI now, will report back shortly

[OliverLeighC]

So I ran it on a dry run first and it looked okay
I ran helm install --dry-run --debug ./charts/graphql --generate-name

install.go:148: [debug] Original chart version: ""
install.go:165: [debug] CHART PATH: /<github user>/<repo>/charts/graphql

NAME: graphql-1578325085
LAST DEPLOYED: Mon Jan  6 10:38:06 2020
NAMESPACE: default
STATUS: pending-install
REVISION: 1
TEST SUITE: None
USER-SUPPLIED VALUES:
{}

COMPUTED VALUES:
git:
  branch: dev-experiment
  email: <email>
  url: <git repo>/kubernetes-gitops.git
  user: RCG-MCU
hpa:
  enabled: false
image: <image>
imagePullPolicy: Always
replicaCount: 1

HOOKS:
MANIFEST:

but without the dry-run flag it error'd out:
helm install --debug ./charts/graphql --generate-name

install.go:148: [debug] Original chart version: ""
install.go:165: [debug] CHART PATH: /<user>/<repo>/charts/graphql

client.go:87: [debug] creating 0 resource(s)
Error: no objects visited
helm.go:76: [debug] no objects visited
helm.sh/helm/v3/pkg/kube.init
    /private/tmp/helm-20191206-12717-1g3ioh/src/helm.sh/helm/pkg/kube/client.go:49
runtime.doInit
    /usr/local/Cellar/go/1.13.4/libexec/src/runtime/proc.go:5222
runtime.doInit
    /usr/local/Cellar/go/1.13.4/libexec/src/runtime/proc.go:5217
runtime.doInit
    /usr/local/Cellar/go/1.13.4/libexec/src/runtime/proc.go:5217
runtime.main
    /usr/local/Cellar/go/1.13.4/libexec/src/runtime/proc.go:190
runtime.goexit
    /usr/local/Cellar/go/1.13.4/libexec/src/runtime/asm_amd64.s:1357

[OliverLeighC]

I was missing templates in my chart directory, I added those and now I can install with the helm CLI

[stefanprodan]

Well that's the same error Helm Operator logs no objects visited.

[OliverLeighC]

I am going to tear down our cluster and try again now that the chart has the templates, and see if the helm release will deploy with flux this time.

[OliverLeighC]

It's working now, thanks so much for your help